Member Avatar for The Idiot

I’m building a tagger that searches through a corpus of IM data and tags any instances of words that occur on a wordlist. I've run into a problem and was hoping to find help. I'd like to try and understand exactly why it's not working, so I've laid out everything I can think of that might be helpful.

data = ["@LINE2@ 04-09-2006/DAT 09:05:30/TIM [Team]/CHT  @NAME@ Digs Phung @NAME@ @CONTENT@ you might be crazy @CONTENT@ @LINE2@\n"]

wordlist = ["a", "an", "aardvark", "aardvarks", "aback", "abacus", "you", "be"]

import re

def tagger(data_string):
	string_copy = data_string
	for entry in wordlist:
		p = re.compile("\s" + entry + "\s", re.IGNORECASE)
		g = p.search(string_copy)
		if g == None:
			pass
		else:
			h = g.group()
			space_copy = string_copy.replace(h, h + "/TAG ")
			string_copy = space_copy.replace(" /TAG", "/TAG")
	return string_copy

tagged = []

for line in data:
	x = tagger(line)
	tagged.append(x)

This works as expected producing:

tagged = /CHT @NAME@ Digs Phung @NAME@ @CONTENT@ you/TAG might be/TAG crazy @CONTENT@ @LINE3@']

But when I do the same thing to the full wordlist (~40k words) and data (a list with ~1 million strings), I get the following error:

Traceback (most recent call last):
  File "<pyshell#197>", line 2, in <module>
    x = tagger(data_string)
  File "<pyshell#195>", line 4, in tagger
    p = re.compile("\s" + entry + "\s", re.IGNORECASE)
  File "C:\Python25\lib\re.py", line 188, in compile
    return _compile(pattern, flags)
  File "C:\Python25\lib\re.py", line 241, in _compile
    raise error, v # invalid expression
error: unexpected end of regular expression

I've run a similar function over the full dataset with smaller wordlists (~50) and haven't had any problems, so I figured that the issue was with something in this particular wordlist. Here's what I've already done:

1) I've tested the wordlist to make sure it only contains alphanumeric characters because I thought that other character might be interfering.

2) The function moves through the wordlist once, and then the error message pops up. The second line of data (where it hangs) is above.

3) I swapped out the variable names to words that couldn't appear on the wordlist, in case there was some conflict there.

4) I did some searches for the error message, but the explanations were way over my head.

Any guidance?

  • 3 Contributors
  • 3 Replies
  • 106 Views
  • 22 Hours Discussion Span
  • Latest Post Latest Post by The Idiot

Dani AI

Generated

A quick, practical summary and some next steps that build on what and tried.

Compiling a new regular expression for every keyword, for every line, is both fragile and very costly. Fragile because a single bad or empty word (or an unescaped regex meta-character) will make the regex engine raise an error; costly because tens of thousands of compiles times millions of lines is expensive. The tokenization + set-membership approach used by is the simplest robust fix for exact-word tagging: split into tokens, normalize case, check membership in a set, and reassemble.

If keeping a regex-based approach is necessary, follow a short debug checklist:

  • Catch and log re.error around the compile call so you can see exactly which list entry caused the failure.
  • Trim and reject empty or whitespace-only entries before building patterns.
  • Escape user-supplied words with re.escape when inserting them into patterns.
  • Use raw strings for patterns (for example, word-boundary markers) to avoid accidental Python string escapes.
  • Precompile patterns once (outside the per-line loop) instead of compiling repeatedly.

For large keyword sets consider algorithms/libraries built for multiple-pattern matching rather than iterating one pattern at a time. Aho–Corasick and FlashText are much faster at scale; see an overview of the Aho–Corasick algorithm and project pages for ready-made Python bindings such as pyahocorasick on PyPI and FlashText on GitHub.

Also watch for correctness/performance details: decide how to treat punctuation (use careful tokenization), use casefold() for Unicode-insensitive matching, avoid repeated string concatenation in tight loops (use list appends + join), and process the corpus as a stream or in batches to keep memory use predictable.

Recommended Answers

All 3 Replies

Member Avatar for mn_kthompson

I've had this happen to me too, where something works on a small scale, but after I've run it against something huge it bombs out. In my case it was a web spider. I don't know what was causing it.

The reason I'm replying is because I was wondering if it would be easier to use string.find() instead of regular expressions to find your tagged words. Or even split the line into individual words and see if each one is in your

data = ["@LINE2@ 04-09-2006/DAT 09:05:30/TIM [Team]/CHT  @NAME@ Digs Phung @NAME@ @CONTENT@ you might be crazy @CONTENT@ @LINE2@\n"]

wordlist = ["a", "an", "aardvark", "aardvarks", "aback", "abacus", "you", "be"]

import re

def tagger(data_string):
	string_copy = ''
	for word in data_string.split():
                if word in wordlist:
                    string_copy += word + ' /TAG '
                else:
                    string_copy += word + ' '
	return string_copy

tagged = []

for line in data:
	x = tagger(line)
	tagged.append(x)

Note that I didn't test this code at all because I was in a hurry. It might be completely wrong, I was just running with an idea. Also, in the time it took me to write this last part I could have just tested the code. I'm also lazy in addition to being in a hurry.

Member Avatar for Member #361407

mn_kthompson ive been making a web crawler (web spider) and i think i know what youre error was: i think it was the login sites because you dont have the auth but i just dont know how to implement mine into a search engine

Member Avatar for The Idiot

thanks thompson. that works perfectly. I've tested it on the full dataset and wordlist and no errors. i guess the lesson here for me is that if you can't figure out why you're getting an error, develop a new strategy that avoids the problematic bit (in this case, .re).

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.