Member Avatar for boiishuvo

I am trying to make a python that can remove any occurences of any word in stopwords from the wordlist, but I don't know what is exactly wrong with this program. Any suggestions?

STOPWORDS = ['a','able','about','across','after','all','almost','also','am','among',
             'an','and','any','are','as','at','be','because','been','but','by','can',
             'cannot','could','dear','did','do','does','either','else','ever','every',
             'for','from','get','got','had','has','have','he','her','hers','him','his',
             'how','however','i','if','in','into','is','it','its','just','least','let',
             'like','likely','may','me','might','most','must','my','neither','no','nor',
           'not','of','off','often','on','only','or','other','our','own','rather','said',
             'say','says','she','should','since','so','some','than','that','the','their',
             'them','then','there','these','they','this','tis','to','too','twas','us',
             'wants','was','we','were','what','when','where','which','while','who',
             'whom','why','will','with','would','yet','you','your']

def remove_stop_words(wordlist, stopwords=STOPWORDS):
    wordlist = raw_input("type a sentence: ")
    marked = []
    for t in wordlist:
        if t.lower() in stopwords:
            marked.append('*')
        else:
            marked.append(t)

remove_stop_words('')
  • 3 Contributors
  • 2 Replies
  • 843 Views
  • 12 Hours Discussion Span
  • Latest Post Latest Post by HiHe

Dani AI

Generated

The immediate bug is that you iterate a string (each character) instead of words. spotted that: you need to split the input into tokens before testing stopword membership. ’s reply shows a working direction, but it can be improved: keep I/O out of the core function, use a set for fast lookups, and handle punctuation and contractions so you don't accidentally glue words or lose spaces.

Keep the function pure (accept text or a list, return the cleaned result) and convert the stopword list to a lowercase set once. A simple but robust tokenization approach uses a regex that preserves whitespace and punctuation tokens; then filter only the word tokens against the stopword set and reassemble the original spacing.

import re

def remove_stop_words(text, stopwords):
    stopset = set(w.lower() for w in stopwords)
    tokens = re.findall(r"\s+|\w+(?:'\w+)?|[^\w\s]+", text)
    out = []
    for tok in tokens:
        if tok.isspace() or not tok[0].isalnum():
            out.append(tok)
        else:
            if tok.lower() not in stopset:
                out.append(tok)
    return ''.join(out)

Notes: pass your existing STOPWORDS to this function (converted to a set each call is cheap). If you prefer masking rather than removal, append a marker instead of skipping. For very robust tokenization (multi-language, edge cases) use a tokenizer from NLTK or spaCy. Also avoid putting raw_input/input inside the logic function — do that only at the UI layer so the function is easy to test.

Recommended Answers

All 2 Replies

Member Avatar for TrustyTony

You do not split user input to words, you should be building the new sentence without stop words, marked does not make sense.

Member Avatar for HiHe

You might want to do it this way:

STOPWORDS = ['a','able','about','across','after','all','almost','also','am','among',
             'an','and','any','are','as','at','be','because','been','but','by','can',
             'cannot','could','dear','did','do','does','either','else','ever','every',
             'for','from','get','got','had','has','have','he','her','hers','him','his',
             'how','however','i','if','in','into','is','it','its','just','least','let',
             'like','likely','may','me','might','most','must','my','neither','no','nor',
           'not','of','off','often','on','only','or','other','our','own','rather','said',
             'say','says','she','should','since','so','some','than','that','the','their',
             'them','then','there','these','they','this','tis','to','too','twas','us',
             'wants','was','we','were','what','when','where','which','while','who',
             'whom','why','will','with','would','yet','you','your']

def remove_stop_words(wordlist, stopwords=STOPWORDS):
    # ask for sentence if wordlist is empty
    if not wordlist:
        sentence = raw_input("type a sentence: ")
        wordlist = sentence.split()
    marked = []
    for t in wordlist:
        if t.lower() in stopwords:
            marked.append('*')
        else:
            marked.append(t)
    return marked

# test empty list
wordlist = []
marked_list = remove_stop_words(wordlist)
print(marked_list)

# test given list
wordlist = "should you beg or steal".split()
marked_list = remove_stop_words(wordlist)
print(marked_list)

''' example ...
type a sentence: hello there
['hello', '*']
['*', '*', 'beg', '*', 'steal']
'''
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.