Member Avatar for gunbuster363

Just as the title.
I want to store some strings inside the program, and later I may store them(the string) in a text file. And I do not want duplicating strings.

As I learned from school, I should use a hash table for it, but I may build a bad hash table. So, I would like to ask all of you, is there someway I can do it like I said in python?

Thanks.

  • 5 Contributors
  • 5 Replies
  • 127 Views
  • 16 Hours Discussion Span
  • Latest Post Latest Post by lrh9

Dani AI

Generated

Summary and practical follow-ups to the replies from , , , and :

Several answers pointed in the right direction; the guidance below clarifies trade-offs and offers practical, maintainable choices that remain valid for modern Python.

For lightweight in-memory uniqueness: the built-in hash-based containers remain the simplest and fastest for membership/insertion. In modern Python (3.7+) dict preserves insertion order if order matters. For memory-sensitive cases where identical string objects appear many times, sys.intern() can reduce duplication. Normalize incoming text (for example with unicodedata.normalize) before storing so canonically equivalent Unicode forms don’t slip through as distinct entries.

For durable, scalable storage: avoid ad-hoc tricks that inject variables into locals (the approach shown by ) — explicit serialization or a small database is safer. shelve gives a dict-like on-disk store for single-process use. For robust, concurrent, disk-backed uniqueness, use SQLite with a UNIQUE/PRIMARY KEY on the string column so the database enforces deduplication; an INSERT OR IGNORE pattern prevents duplicate errors and scales to large sets:

import sqlite3

conn = sqlite3.connect('strings.db')
cur = conn.cursor()
cur.execute('CREATE TABLE IF NOT EXISTS strings (value TEXT PRIMARY KEY)')
cur.executemany('INSERT OR IGNORE INTO strings(value) VALUES(?)', [(s,) for s in iterable_of_strings])
conn.commit()
conn.close()

When to consider other tools: Bloom filters give huge memory savings at the cost of false positives (good for pre-filtering). For batch jobs, external sort/uniq or database bulk-loads are straightforward. For interchange and long-term files prefer stable formats (JSON, SQLite) and explicit UTF-8 encoding; avoid marshal for long-term storage. Measure with simple profiling (time and memory) and pick the approach that matches scale and concurrency needs.

Relevant docs: sqlite3, shelve, sys.intern, unicodedata, Bloom filter overview.

Recommended Answers

All 5 Replies

Member Avatar for kudresov

If you do not have many strings, array of string should work. Normally you shouldn't worry about python efficency it has quite good code optimisation and python is not c it is not made for speed, so I would keep it simple and use something like array of strings.

Member Avatar for gunbuster363

but I want distinct strings.........and I don't want to loop it over to see if there is a existed string of the same.

Member Avatar for Gribouillis

but I want distinct strings.........and I don't want to loop it over to see if there is a existed string of the same.

You can use a set of strings

s = set(["hello", "world"])
s.add("anotherstring")
s.add("hello")
print s

or a dictionary (dict). Dict is the name of 'hash table' in python.

Member Avatar for vegaseat

Just for the fun of Python ...

# using the Python local dictionary vars()
# you can dump/save and load mystr_dict with module pickle

def mystrings():
    s1 = 'I '
    s2 = 'like '
    s3 = 'Python'
    # in this case vars() is local to this function
    return vars()

mystr_dict = mystrings()

print(mystr_dict)  # {'s3': 'Python', 's2': 'like', 's1': 'I'}

# add mystr_dict to vars() local to __main__
vars = vars().update(mystr_dict)

print(s1+s2+s3)  # I like Python
commented: Nice, I never thought about returning vars() :) +3
Member Avatar for lrh9

Gribouillis makes some good suggestions.

As for storing the data to a file there are really two standard, commonplace options that I think are suited to this.

I'd either pickle or marshal the object.

pickle and marshal can store and retrieve a Python object on file.

Both are similar, but pickle creates a file that will work across platforms and marshal creates a file that will only work on a type of system it is created on.

I read another post you have, and if structured text is necessary I'd try to devise a solution using json, but I'm not familiar with it so I can't help you there.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.