Choosing the right data structure

[group256]

Dear everyone,

I'm trying to implement something like gmail conversational view and the way I've designed it is that I store "subject" of the mail + other parties email address (in case you send, other party is recipient, and in case of receiving, sender is the other party).

What I do is, I store these data in a file using a dictionary and cPickle, and the key is as I explained above, "subject + other party". But I believe this key is not efficient enough as I need to do search of the email address of the party in the key.

Is there any suggestion how I could approach this and what the proper data structure for storing list of addresses and subject is??

[TrustyTony]

Maybe you could check how they did it http://libgmail.sourceforge.net/

[griswolf]

There is nothing stopping you from using either multiple dictionaries or a larger dictionary with a variety of keys. For instance, I am now working on a project where I have a dictionary like this

{
  offset: date, #offset is an integer
  date: filename, #date is datetime.date
  filename: lines-from-the-file, #filename is a string
}

Where most of the time, there are about 30 to 100 dates, about the same number of offsets and one to three filenames, each with perhaps 300 lines.

Holding multiple (kinds of) keys in your dict(s) allows you to avoid secondary searching (costs time) at the cost of holding more data in memory (costs space).

[TrustyTony]

offset: date # date is integer?

[group256]

Thank you so much for your answers.

But please keep in mind that time is a huge factor in this project. Nested dictionaries brings up the fear in me that I may be wasting so much CPU/IO time to achieve my goal.

[TrustyTony]

griswolfs solution brings just benefits in running time. What you should be conserned if you choose that way is to plan , code and test carefully access functions for delete/insert to keep everything in sync. OO solution could make sense for encapsulation.

[griswolf]

As soon as the word 'huge' enters the discussion, it is time to start thinking about a database solution. Not that you will actually end up using one, but time to start thinking. In this case, the desire to have multiple keys into the data makes the idea of a database more persuasive.

And always remember the three laws of optimization:

1. Don't do it
2. Measure first
3. If your code is a hotspot, look for a different algorithm first.

[group256]

griswolf, you're absolutely right. I may need to move to a database, but certain conditions will not allow such transition at the moment.

1- I can't "not to do it". It needs to be done.
2- This is probably the most practical thing I can do.
3- That's why I posted here to find a better solution. I came up with some other ideas but they are not any better than the original.

Anyhow, at the moment, I will concentrate on just using the subject. Maybe down the road, something easier would pop in my mind to simplify the storage.

[griswolf]

Mm. How 'bout an embedded Database? I've used SQLite with only marginal success (but: I was trying to do something that would have been very hard for any database to handle; and I was doing it from Java). The general consensus seems to favor SQLite. I have also looked into using the Berkeley DB which seemed very good when I looked (10 years ago??).

[group256]

Thanks a lot man. I will take a look into these mentioned by you. A SQLite eems interesting to me.