strange dictionary definition/object instance problem

[jmroach]

I am trying to add entries to a dictionary with the following loop:

for p in pressures:
	q70in.resetKeyword( "expansionRatio", p )
	q70in.writeInput( prefix + tempInput )  
	commands.getstatusoutput( runq70 )
	q70out = q70.Q70Output( prefix + outputFile )
	results[p] = {}
	#results[p]['Performance'] = q70out.performance
	results[p]['Conditions'] = q70out.conditions
	print q70out
	print results
	print ''

however, earlier entries of the dictionary are being 'overwritten' with new values as follows:

(here's the output from the code print statements)

<q70.Q70Output instance at 0x400cbccc>
{'2.0': {'Conditions': {'Vane2INLET': {'WG': '3.200'}, 'Blade1INLET': {'WG': '3.191'}, 'Blade2INLET': {'WG': '3.200'}, 'Vane1INLET': {'WG': '3.191'}}}}

<q70.Q70Output instance at 0x400cbccc>
{'2.0': {'Conditions': {'Vane2INLET': {'WG': '3.320'}, 'Blade1INLET': {'WG': '3.311'}, 'Blade2INLET': {'WG': '3.320'}, 'Vane1INLET': {'WG': '3.311'}}}, '3.0': {'Conditions': {'Vane2INLET': {'WG': '3.320'}, 'Blade1INLET': {'WG': '3.311'}, 'Blade2INLET': {'WG': '3.320'}, 'Vane1INLET': {'WG': '3.311'}}}}

<q70.Q70Output instance at 0x400cbccc>
{'2.0': {'Conditions': {'Vane2INLET': {'WG': '3.326'}, 'Blade1INLET': {'WG': '3.316'}, 'Blade2INLET': {'WG': '3.326'}, 'Vane1INLET': {'WG': '3.316'}}}, '3.5': {'Conditions': {'Vane2INLET': {'WG': '3.326'}, 'Blade1INLET': {'WG': '3.316'}, 'Blade2INLET': {'WG': '3.326'}, 'Vane1INLET': {'WG': '3.316'}}}, '3.0': {'Conditions': {'Vane2INLET': {'WG': '3.326'}, 'Blade1INLET': {'WG': '3.316'}, 'Blade2INLET': {'WG': '3.326'}, 'Vane1INLET': {'WG': '3.316'}}}}

So what's happening?

Note that #### in:

results{ '2.0':{'Conditions':{'Vane2INLET':{'WG': ####}}}}

SHOULD NOT CHANGE WITH ITERATION based on the loop

[woooee]

New entries are not being over-written. You are initializing to an empty dictionary with the statement results[p] = {}. Perhaps (and just perhaps) a dictionary of classes would serve better. Note that SQLite can also be used in memory, i.e you can store and lookup '2.0' + 'Conditions' + 'Vane2INLET', etc. or however you want to store it. Either would be less confusing IMHO.

class Class1 :
   def __init__ (self) :
      self.field1 = ""
      self.field2 = 0
      self.field3 = "***"

#====================================================
sample_dic = {}
C = Class1()
C.field1 = "Test1"
C.field2 = 1
C.field3 = "Test1A"
sample_dic["test1"] = C

C = Class1()
C.field1 = "Test2"
C.field2 = 2
C.field3 = "Test2A"
sample_dic["test2"] = C

C = Class1()
sample_dic["test3"] = C

##   note that dictionary keys are not in any order
for key in sample_dic.keys() :
   print key, sample_dic[key].field1, sample_dic[key].field2, \
                 sample_dic[key].field3

sample_dic["test3"].field2=3
print "updated field2 =", sample_dic["test3"].field2

[jmroach]

i'll definitely give the dictionary of classes a try, but i still think i might be missing something...

'results[p] = {}' should only initialize that key's value since every value of 'p' is unique. for example:

dict = {}
dict[0] = 0
dict[1] = {}
dict[2] = 2
print dict

{0: 0, 1: {}, 2: 2}

[woooee]

i'll definitely give the dictionary of classes a try, but i still think i might be missing something...

'results[p] = {}' should only initialize that key's value since every value of 'p' is unique. for example:

dict = {}
dict[0] = 0
dict[1] = {}
dict[2] = 2
print dict

{0: 0, 1: {}, 2: 2}

Either results[p] is unique or it's being overwritten/initialized to an empty dictionary. I don't know which it is without being able to see more data. Add a print statement so you can see what p is in every loop.

[jmroach]

i just tried the dictionary of classes approach and it didn't work... I also added some more printing. Here's the snippet and output:

for p in pressures:
	q70in.resetKeyword( "expansionRatio", float(p) )
	q70in.writeInput( prefix + tempInput )  
	commands.getstatusoutput( runq70 )
	q70out = q70.Q70Output( prefix + outputFile )
	#results[p] = {}
	#results[p]['Performance'] = q70out.performance
	#results[p]['Conditions'] = q70out.conditions
	results[p] = indyResult()
	results[p].performance = q70out.performance
	results[p].conditions = q70out.conditions
	print type(p), len(p), p
	for i in results:
		print i
		print results[i].performance

<type 'str'> 3 2.0
2.0
{'Stage1': {'REAC-PS': '0.509', 'WR(T)/P': '4.3010', 'TT0': '1602.30', 'REAC-W': '0.515', 'W0': '3.191', '0WR(THCR)E/D': '2.8191', 'PT0': '29.7010'}, 'Stage2': {'REAC-PS': '0.488', 'WR(T)/P': '6.0074', 'TT0': '1470.43', 'REAC-W': '0.500', 'W0': '3.200', '0WR(THCR)E/D': '3.9309', 'PT0': '20.4275'}}
<type 'str'> 3 3.0
2.0
{'Stage1': {'REAC-PS': '0.524', 'WR(T)/P': '4.4623', 'TT0': '1602.30', 'REAC-W': '0.560', 'W0': '3.311', '0WR(THCR)E/D': '2.9243', 'PT0': '29.7010'}, 'Stage2': {'REAC-PS': '0.579', 'WR(T)/P': '6.6682', 'TT0': '1444.78', 'REAC-W': '0.700', 'W0': '3.320', '0WR(THCR)E/D': '4.3593', 'PT0': '18.9262'}}
3.0
{'Stage1': {'REAC-PS': '0.524', 'WR(T)/P': '4.4623', 'TT0': '1602.30', 'REAC-W': '0.560', 'W0': '3.311', '0WR(THCR)E/D': '2.9243', 'PT0': '29.7010'}, 'Stage2': {'REAC-PS': '0.579', 'WR(T)/P': '6.6682', 'TT0': '1444.78', 'REAC-W': '0.700', 'W0': '3.320', '0WR(THCR)E/D': '4.3593', 'PT0': '18.9262'}}
<type 'str'> 3 3.5
2.0
{'Stage1': {'REAC-PS': '0.525', 'WR(T)/P': '4.4695', 'TT0': '1602.30', 'REAC-W': '0.563', 'W0': '3.316', '0WR(THCR)E/D': '2.9290', 'PT0': '29.7010'}, 'Stage2': {'REAC-PS': '0.633', 'WR(T)/P': '6.7033', 'TT0': '1443.42', 'REAC-W': '0.842', 'W0': '3.326', '0WR(THCR)E/D': '4.3819', 'PT0': '18.8486'}}
3.5
{'Stage1': {'REAC-PS': '0.525', 'WR(T)/P': '4.4695', 'TT0': '1602.30', 'REAC-W': '0.563', 'W0': '3.316', '0WR(THCR)E/D': '2.9290', 'PT0': '29.7010'}, 'Stage2': {'REAC-PS': '0.633', 'WR(T)/P': '6.7033', 'TT0': '1443.42', 'REAC-W': '0.842', 'W0': '3.326', '0WR(THCR)E/D': '4.3819', 'PT0': '18.8486'}}
3.0
{'Stage1': {'REAC-PS': '0.525', 'WR(T)/P': '4.4695', 'TT0': '1602.30', 'REAC-W': '0.563', 'W0': '3.316', '0WR(THCR)E/D': '2.9290', 'PT0': '29.7010'}, 'Stage2': {'REAC-PS': '0.633', 'WR(T)/P': '6.7033', 'TT0': '1443.42', 'REAC-W': '0.842', 'W0': '3.326', '0WR(THCR)E/D': '4.3819', 'PT0': '18.8486'}}

[jrcagle]

What's happening is that dictionaries, like lists, are mutable objects. Furthermore, they are assigned by reference and not by value. So under certain conditions, you might think you are changing one mutable object when in fact you are changing all of the ones that are referenced:

>>> a = []
>>> b = a  # b and a point to the same list now.
>>> b.append(1)
>>> b
[1]
>>> a
[1]
>>>

Now as far as I can tell, the issue here is subtle. These two functions illustrate the problem:

>>> def test_func():
	results = {}
	for x in range(4):
		results[x] = {}
		results[x]["conditions"] = x
	print results

	
>>> test_func()
{0: {'conditions': 0}, 1: {'conditions': 1}, 2: {'conditions': 2}, 3: {'conditions': 3}}
>>> 
>>> def test_func2():
	results = {}
	mydict = {}
	for x in range(4):
		results[x] = mydict
		results[x]["conditions"] = x
	print results

	
>>> test_func2()
{0: {'conditions': 3}, 1: {'conditions': 3}, 2: {'conditions': 3}, 3: {'conditions': 3}}
>>>

The difference between the first version and the second is that mydict is getting clobbered successively. I *think* that's what's happening with q70out.conditions. I suspect there's a bug in the code for q70.Q70Output( prefix + outputFile ) that returns a mutable object that's getting clobbered every time.

I whole-heartedly agree with wooeee ... your data structure would be a whole lot happier as a class object with members called .conditions and .performance.

Jeff

[woooee]

This is a very simple example of how you could use SQLite in memory. This may be a case of the difficult way actually being the simpliest. Note that this is just a simple/quick example. There is a lot more on the web. If this is of some help you can thank The Blues. I was looking for something that I could do while listening to Bad Dog Blues.

import sqlite3 as sqlite

##----------------------------------------------------------------------
def add_rec(cur, con, add_tuple):
   print "add_rec", len(add_tuple)
   cur.execute("insert into test_it values (?, ?, ?, ?, ?, ?, ?, ?, ?)", add_tuple)
   con.commit()

##----------------------------------------------------------------------
def print_all_recs(cur):
   # Execute the SELECT statement:
   print "Printing all recs"
   cur.execute("select * from test_it")

   # Retrieve all rows as a sequence and print that sequence:
   recs_list = cur.fetchall()
   for rec in recs_list:
      print rec
      print "     stage, REAC_W =", rec[1], rec[5]

##----------------------------------------------------------------------
def add_test_data(cur, con):
   #----->              Stage     REAC_PS  WR(T)/P    TTO       REAC_W    W0       0wR(THCR)E/D  PTO
   data_list=[ ('2.0', 'Stage1', '0.509',  '4.3010', '1602.30', '0.515', '3.191',   '2.8191',   '29.7010'), \
               ('2.0', 'Stage2', '0.488',  '6.0074', '1470.43', '0.500', '3.200',   '3.9309',   '20.4275'), \
               ('2.0', 'Stage1', '0.524',  '4.4623', '1602.30', '0.560', '3.311',   '2.9243',   '29.7010'), \
               ('2.0', 'Stage2', '0.579',  '6.6682', '1444.78', '0.700', '3.320',   '4.3593',   '18.9262'), \
               ('3.0', 'Stage1', '0.524',  '4.4623', '1602.30', '0.560', '3.311',   '2.9243',   '29.7010'), \
               ('3.0', 'Stage2', '0.579',  '6.6682', '1444.78', '0.700', '3.320',   '4.3593',   '18.9262'), \
               ('3.5', 'Stage1', '0.525',  '4.4695', '1602.30', '0.563', '3.316',   '2.9290',   '29.7010') ]

   for data_tuple in data_list :
      add_rec(cur, con, data_tuple)

##----------------------------------------------------------------------
if __name__ == "__main__":
   # Create a connection to the (memory) database file
   con = sqlite.connect(':memory:')

   # Get a Cursor object that operates in the context of Connection con
   cur = con.cursor()

   cur.execute("CREATE TABLE test_it (number varchar, stage varchar, REAC_PS varchar, WR_T_P varchar, TTO varchar, REAC_W varchar, W0 varchar, wR_THCR_E_D varchar, PTO varchar)")

   add_test_data(cur, con)
   print_all_recs(cur)
   
   ##----------------------------------------------------------------------
   print '\n-----SELECT * FROM test_it where number="2.0"-------'
   recs_list=cur.execute('SELECT * FROM test_it where number="2.0"')
   ctr=0
   for row in recs_list:
      print row
      ctr += 1
   print ctr, "recs found"
   
   print '\n-----SELECT * FROM test_it where number="2.0" and stage="Stage1"'
   lookup_dic={"dic_num":"2.0", "dic_st":"Stage1"}
   recs_list=cur.execute('SELECT * FROM test_it where number=:dic_num and stage=:dic_st', \
	                  lookup_dic)
   ctr=0
   for row in recs_list:
      print row
      ctr += 1
   print ctr, "recs found"

[jmroach]

Thanks a ton for the help and great example, woooee & jrcagle! i did a little google'ing since i have absolutely no experience with sql, but it does seem like a really good fit for this type of application. i'll give it a try this morning and see if i can't get things working. i'll post the resolution upon completion.

-john