CSV Reader Recording Entries

Question

rhuffman8 0 Light Poster

13 Years Ago

I am working with a CSV file with multiple rows, which I need to be able to access individual data elements from a row so I can assign them to variables and send them to the other program I am working with in a loop, going through each individual row.

I was previously working on a program originally written by another developer which used this

for i, record in enumerate(reader):
        if i > 1:  # dump first two
            blah1, blah2, blah3, blah4, blah5, blah6, blah7, \
                blah8, blah9, blah10, blah11, blah12, blah13, blah14, \
                blah14, blah15, blah16, blah17, blah18= record

to take the rows out and collect the individual data items from each. This all worked fine and I thought my understanding of the concept was enough that I could duplicate it in a similar program I am now writing from scratch.
My code for this section in the new program is as follows:

for i, record in enumerate(reader):
        if i > 0: #skip headers
            date, hk, ba, bsp, hgch, v995, v996, vct = record

When I run the code in cmd, I get an error message that reads

ValueError: unpack list of wrong size

I have looked into this briefly but haven't had much luck in applying solutions to my own problem.

If anyone can offer any insight to either reading csv files or the trouble with my code, that would be greatly appreciated.

python

3 Contributors
8 Replies
368 Views
2 Weeks Discussion Span
Latest Post 13 Years Ago Latest Post by rhuffman8

Gribouillis 1,391 Programming Explorer

13 Years Ago

The error means that one of the records does not have length 8. The first solution is to print it to see what happens

from itertools import islice

for i, record in islice(enumerate(reader), 1, None):
    if len(record) != 8:
        raise ValueError, "Bad record at line %d : %s" % (i+1, str(record))
    else:
        date, hk, ba, bsp, hgch, v995, v996, vct = record

woooee 814 Nearly a Posting Maven

13 Years Ago

You should also be able to use the list as is, something like this:

for i, record in enumerate(reader):
    if (i > 0) and (len(record) > 7): #skip headers & record has all the fields
        print "date =", record[0]
        print "hk   =", record[1]
        print "ba   =", record[2]
        ## etc.

Edited 13 Years Ago by woooee because: n/a

Reply to this topic

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.

rhuffman8 0 Light Poster · Answer 1 · 2011-07-14T20:25:46+00:00

I tried both solutions and didn't seem to have a problem with either one when running the commands in a command line but the variables still don't seem to be sending over properly. Perhaps the error is somewhere else in my code?
The function is supposed to take each line the the csv file and first decide if the first entry is a date and if the date is for the present month (these two functions are laid out elsewhere in the python file). Then, it checks if the date of the entry is within the last week. Depending on how far into the past week, it assigns the entries for the line to variables shared with another program and then should go through the same process for all the other rows in the csv file.
Here is the whole function definition:

def send_data():
    for i, record in enumerate(reader):
        if (i > 0) and (len(record) > 7): #skip headers
            date = record[0]
            hk = record[1]
            ba = record[2]
            bsp = record[3]
            hgch = record[4]
            v995 = record[5]
            v996 = record[6]
            vct = record[7]
            month, day, year = [int(x) for x in date.split('/')]
            evt_date = dt.datetime(year, month, day)
            if (event(date)) and (curr_month(date)):
                if evt_date.day >= curr.day - 6:
                    if evt_date.day == curr.day - 6:
                        svars.date1 = date
                        svars.hk1 = hk
                        svars.ba1 = ba
                        svars.bsp1 = bsp
                        svars.hgch1 = hgch
                        svars.v9951 = v995
                        svars.v9961 = v996
                        svars.vct1 = v995 + v996
                    elif evt_date.day == curr.day - 5:
                        svars.date2 = date
                        svars.hk2 = hk
                        svars.ba2 = ba
                        svars.bsp2 = bsp
                        svars.hgch2 = hgch
                        svars.v9952 = v995
                        svars.v9962 = v996
                        svars.vct2 = v995 + v996
                    elif evt_date.day == curr.day - 4:
                        svars.date3 = date
                        svars.hk3 = hk
                        svars.ba3 = ba
                        svars.bsp3 = bsp
                        svars.hgch3 = hgch
                        svars.v9953 = v995
                        svars.v9963 = v996
                        svars.vct3 = v995 + v996
                    elif evt_date.day == curr.day - 3:
                        svars.date4 = date
                        svars.hk4 = hk
                        svars.ba4 = ba
                        svars.bsp4 = bsp
                        svars.hgch4 = hgch
                        svars.v9954 = v995
                        svars.v9964 = v996
                        svars.vct4 = v995 + v996
                    elif evt_date.day == curr.day - 2:
                        svars.date5 = date
                        svars.hk5 = hk
                        svars.ba5 = ba
                        svars.bsp5 = bsp
                        svars.hgch5 = hgch
                        svars.v9955 = v995
                        svars.v9965 = v996
                        svars.vct5 = v995 + v996
                    elif evt_date.day == curr.day - 1:
                        svars.date6 = date
                        svars.hk6 = hk
                        svars.ba6 = ba
                        svars.bsp6 = bsp
                        svars.hgch6 = hgch
                        svars.v9956 = v995
                        svars.v9966 = v996
                        svars.vct6 = v995 + v996
                    elif evt_date.day == curr.day:
                        svars.date7 = date
                        svars.hk7 = hk
                        svars.ba7 = ba
                        svars.bsp7 = bsp
                        svars.hgch7 = hgch
                        svars.v9957 = v995
                        svars.v9967 = v996
                        svars.vct7 = v995 + v996						
                else:
                    pass
            elif not event(date):
                svars.hk_goal = hk
                svars.ba_goal = ba
                svars.bsp_goal = bsp
                svars.hgch_goal = hgch
                svars.v995_goal = v995
                svars.v996_goal = v996
                svars.vct_goal = vct
            else:
                pass

(if there is a more efficient/better way to do any of that, that would be great too...I'm still fairly new to python)

Gribouillis 1,391 Programming Explorer Team Colleague · Answer 2 · 2011-07-14T21:43:44+00:00

I tried both solutions and didn't seem to have a problem with either one when running the commands in a command line but the variables still don't seem to be sending over properly. Perhaps the error is somewhere else in my code?
The function is supposed to take each line the the csv file and first decide if the first entry is a date and if the date is for the present month (these two functions are laid out elsewhere in the python file). Then, it checks if the date of the entry is within the last week. Depending on how far into the past week, it assigns the entries for the line to variables shared with another program and then should go through the same process for all the other rows in the csv file.
Here is the whole function definition:

def send_data():
    for i, record in enumerate(reader):
        if (i > 0) and (len(record) > 7): #skip headers
            date = record[0]
            hk = record[1]
            ba = record[2]
            bsp = record[3]
            hgch = record[4]
            v995 = record[5]
            v996 = record[6]
            vct = record[7]
            month, day, year = [int(x) for x in date.split('/')]
            evt_date = dt.datetime(year, month, day)
            if (event(date)) and (curr_month(date)):
                if evt_date.day >= curr.day - 6:
                    if evt_date.day == curr.day - 6:
                        svars.date1 = date
                        svars.hk1 = hk
                        svars.ba1 = ba
                        svars.bsp1 = bsp
                        svars.hgch1 = hgch
                        svars.v9951 = v995
                        svars.v9961 = v996
                        svars.vct1 = v995 + v996
                    elif evt_date.day == curr.day - 5:
                        svars.date2 = date
                        svars.hk2 = hk
                        svars.ba2 = ba
                        svars.bsp2 = bsp
                        svars.hgch2 = hgch
                        svars.v9952 = v995
                        svars.v9962 = v996
                        svars.vct2 = v995 + v996
                    elif evt_date.day == curr.day - 4:
                        svars.date3 = date
                        svars.hk3 = hk
                        svars.ba3 = ba
                        svars.bsp3 = bsp
                        svars.hgch3 = hgch
                        svars.v9953 = v995
                        svars.v9963 = v996
                        svars.vct3 = v995 + v996
                    elif evt_date.day == curr.day - 3:
                        svars.date4 = date
                        svars.hk4 = hk
                        svars.ba4 = ba
                        svars.bsp4 = bsp
                        svars.hgch4 = hgch
                        svars.v9954 = v995
                        svars.v9964 = v996
                        svars.vct4 = v995 + v996
                    elif evt_date.day == curr.day - 2:
                        svars.date5 = date
                        svars.hk5 = hk
                        svars.ba5 = ba
                        svars.bsp5 = bsp
                        svars.hgch5 = hgch
                        svars.v9955 = v995
                        svars.v9965 = v996
                        svars.vct5 = v995 + v996
                    elif evt_date.day == curr.day - 1:
                        svars.date6 = date
                        svars.hk6 = hk
                        svars.ba6 = ba
                        svars.bsp6 = bsp
                        svars.hgch6 = hgch
                        svars.v9956 = v995
                        svars.v9966 = v996
                        svars.vct6 = v995 + v996
                    elif evt_date.day == curr.day:
                        svars.date7 = date
                        svars.hk7 = hk
                        svars.ba7 = ba
                        svars.bsp7 = bsp
                        svars.hgch7 = hgch
                        svars.v9957 = v995
                        svars.v9967 = v996
                        svars.vct7 = v995 + v996						
                else:
                    pass
            elif not event(date):
                svars.hk_goal = hk
                svars.ba_goal = ba
                svars.bsp_goal = bsp
                svars.hgch_goal = hgch
                svars.v995_goal = v995
                svars.v996_goal = v996
                svars.vct_goal = vct
            else:
                pass

(if there is a more efficient/better way to do any of that, that would be great too...I'm still fairly new to python)

You can shorten the function like this

def set_fields(svars, suffix, **kwd):
    "helper function for send_data"
    for key, value in kwd.items():
        setattr(svars, key + suffix, value)

def send_data():
    for i, record in enumerate(reader):
        if (i > 0) and (len(record) > 7): #skip headers
            date, hk, ba, bsp, hgch, v995, v996, vct = record[:8]
            month, day, year = [int(x) for x in date.split('/')]
            evt_date = dt.datetime(year, month, day)
            if (event(date)) and (curr_month(date)):
                if evt_date.day >= curr.day - 6:
                    k = curr.day - evt_date.day
                    assert 0 <= k <= 6
                    set_fields(svars, str(7-k),
                        date = date, hk = hk, ba = ba, bsp = bsp, hgch = hgch,
                        v995 = v995, v996 = v996, vct = v995 + v996)
            elif not event(date):
                set_fields(svars, "_goal",
                    hk = hk, ba = ba, bsp = bsp, hgch = hgch,
                    v995 = v995, v996 = v996, vct = vct)

You say that the variables still dont send properly, but you don't describe where the problem is, so it's difficult to help ...

rhuffman8 0 Light Poster · Answer 3 · 2011-07-14T21:53:05+00:00

Thank you for the suggestion on shortening the code.
I understand though. I'm having problems figuring out where the problem is exactly. I was just trying to see if anything specific stood out in the code that was obviously wrong. I will keep looking into it and if I find anything specific, I'll post that.

rhuffman8 0 Light Poster · Answer 4 · 2011-07-15T20:49:00+00:00

So, after trying a few things, I'm starting to think there is something wrong with my command to open the csv file

reader = csv.reader(open('C:/Documents and Settings/VPC.VPC-SCALA.000/Desktop/RAH 7-11-11/RH/Projected Cash Goals Template.csv', 'rb'))

but from past experience, I cannot figure out what it would be. I commented out the rest of the program so I can go through line by line and figure out what's wrong and put in a statement to test whether things were working or not. The statement does not include the file opened so it should not really be affected but when the test statement is above the open command, everything works fine. When I move it below the open command, the test statement no longer works at all. I get no errors while compiling and when I run the same commands in a Python command line, I can get everything to work just fine. I'm running out of ideas and things to try.
(I'm working in Windows, by the way, in case that makes any difference to anything)

woooee 814 Nearly a Posting Maven · Answer 5 · 2011-07-15T23:14:47+00:00

Are there any records in the file? Create a test file so you know what it contains and try it with the program. If the test file works, the problem is the csv file, not the program.

rhuffman8 0 Light Poster · Answer 6 · 2011-07-30T02:10:58+00:00

Turned out there was a problem with the file location. Just a big ole' mistake on my part.