Member Avatar for Tommymac501

This code worked fine under 2.6:

f = open("v20100515.csv",'rt')
r = csv.reader(f)

try:
    r.next()  # skip the header record
    for row in r:
        n = 1
        for col in row:
            print str(n) + ": " + col
            n += 1

......

But since I upgraded to 3.1, I modified it to say:

f = open("v20100515.csv",'r')
r = csv.reader(f)

try:
    r.next()  # skip the header record
    for row in r:
        n = 1
        for col in row:
            print (str(n) + ": " + col)  #new print method in 3.1
            n += 1

......

When I try and read a row ( r.next() ), I get:

Message File Name Line Position
Traceback
<module> C:\Users\Dads_Laptop\Documents\Downloads\Toms_ 90
TypeError: must be bytes or buffer, not str

Also, when I tried to run the code unchanged (except for adjusting the print routines), I was getting Error: 'row' undefined. ????
What changed??

Edited by Tommymac501 because: n/a
  • 4 Contributors
  • 15 Replies
  • 2K Views
  • 5 Days Discussion Span
  • Latest Post Latest Post by Tommymac501

Dani AI

Generated

Two things changed from 2.x to 3.x that explain the errors you are seeing. First, the csv module in Python 3 expects text strings, not bytes, so opening files in binary mode ('rb') will trigger "iterator should return strings, not bytes." Open in text mode and let csv handle newlines. Second, iterator.next() was removed in Python 3; use the built-in next(iterator) instead. (docs.python.org)

A small, 3.x-friendly pattern that mirrors what suggested and avoids the blank-line/CRLF pitfalls bumped into:

import csv

# read
with open('v20100515.csv', 'r', encoding='utf-8', newline='') as f:
    reader = csv.reader(f)
    next(reader, None)              # skip header safely
    for row in reader:
        for i, col in enumerate(row, 1):
            print(str(i) + ': ' + col)

Key points:

  • newline='' disables the text layer’s newline translation so the csv module can do its own universal newline handling. Without it, embedded newlines in quoted fields may be misread and on Windows you can get stray '\r' characters. (docs.python.org)
  • Use next(reader, None) rather than reader.next(). This is the Python 3 iterator protocol (PEP 3114). (peps.python.org)

If you need to copy rows to a temp file without adding extra blank lines (your CRCRLF case), keep text mode, still use newline='', and control the writer’s line endings explicitly:

import csv

with open('TEST.csv', 'r', encoding='utf-8', newline='') as src, \
     open('temp1.tmp', 'w', encoding='utf-8', newline='') as dst:
    r = csv.reader(src)
    w = csv.writer(dst, lineterminator='\n')   # keep Unix LF if desired
    next(r, None)                              # skip header if present
    for row in r:
        w.writerow(row)

The lineterminator parameter lets you choose '\n' or '\r\n' explicitly, while newline='' ensures Python does not add its own translation. (docs.python.org)

Recommended Answers

All 15 Replies

Member Avatar for jcao219

Try changing 'r' to 'rb'.

Member Avatar for Tommymac501

Same thing:

TypeError: must be bytes or buffer, not str

Member Avatar for jcao219

Try this:

r = csv.reader(open("v20100515.csv", newline=''))
Member Avatar for Beat_Slayer

Have you tried like this?

r = csv.reader(open('v20100515.csv', 'rb'))

EDIT: Sorry i began the reply and went eat some sandwich, finnished when I came, and there were already all this ones posted, but I didn't saw them.

Edited by Beat_Slayer because: n/a
Member Avatar for Tommymac501

Have you tried like this?

r = csv.reader(open('v20100515.csv', 'rb'))

EDIT: Sorry i began the reply and went eat some sandwich, finnished when I came, and there were already all this ones posted, but I didn't saw them.

I tried the most simple;

r = csv.reader(open('v20100515.csv', 'rb'))
for row in r:
    print(row)

Fails with:


Message File Name Line Position
Traceback
<module> <module1> 3
Error: iterator should return strings, not bytes (did you open the file in text mode?)


Next, I tried a simple 'r' instead, and it read. Finally, I tried skiping a record with r.next() and got:


Message File Name Line Position
Traceback
<module> <module1> 3
AttributeError: '_csv.reader' object has no attribute 'next'

Yet the spec clearly states it does:
Reader objects (DictReader instances and objects returned by the reader() function) have the following public methods:

csvreader.next()¶
Return the next row of the reader’s iterable object as a list, parsed according to the current dialect.

I guess that changed for 3.1 also, I need to find the new spec....

Member Avatar for jcao219

This is the new spec:
http://docs.python.org/py3k/library/csv.html

Anyways, in Python 3 you don't do (object).next(), you do next(object).

From the docs you see:

The simplest example of reading a CSV file:

import csv
reader = csv.reader(open("some.csv", newline=''))
for row in reader:
    print(row)
Edited by jcao219 because: n/a
Member Avatar for vegaseat

Python2 seems to take care of empty lines
Python3 needs newline='\n', for instance ...
reader = csv.reader(open("some.csv", newline='\n'))

Member Avatar for jcao219

Python2 seems to take care of empty lines
Python3 needs newline='\n', for instance ...
reader = csv.reader(open("some.csv", newline='\n'))

I think the newline='' is required for csv in python 3.

Member Avatar for vegaseat

Let's put it to the test ...

# explore module csv with Python 3.1.2

import csv

fout = open('eggs.csv', 'w')
writer = csv.writer(fout, delimiter=';')
writer.writerow(['Spam'] * 5 + ['Baked Beans'])
writer.writerow(['Spam'] * 5 + ['Green Eggs'])
# needed for fin to work (or use with ...)
fout.close()

"""resulting file read with an editor (has 2 empty lines) -->
Spam;Spam;Spam;Spam;Spam;Baked Beans

Spam;Spam;Spam;Spam;Spam;Green Eggs

"""

# read the file back in
# Python2 seems to take care of empty lines
# Python3 needs newline='\n'
fin = open('eggs.csv', newline='\n')
reader = csv.reader(fin, delimiter=';')
# note that reader is a generator
for row in reader:
    print(row)
fin.close()

"""my result -->
['Spam', 'Spam', 'Spam', 'Spam', 'Spam', 'Baked Beans']
['Spam', 'Spam', 'Spam', 'Spam', 'Spam', 'Green Eggs']
"""

print('-'*60)

# test Python3 and newline=''
fin = open('eggs.csv', newline='')
reader = csv.reader(fin, delimiter=';')
# note that reader is a generator
for row in reader:
    print(row)
fin.close()

"""my result (empty line turns into empty list) -->
['Spam', 'Spam', 'Spam', 'Spam', 'Spam', 'Baked Beans']
[]
['Spam', 'Spam', 'Spam', 'Spam', 'Spam', 'Green Eggs']
[]
"""
Member Avatar for Tommymac501

Thanks for all of the input guys, boy, was this frustrating. I understand that languages need to evolve, but the concept behind Python (and Ruby) was to let the developer spend more time developing and less time worring about syntax and structure. I guess I'm just a crumudgen.. lol

Member Avatar for Tommymac501

ok, maybe it's me...

f = open('v20100515.csv', 'r', newline='\n')
Try:
    r = csv.reader(f, quoting = csv.QUOTE_ALL)
        for row in r:
        print (row[76][0:5])
        .......
except:
    print ("Unexpected error:", sys.exc_info()[0])

Produces the error: error: <class 'IndexError'> Implying that the data wasn't separated into columns?..

If I try:

f = open('v20100515.csv', 'rb', newline='\n')

It throws: ValueError: binary mode doesn't take a newline argument... I think I need to start over..


error: <class 'IndexError'>

Member Avatar for Tommymac501

This is making me insane...

Input data (in hex, end of line, start of new line):

'5F 43 4F 44 45 0A 22 52 ......' The record clearly ends with a newline.

Reading with 'r = csv.reader(open('TEST.csv', newline=''))' and writing with 'w = csv.writer(open('temp1.tmp', 'w'))' produces: '5F 43 4F 44 45 0D 0D 0A 22 52 ......' (2) Carraige returns and one line feed.

Reading with ''r = csv.reader(open('TEST.csv', newline='\n'))' produces the same result. Opening with 'rb' and skipping the 'newline =' (it's not allowed) causes an 'iterator should return strings, not bytes (did you open the file in text mode?)' Error.

All I want to do is open a file, read a row, and save it to a temp file.. ugh!!

Member Avatar for Tommymac501

It appears that my major issue is that the data comes from a Unix box, and has newlines as line ends. Opening in binary mode works, but, you cannot use the

for x in Y

because the data types dont work, and, you can't use the 'newline' in binary mode. If you use the default newline = '', or, newline = '\n', the existing newline is converted to a carraige return, and a CRLF is appended at write time, making the line end \n\r\n and causing space between rows.

Answer?, go back to 2.6, it all works just fine.. LOLOL

(There is a hot issue going on in the python bug reports on this issue actually, and I thought it was just me. http://bugs.python.org/issue5455)

Thanks everyone for you're help!!

Member Avatar for Tommymac501

Is it solved???

Near as can tell, they agreed to change the documentation to reflect the limitaiton, they didn't fix the problem.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.