Member Avatar for RogerI

I need to read in a large amount of data from a csv file into arrays. I am using Numpy as I need to later manipulate and plot the results. I have decided to use numpy.loadtxt as this appears to be the most efficient method to read in the data. I can read the data into one big string array but I am struggling to convert the data as I read it in.
Here is the code I am using...

# -*- coding: utf-8 -*-
"""
Created on Sat Nov 21 21:41:05 2009

@author: Roger Iles
"""
import numpy as np

test=np.loadtxt('data_dev_test.csv',delimiter=',', unpack=True,
                  dtype={'names':('localtime','localdate','elapsedtime','dose','units'),
                         'formats': ('s8','s7','s10','.3f','s2')})


print test

And the data has the format...

11:16:29, 24NOV00, 0000:00:00,1.087,nG
11:17:34, 24NOV00, 0000:01:04,1.087,nG

On running the code I get the following error...

Traceback (most recent call last):
File "C:\Python26\Lib\site-packages\xy\tepc.py", line 11, in <module>
'formats': ('s8','s7','s10','.4f','s2')})
File "C:\Python26\lib\site-packages\numpy\lib\io.py", line 443, in loadtxt
dtype = np.dtype(dtype)
TypeError: data type not understood

I'd really appreciate any assistance here as I am farily new to Python and do not know where to go next.

Many thanks in advance,

RogerI

  • 2 Contributors
  • 2 Replies
  • 2K Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by RogerI
Member Avatar for jwxie

I am also relatively new to python but I did something with datetime in the past.

import datetime
import time
m = datetime.datetime.now()
k = datetime.date.today()
b = k.isoweekday()
b = time.strftime("%A")
mm = m.month
yy = m.year
dd = m.day
ds = datetime.date(yy,mm,dd)
hh = m.hour
mi = m.minute
ts = datetime.time(hh,mi)
print "It is now",ts.strftime("%H:%M:%p"),b,ds.strftime("%m/%d/%Y")
Member Avatar for RogerI

Hi
Thanks for your advice. What I did in the end was to use the Python tabular module which interprets the data type automatically while reading.

Following that I will now progress with using some of your advice for dealling with the data / time conversion specifically.

Thanks
RogerI

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.