python/perl log file script

[sparso]

I am in need of a script that reads a log file, takes some data from the file and saves it in a .csv (coma separated value) format. The example log is below:

log.txt:

Date: 3/14/09

Device #: 1
Test: Cont: pass
Test: Leakage: pass
Test: INL: 0.88
Test: DNL: 0.56

Device #: 2
Test: Cont: pass
Test: Leakage: pass
Test: INL: 0.76
Test: DNL: 0.44

Device #: 3
Test: Cont: pass
Test: Leakage: pass
Test: INL: 1.2
Test: DNL: 0.84

format for the .csv file:
log.csv:

device, INL, DNL
1,0.88,0.56
2,0.76,0.44
3,1.2,0.84

Any help will be greatly appreciated.

[Gribouillis]

Here is a script

class Device(object):
  def __init__(self, number):
    self.number = number
    self.items = {}

def getDevices(logPath):
  source = (line.strip() for line in open(logPath))
  device_list = []
  prefix_dev = "Device #: "
  prefix_test = "Test: "
  try:
    while True:
      while True:
        line = source.next()
        if line.startswith(prefix_dev):
          device = Device(line[len(prefix_dev):].strip())
          device_list.append(device)
          break
      # we have a device
      while True:
        line = source.next()
        if line.startswith(prefix_test):
          t = tuple(s.strip() for s in line[len(prefix_test):].split(":"))
          if len(t) != 2:
            raise Exception("unexpected line: '%s'" % line)
          device.items[t[0]] = t[1]
        else:
          break
  except StopIteration:
    pass
  return device_list 

def log2csv(logPath, csvPath):
  device_list = getDevices(logPath)
  out = open(csvPath, "w")
  try:
    out.write("Device, INL, DNL\n")
    for dev in device_list:
      n, inl, dnl = dev.number, dev.items["INL"], dev.items["DNL"]
      out.write(",".join((n, inl, dnl)))
      out.write("\n")
  finally:
    out.close()

if __name__ == "__main__":
  log2csv("log.txt", "csv.txt")

[sparso]

Thank you very much for the script. As a newbie to python I am having some issue with it. I typed it in and ran it using python 3.0 idle and got the following errors:

>>>
Traceback (most recent call last):
File "C:\Python30\work\datalog.py", line 46, in <module>
log2csv("log.txt", "csv.txt")
File "C:\Python30\work\datalog.py", line 34, in log2csv
device_list = getDevices(logPath)
File "C:\Python30\work\datalog.py", line 14, in getDevices
line = source.next()
AttributeError: 'generator' object has no attribute 'next'
>>>
Do I have to add some import modules to run this?

Thanks again.

[Gribouillis]

With python 3.0, replace .next() , by .send(None) , sorry :)
In fact, here is a slightly modified version (which doesn't suppose that there is a blank line between devices)

# use with python >= 3.0
class Device(object):
  def __init__(self, number):
    self.number = number
    self.items = {}

def getDevices(logPath):
  source = (line.strip() for line in open(logPath))
  device_list = []
  prefix_dev = "Device #: "
  prefix_test = "Test: "
  try:
    line = source.send(None)
    while True:
      while True:
        if line.startswith(prefix_dev):
          device = Device(line[len(prefix_dev):].strip())
          device_list.append(device)
          break
        else:
          line = source.send(None)
      # we have a device
      while True:
        line = source.send(None)
        if line.startswith(prefix_test):
          t = tuple(s.strip() for s in line[len(prefix_test):].split(":"))
          if len(t) != 2:
            raise Exception("unexpected line: '%s'" % line)
          device.items[t[0]] = t[1]
        else:
          break
  except StopIteration:
    pass
  return device_list 

def log2csv(logPath, csvPath):
  device_list = getDevices(logPath)
  with open(csvPath, "w") as out:
    out.write("Device, INL, DNL\n")
    for dev in device_list:
      n, inl, dnl = dev.number, dev.items["INL"], dev.items["DNL"]
      out.write(",".join((n, inl, dnl)))
      out.write("\n")

if __name__ == "__main__":
  log2csv("log.txt", "csv.txt")

[sparso]

Brilliant! That works!. Thank you very much.

[sparso]

Gribouillis,

Thanks for the help ...unfortunately they changed the log file a bit and I was trying to modify your code over the weekend and had no luck. The error messages are still too cryptic for me. Hopefully by looking at the two different code I would be able to fly myself in the future.

The log file now looks like this:

log.txt:

Date: 3/14/09

Device #: 1
Test: Cont: pass
Test: Leakage: pass
Bin: 1 ADC Result: (0.88, 0.56)

Device #: 2
Test: Cont: pass
Test: Leakage: pass
Bin: 24 ADC Result: (1.88, 0.36)

Device #: 3
Test: Cont: pass
Test: Leakage: pass
Bin: 16 ADC Result: (0.68, 1.6)

and the output file looks like this:

Device, INL, DNL, Bin
1, 0.88, 0.56, 1
2, 1.88, 0.36, 24
3, 0.68, 1.6, 16

Thank you very much for your help.

[Gribouillis]

Ok, here is a new version. If you're completely new to python, there are many subtle details. I didn't put many comments, because I don't know what you will understand or not in the code. Note that this version won't work with the previous format of your log files.

# python 3.0
import re

class Device(object):
  def __init__(self, identity):
    self.identity = identity
  def doTest(self, args):
    #print "Test", args
    pass
  def doBin(self, args):
    #print "Bin", args
    self.binNumber = args[0].split()[0]
    self.inldnl = args[1][1:-1]

def genDevices(logPath):
  content = open(logPath).read()
  pattern = re.compile("\n{2,}")
  blocks = pattern.split(content)
  blocks = (x for x in blocks if x.startswith("Device #:"))
  for block in blocks:
    lines = block.split("\n")
    lines = [[x.strip() for x in line] for line in (s.split(":") for s in lines)]
    assert(len(lines[0]) == 2)
    device = Device(lines[0][1])
    for line in lines[1:]:
      if line[0] in ("Test", "Bin"):
        # we call device.doTest or device.doBin, depending on the line's first word
        func = getattr(device, "do%s" % line[0])
        func(line[1:])
    yield device

def log2csv(logPath, csvPath):
  with open(csvPath, "w") as out:
    out.write("Device, INL, DNL, Bin\n")
    for dev in genDevices(logPath):
      out.write("%s, %s, %s\n" % (dev.identity, dev.inldnl, dev.binNumber))

if __name__ == "__main__":
  log2csv("log2.txt", "csv2.txt")