Extracting specific fields from a log file and export it to csv file using C#

Question

vishal.patil 0 Newbie Poster

13 Years Ago

This is the code that I have written in C# which shows the contains in the sample.log file.

class FileRead
	{
		public void ReadData()
		{
			FileStream fs = new FileStream("sample.log", FileMode.Open, FileAccess.Read);
			StreamReader sr = new StreamReader(fs);
			sr.BaseStream.Seek(0, SeekOrigin.Begin);
			string str = sr.ReadLine();
			while (str != null)
			{
				Console.WriteLine("{0}", str);
				str = sr.ReadLine();
			}
			sr.Close();
			fs.Close();
			Console.ReadLine();
		}
		public static void Main (string[] args)
		{
			FileRead fr = new FileRead();
			fr.ReadData();
		}
	}

->The sample.log file contains following data fields
sample.log:
8=FIX.4.39=6135=534=149=IDE50=FX52=20101219-18:05:01.52256=SAXOQUOTE10=171
8=FIX.4.39=8135=A49=SAXOQUOTE56=IDE34=157=FX52=20101219-18:06:02369=198=0108=30141=Y10=082
8=FIX.4.39=8535=149=SAXOQUOTE56=IDE34=257=FX52=20101219-18:06:02369=1112=20101219-18:06:0210=053

->Now the problem is I just wanted to print the specific fields from sample.log file in column wise manner or export the specific fields in csv file.
->That is the output should be like this :

Output:
9=61 35=5 52=20101219-18:05:01.522
9=81 35=A 52=20101219-18:06:02
9=85 35=1 52=20101219-18:06:02

Please help me with this.

c#

4 Contributors
13 Replies
409 Views
4 Days Discussion Span
Latest Post 13 Years Ago Latest Post by mcriscolo

mcriscolo 47 Posting Whiz in Training

13 Years Ago

In the message, your excerpt from your "sample.log" file seems messed up. What are the field separators? Are they commas, or tabs, or something else?

Assuming they are commas or tabs, you need to split the string that you read in on that delimiter. See this post on how to do that.

Once you have them in an array of strings, you can select the ones you want.

ddanbe 2,724 Professional Procrastinator

13 Years Ago

Perhaps this could do the trick:

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            string StrFromFile = "8=FIX.4.39=6135=534=149=IDE50=FX52=20101219-18:05:01.52256=SAXOQUOTE10=171";
            char[] chArray = StrFromFile.ToCharArray();
            // your separator char seems to be char 1
            string[] SplitStrs = StrFromFile.Split(new char[] {(char)1});
            string ResultStr = string.Empty;
            for (int i = 0; i < SplitStrs.Length; i++)
            {
                if (SplitStrs[i].StartsWith("9="))
                {
                    ResultStr += SplitStrs[i] + " ";
                }
                else if (SplitStrs[i].StartsWith("35="))
                {
                    ResultStr += SplitStrs[i] + " ";
                }
                else if (SplitStrs[i].StartsWith("52="))
                {
                    ResultStr += SplitStrs[i];
                }
            }
            Console.WriteLine(ResultStr);
            Console.ReadKey();
        }
    }
}

Momerath 1,327 Nearly a Senior Poster

13 Years Ago

Line 8 isn't needed.

Use StringBuilder to build strings. Depending on the log size you could be creating thousands to millions of strings which will result in the Garbage Collector making frequent runs.

ddanbe 2,724 Professional Procrastinator

13 Years Ago

Thanks for pointing that out Momerath :)
As I'm a learner I always appreciate such remarks.
I used line 8 in the debugger to find out what char separator was used. Could have used some hexdump utility of course.
And StringBuilder, you are absolutely right, is the way to go.
I showed (quick and dirty) how it can be done not how it should be done.
EDIT: could also be I'm not perfectly used to the fact that strings in C# are immutable.

Edited 13 Years Ago by ddanbe because: n/a

Reply to this topic

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.

vishal.patil 0 Newbie Poster · Answer 1 · 2011-02-19T12:13:00+00:00

Perhaps this could do the trick:

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            string StrFromFile = "8=FIX.4.39=6135=534=149=IDE50=FX52=20101219-18:05:01.52256=SAXOQUOTE10=171";
            char[] chArray = StrFromFile.ToCharArray();
            // your separator char seems to be char 1
            string[] SplitStrs = StrFromFile.Split(new char[] {(char)1});
            string ResultStr = string.Empty;
            for (int i = 0; i < SplitStrs.Length; i++)
            {
                if (SplitStrs[i].StartsWith("9="))
                {
                    ResultStr += SplitStrs[i] + " ";
                }
                else if (SplitStrs[i].StartsWith("35="))
                {
                    ResultStr += SplitStrs[i] + " ";
                }
                else if (SplitStrs[i].StartsWith("52="))
                {
                    ResultStr += SplitStrs[i];
                }
            }
            Console.WriteLine(ResultStr);
            Console.ReadKey();
        }
    }
}

Thanks for this code I have tried and it works but I have log file which is 100mb in size
and I have to display result line by line and export the result in csv file.

ddanbe 2,724 Professional Procrastinator Featured Poster · Answer 2 · 2011-02-19T13:42:40+00:00

Change the declaration of ResultStr
in StringBuilder ResultStr = new StringBuilder(); as Momerath suggested.
Now use the Append method: ResultStr.Append(SplitStrs + " ");
Display on screen and write to file. I used " " as separator, use , instead.

vishal.patil 0 Newbie Poster · Answer 3 · 2011-02-19T14:29:05+00:00

Change the declaration of ResultStr
in StringBuilder ResultStr = new StringBuilder(); as Momerath suggested.
Now use the Append method: ResultStr.Append(SplitStrs + " ");
Display on screen and write to file. I used " " as separator, use , instead.

Thanks again.
Now I have this sample.log file :
8=FIX.4.39=6335=049=SAXOQUOTE56=IDE34=457=FX52=20101219-18:06:32369=310=003
8=FIX.4.39=6135=034=449=IDE50=FX52=20101219-18:06:32.13056=SAXOQUOTE10=169
8=FIX.4.39=6335=049=SAXOQUOTE56=IDE34=557=FX52=20101219-18:07:02369=410=003
8=FIX.4.39=6135=034=549=IDE50=FX52=20101219-18:07:02.50156=SAXOQUOTE10=170

I have changed some code because I have to read the file from directory and
this code is reading only one line from the sample.log file,

using System;
using System.IO;
using System.Text;

    class Program
    {
        static void Main(string[] args)
        {
            
            FileStream fs = new FileStream("sample.log", FileMode.Open, FileAccess.Read);
            StreamReader sr = new StreamReader(fs);
            string StrFromFile = sr.ReadLine();
            // your separator char seems to be char 1
            string[] SplitStrs = StrFromFile.Split(new char[] {(char)1});
            //string ResultStr = string.Empty;
            StringBuilder ResultStr = new StringBuilder();
        
            for (int i = 0; i < SplitStrs.Length; i++)
            {
               
                if (SplitStrs[i].StartsWith("52="))
                {
                    ResultStr.Append(SplitStrs[i] + " ");
                }
                else if (SplitStrs[i].StartsWith("55="))
                {
                    ResultStr.Append(SplitStrs[i] + " ");
                }
                else if (SplitStrs[i].StartsWith("132="))
                {
                    ResultStr.Append(SplitStrs[i] + " ");
                }
                else if (SplitStrs[i].StartsWith("133="))
                {
                    ResultStr.Append(SplitStrs[i] + " ");
                }
                else if (SplitStrs[i].StartsWith("35="))
                {
                    ResultStr.Append(SplitStrs[i] + " ");
                }
                
            }
          
            Console.WriteLine(ResultStr);
            sr.Close();
			fs.Close();
            Console.ReadKey();
        }
    }

and the output is this :
35=0 52=20101219-18:06:32

Now I am not getting rest lines from sample.log file.
So please tell me how to proceed

ddanbe 2,724 Professional Procrastinator Featured Poster · Answer 4 · 2011-02-19T15:02:15+00:00

I was merely given some sample code on how to process one line read from a file and make a ResultStr out of it. Think you can figure the rest out for yourself.
I would start doing it like this:
while inputfile != EOF
read a line from inputfile
make a resultstring ou of it
write it to the console
write it to a file

vishal.patil 0 Newbie Poster · Answer 5 · 2011-02-22T09:39:04+00:00

I was merely given some sample code on how to process one line read from a file and make a ResultStr out of it. Think you can figure the rest out for yourself.
I would start doing it like this:
while inputfile != EOF
read a line from inputfile
make a resultstring ou of it
write it to the console
write it to a file

I have added the while loop which will check till end of file and print all the records,
but I am not getting the first record and also not getting each record in new line

using System;
using System.IO;
using System.Text;

    class Program
    {
        static void Main(string[] args)
        {
            
            FileStream fs = new FileStream("sample.log", FileMode.Open, FileAccess.Read);
            StreamReader sr = new StreamReader(fs);
            string StrFromFile = sr.ReadLine();

            StringBuilder ResultStr = new StringBuilder();
            
          while ((StrFromFile = sr.ReadLine()) != null)
          { 
            // your separator char seems to be char 1
            string[] SplitStrs = StrFromFile.Split(new char[] {(char)1});        
            for (int i = 0; i < SplitStrs.Length; i++)
            {   
                if (SplitStrs[i].StartsWith("52="))
                {
                    ResultStr.Append(SplitStrs[i] + " ");
                }
                else if (SplitStrs[i].StartsWith("55="))
                {
                    ResultStr.Append(SplitStrs[i] + " ");
                }
                else if (SplitStrs[i].StartsWith("132="))
                {
                    ResultStr.Append(SplitStrs[i] + " ");
                }
                else if (SplitStrs[i].StartsWith("133="))
                {
                    ResultStr.Append(SplitStrs[i] + " ");
                }
                else if (SplitStrs[i].StartsWith("35="))
                {
                    ResultStr.Append(SplitStrs[i] + " ");
                }   
            }  
         }
            Console.WriteLine(ResultStr);
            sr.Close();
			fs.Close();
            Console.ReadKey();
        }
    }

This is the sample.log file:
8=FIX.4.39=6335=049=SAXOQUOTE56=IDE34=457=FX52=20101219-18:06:32369=310=003
8=FIX.4.39=6135=034=449=IDE50=FX52=20101219-18:06:32.13056=SAXOQUOTE10=169
8=FIX.4.39=6335=049=SAXOQUOTE56=IDE34=557=FX52=20101219-18:07:02369=410=003
8=FIX.4.39=6135=034=549=IDE50=FX52=20101219-18:07:02.50156=SAXOQUOTE10=170

Also I am not getting why it is not showing the first record.

I am getting output is like this,
35=0 52=20101219-18:06:32.130 35=0 52=20101219-18:07:02 35=0 52=20101219-18:07:02.501

but I want the output like this,
35=0 52=20101219-18:06:32.130
35=0 52=20101219-18:07:02
35=0 52=20101219-18:07:02.501

Please help me with this.

mcriscolo 47 Posting Whiz in Training · Answer 6 · 2011-02-22T10:48:06+00:00

On line 12, you read the first line of the file. When you go into your "while" loop at line 16, you read the file again, thereby destroying the first line you read from the file without processing it. Delete line 12.

Second, when you output your results, it's outside of the "while" loop. The "ResultStr" will have all of the matches for the entire file, with no CR/LFs in them.

1) If the term starting with "52=" is always the end of a "line", then you can add a CR/LF when you put the value into the "ResultStr";
2) Otherwise, move the "Console.WriteLine(ResultStr)" into the "while" loop, and after you print it, clear the contents of "ResultStr", so you are ready to process the next line.

vishal.patil 0 Newbie Poster · Answer 7 · 2011-02-22T11:04:49+00:00

On line 12, you read the first line of the file. When you go into your "while" loop at line 16, you read the file again, thereby destroying the first line you read from the file without processing it. Delete line 12.
Second, when you output your results, it's outside of the "while" loop. The "ResultStr" will have all of the matches for the entire file, with no CR/LFs in them.
1) If the term starting with "52=" is always the end of a "line", then you can add a CR/LF when you put the value into the "ResultStr";
2) Otherwise, move the "Console.WriteLine(ResultStr)" into the "while" loop, and after you print it, clear the contents of "ResultStr", so you are ready to process the next line.

Thanks for your reply.
Now I am getting the first record but how to use CR/LF, please give me one example.

vishal.patil 0 Newbie Poster · Answer 8 · 2011-02-22T12:02:24+00:00

On line 12, you read the first line of the file. When you go into your "while" loop at line 16, you read the file again, thereby destroying the first line you read from the file without processing it. Delete line 12.
Second, when you output your results, it's outside of the "while" loop. The "ResultStr" will have all of the matches for the entire file, with no CR/LFs in them.
1) If the term starting with "52=" is always the end of a "line", then you can add a CR/LF when you put the value into the "ResultStr";
2) Otherwise, move the "Console.WriteLine(ResultStr)" into the "while" loop, and after you print it, clear the contents of "ResultStr", so you are ready to process the next line.

Thanks for your help.
I got the result in new line per record

Output :
35=5 52=20101219-18:05:01.522
35=A 52=20101219-18:06:01.504
35=A 52=20101219-18:06:02
35=1 52=20101219-18:06:02
35=R 52=20101219-18:06:01.847 55=EUR/USD

Now how do I export this output in CSV file?

This is the code :

using System;
using System.IO;
using System.Text;

    class Program
    {
        static void Main(string[] args)
        {

            FileStream fs = new FileStream("sample.log", FileMode.Open, FileAccess.Read);
            StreamReader sr = new StreamReader(fs);
            string StrFromFile;
            StringBuilder ResultStr = new StringBuilder();


          while ((StrFromFile = sr.ReadLine()) != null)
          { 
            // your separator char seems to be char 1
            string[] SplitStrs = StrFromFile.Split(new char[] {(char)1});        
            for (int i = 0; i < SplitStrs.Length; i++)
            {   
                if (SplitStrs[i].StartsWith("52="))
                {
                    ResultStr.Append(SplitStrs[i] + " ");
                }
                else if (SplitStrs[i].StartsWith("55="))
                {
                    ResultStr.Append(SplitStrs[i] + " ");
                }
                else if (SplitStrs[i].StartsWith("132="))
                {
                    ResultStr.Append(SplitStrs[i] + " ");
                }
                else if (SplitStrs[i].StartsWith("133="))
                {
                    ResultStr.Append(SplitStrs[i] + " ");
                }
                else if (SplitStrs[i].StartsWith("35="))
                {
                    ResultStr.Append(SplitStrs[i] + " ");
                }   
            }

            Console.WriteLine(ResultStr);
            ResultStr.Length = 0;
         }
            sr.Close();
            fs.Close();
            Console.ReadKey();
        }
    }

mcriscolo 47 Posting Whiz in Training · Answer 9 · 2011-02-22T18:15:15+00:00

Instead of separating the fields with spaces, use commas. Then rather than writing to the console, write to a file with a ".CSV" at the end. It's that simple. See this link: http://www.javapractices.com/topic/TopicAction.do?Id=42 for examples of writing to a file.