Multi-threading with C++.NET

[halbichmi]

Hi,

Can someone show me how I would use the multi-threading instead of a case statement for parsing out a file? I have a file that is currently being parsed in C#. Creating Primary keys and foreign keys and putting the data in several dat files. Then I use dts packages in SQL Server DB to tables.

There is an initial record that has coresponding records below it. If I could create something to make each new instance start running the parse it would make the whole thing run a lot faster.

If anyone has any suggestions, please let me know. I could really use the help.

Thanks!!!! :rolleyes: :o

[Chainsaw]

I'm not clear on what you are asking. It sounds like you have an input file you are reading and for some set of records on the input file you use a package to create records in an SQL database. Ok, fine. It also sounds like you have something like:

<initial record>
<set of records>
<initial record>
<set of records>
etc.

And finally, it sounds like you want to spin off a thread so that each set of records is parsed on it's own thread?

Threads are great for backround processing (especially batch processing) or processing a natural stream (like reading sockets), but they would suck at competing for this input file. The reason is that each thread would be fighting for the same set of resources (the input file and the sql databases).

I could see running a thread to do ALL the parsing and writing to SQL so that your foreground thread could do something else (like running the UI), but it would be a bunch of work (you'd have to know when you were done or use semaphores and so on).

Did I understand your question right?

[halbichmi]

let me give you a snid bit of the code I already have in C#. I do not create the DTS packages in my code, I do that separately and have another application that runs the parse file, then the dts, and then finally a stored procedure that renames the new data tables.

This is the multi-thread I started but found out that you can not pass more than one variable to the threads...

using System;
using System.IO;
using System.Threading;
using System.Collections;
using System.ComponentModel;
using System.Data;


namespace Multi_Parse
{
    /// <summary>
    /// Summary description for Class1.
    /// </summary>
    class MyThread
    {

        void Start_Parse()//(int startnum, int linecount)
        {
            //blah blah blah
            try
            {
                //Thread.Sleep(Timeout.Infinite);
            }
            catch(ThreadInterruptedException){/*Continue Processing*/}
            //Console.WriteLine("After C# there will be nothing!",Thr.Name);
        }

        public static string Trim(string temp)
        {
            string tempstr = "";
            tempstr = temp.Replace("'"," ");

            return tempstr.Trim();
            //return temp;
        }

        static void parsefund(string filepath, string destpath)
        {
            string temp;
            string destpath2, filepath2;
            destpath2 = destpath;
            filepath2 = filepath;
            StreamReader reader = null;
            string delim_line = null;
            int countline = 0;
            int lcount;
            int cPkey;


            reader = new StreamReader (filepath);
            for (string line = Trim(reader.ReadLine()); line !=null;line = 
reader.ReadLine())
            {
                //Console.WriteLine(line.Length);
                ++countline;
                Console.WriteLine(countline);
                temp = line.Substring(0,4);
                if (temp == "0000")
                {
                    lcount = countline;
                    cPkey = 10000000;
                }



                MyThread myThr = new MyThread();

                Thread Thr1 = new Thread(new ThreadStart(myThr.Start_Parse));//(lcount, 
cPkey)));
                //Thread Thr2 = new Thread(new ThreadStart(myThr.Start_Parse));
                //Thread Thr3 = new Thread(new ThreadStart(myThr.Start_Parse));
                //Thread Thr4 = new Thread(new ThreadStart(myThr.Start_Parse));
                //Thread Thr5 = new Thread(new ThreadStart(myThr.Start_Parse));
                //Thread Thr6 = new Thread(new ThreadStart(myThr.Start_Parse));
            }
        }

        /// <summary>
        /// The main entry point for the application.
        /// </summary>
        [STAThread]

        static void Main(string[] args)
        {
            //
            // TODO: Add code to start application here
            //
            Console.WriteLine("Main Program has Fricken Started");

        {
            if (args.Length == 0 )
            {
                Console.WriteLine ("Incorrect syntax");
                return;
            }
            else
                if (args.Length ==1)
            {
                Console.WriteLine ("Incorrect syntax:Source file is missing");
                return;
            }
            else
                if (args.Length ==2)
            {
                Console.WriteLine ("Incorrect syntax:Missing destination path");
                return;
            }

            parsefund(args[1],args[2]);




        }
        }
    }
}

This is the current way I am parseing the flat file into individual dat files that are imported into new tables each day:

#region Parse FUND
        static void parsefund(string filepath, string destpath)
        {
            StreamReader reader = null;
            string delim_line = null;
            int count = 0, count2 = 0, count3 = 0, count5 = 0, count6 = 0, 
count625 = 0, count825 = 0;


try
    {
        FileStream strm_c0;
        FileStream strm_c2;
        FileStream strm_c3;
        FileStream strm_c5;
        FileStream strm_c6;
        FileStream strm_c625;
        FileStream strm_c725;
        FileStream strm_c825;
try
    {
        strm_c0000 = new FileStream(destpath + 
"\\rec_c0.dat",FileMode.Truncate);
        strm_c2 = new FileStream(destpath + 
"\\rec_c2.dat",FileMode.Truncate);
        strm_c3 = new FileStream(destpath + 
"\\rec_c3.dat",FileMode.Truncate);
        strm_c5 = new FileStream(destpath + 
"\\rec_c5.dat",FileMode.Truncate);
        strm_c6 = new FileStream(destpath + 
"\\rec_c6.dat",FileMode.Truncate);
        strm_c625 = new FileStream(destpath + "\\rec_c625.dat",FileMode.Truncate);

}
catch (FileNotFoundException)
    {
strm_c0000 = new FileStream(destpath + 
"\\rec_c0000.dat",FileMode.OpenOrCreate,FileAccess.Write);
strm_c2 = new FileStream(destpath + 
"\\rec_c2.dat",FileMode.OpenOrCreate,FileAccess.Write);
strm_c3 = new FileStream(destpath + 
"\\rec_c3.dat",FileMode.OpenOrCreate,FileAccess.Write);
strm_c5 = new FileStream(destpath + 
"\\rec_c5.dat",FileMode.OpenOrCreate,FileAccess.Write);
strm_c6 = new FileStream(destpath + 
"\\rec_c6.dat",FileMode.OpenOrCreate,FileAccess.Write);
strm_c625 = new FileStream(destpath + 
"\\rec_c625.dat",FileMode.OpenOrCreate,FileAccess.Write);

}
                strm_c0.Close();
                strm_c2.Close();
                strm_c3.Close();
                strm_c5.Close();
                strm_c6.Close();
                strm_c625.Close();

FileInfo fund_c0 = new FileInfo(destpath + "\\rec_c0.dat");
FileInfo fund_c2 = new FileInfo(destpath + "\\rec_c2.dat");
FileInfo fund_c3 = new FileInfo(destpath + "\\rec_c3.dat");
FileInfo fund_c5 = new FileInfo(destpath + "\\rec_c5.dat");
FileInfo fund_c6 = new FileInfo(destpath + "\\rec_c6.dat");
FileInfo fund_c625 = new FileInfo(destpath + "\\rec_c625.dat");

StreamWriter Stream_fund_c0 = fund_c0.AppendText();
StreamWriter Stream_fund_c2 = fund_c2.AppendText();
StreamWriter Stream_fund_c3 = fund_c3.AppendText();
StreamWriter Stream_fund_c5 = fund_c5.AppendText();
StreamWriter Stream_fund_c6 = fund_c6.AppendText();
StreamWriter Stream_fund_c625 = fund_c625.AppendText();

reader = new StreamReader (filepath);
for (string line = Trim(reader.ReadLine()); line !=null;line =reader.ReadLine())
{
Console.WriteLine(line.Length);
    ++countline;
Console.WriteLine(countline);
switch (line.Substring(0,4))
{
case "0":
{
    //Console.WriteLine("0000");
    ++count;
   delim_line = count + "|" + Trim(line.Substring(4,14)) + "|" + 
Trim(line.Substring(18,30)) +"|" + Trim(line.Substring(48,1)) + "|" + Trim(line.Substring(49,2)) + "|" + Trim(line.Substring(51,6)) ;

Stream_fund_c0.WriteLine (delim_line);
break;
}
case "2" :
{
++count2;
  delim_line = count2 + "|" + count + "|" + conv_to_money(line.Substring(4,14)) +"|" + conv_to_money(line.Substring(18,14));
Stream_fund_c2.WriteLine (delim_line);
break;
}
case "3" :
{
  ++count3;
  delim_line = count3 + "|" + count + "|" +Trim(line.Substring(4,15)) +
"|" + Trim(line.Substring(19,15)) + "|" + Trim(line.Substring(34,1));

Stream_fund_c3.WriteLine (delim_line);
break;
}
case "14" :
{
++count1400;
delim_line = count14 + "|" + count3 + "|" + 
Trim(line.Substring(4,15)) +"|" + Trim(line.Substring(19,15)) + "|" + Trim(line.Substring(34,3));

That is just a snid bit, but as you can see, each time a record of c0 is read, it has other records below it and are associated with it. So what I want to do, is each time it hit the c0, start a new thread that runs each of the lower records(ie. c2,c3,and so on) to see if it will speed up the parsing process.

[halbichmi]

One more note, each thread of the c0 has to start with a new instance of the c0, but append the current primary key. So I have to pass the c0 and the current primary key number to keep the records in order and not have duplicate keys for different records.