OutOfMemoryException

[Diamonddrake]

I have written a text editor that sorts lines, I plan to use it with some large files. say about a million lines, ranging between 10 and 75 MB. I am currently testing my app using about a 50mb file with about 900,000 lines. the program adds all the lines from the text box to an array and sorts them, then writes them back, sorted, to the text box. it also copies the text box to a stack for an undoer. other sort methods in my program create multiple arrays and use one as a key to sort the other, and the slowest sorting method using an IComparer class to sort in natural order.

my dilemma is that I get OutOfMemoryExceptions frequently. I have read up the best I could and I don't understand what to do about it. all I can find is something about the compact framework and 64K methods, but Im not using the compact framework. is there something I can do to better handle the large files. is there a better way to implement and undoer, or would dropping the undoer really fix the problem?

what is there to do about this error?

[LizR]

If you truely are running out of memory to do it, if the files are 75mb somethings going wrong, your app shouldnt run out of memory unless you are keeping references to mulitple copies, storing it in class that bulk out the data considerably (and i mean hugely there)

when you assign values to a variable such as maybe dup your array/list or parts of etc, do a try and catch..

Compact framework is for making apps on things like HPs ipac etc.. and other mobile windows units

What you're trying to do doesnt sound unreasonable, so Im just guessing somehow you're creating something and keeping references, and not noticing.

[Diamonddrake]

as it stands, one function adds the lines to a list, and calls the list<string>.Sort(); method. another adds the lines to a list, converts the list to an array, and sorts the array using a custom Icomparer class, not neccessary i believe, I think the list.sort method accepts an Icomparer argument. I will look into that. but as for my most critical sort method. I add all the lines to one list, then I add only the selected characters from the first line, and matching colums of characters for each additional line to another list, then I convert both lists to an array using .toArray(); and then sort the array using the second one as a key.

is there a better way to do that? i tried using an arraylist, but its has to be converted to an array to be sorted as well, at least through the methodology that I have studied.

also
I use the "using" method to create my string readers and writers to ensure that they go out of scope and are destroyed

is there something more I should do?

[LizR]

Well a database would do that easily.

[Diamonddrake]

I am not following you with that statement.

and I have finished the program. It is fully functional, with the exception of frequent crashes for the outOfMemoryException that gets thrown.

it never gave me any trouble until I added a progress bar on the statusstrip for the sorting methods. might there be anything related to that?

[LizR]

It shouldnt be, as a call to update a progress bar shouldnt make any difference, if you got stack exceptions maybe..

Without seeing code though its kinda hard to tell

[Diamonddrake]

It seems that the error always occurs on a line that consists of my main richtextbox control, and occasionally visual studio creates a listing for it in its autos, section that says there was an accessviolation to the rich text box control. anyhow here is the code.

private void linesAlphabeticallyToolStripMenuItem_Click(object sender, EventArgs e)
        {
            Cursor.Current = Cursors.WaitCursor;

            statusBarText.Text = "Sorting Alphabetically, Please be Patient.";
        //    ProgressBar.Visible = true;
            statusBar.Refresh();
        //    ProgressBar.Value = 0;
        //    ProgressBar.Maximum = getLineCount(txtMain.Text, false) * 2;

            myundo.CheckPoint(txtMain.Text);

            //  List<String> myalphalines = txtMain.Text.Split('\n').ToList();
            //  myalphalines.Sort();
            //  txtMain.Text = String.Join("\n", myalphalines.ToArray());

            List<string> myLines = new List<string>();
            using (StringReader r2 = new StringReader(txtMain.Text))
            {
                string txtLine;
                while ((txtLine = r2.ReadLine()) != null)
                {
                    myLines.Add(txtLine);
               //     ProgressBar.Value += 1;
                }
            }
            myLines.Sort();
            using (StringWriter sw = new StringWriter())
            {
                foreach (string s in myLines)
                {
                    sw.WriteLine(s);
              //      ProgressBar.Value += 1;
                }
                txtMain.Text = sw.ToString();
                txtMain.Modified = true;
             //   ProgressBar.Visible = false;
                statusBarText.Text = "";
            }

            myLines = null;
            if (Settings.Default.showLineCount == true)
                this.currentLineCountSSL.Text = getLineCount(txtMain, false).ToString();


            Cursor.Current = Cursors.Default;


        }

        

        private void linesAlphToolStripMenuItem_Click(object sender, EventArgs e)
        {
            Cursor.Current = Cursors.WaitCursor;
            myundo.CheckPoint(txtMain.Text);

            statusBarText.Text = "Sorting Alphanumerically, Please be Patient.";
     //       ProgressBar.Visible = true;
            statusBar.Refresh();
      //      ProgressBar.Value = 0;
       //     ProgressBar.Maximum = getLineCount(txtMain.Text, false) * 2;


            List<string> myANLines = new List<string>();

            NumericComparer ns = new NumericComparer();

           using (StringReader rAn = new StringReader(txtMain.Text))
            {
                string txtLine;
                while ((txtLine = rAn.ReadLine()) != null)
                {
                    myANLines.Add(txtLine);
     //               ProgressBar.Value += 1;
                }
            }
            string[] sortArray = myANLines.ToArray();
            myANLines = null;

          //  string[] sortArray = txtMain.Text.Split('\n').ToArray(); alternet method, doesn't allows for progress bar upadating
            Array.Sort(sortArray, ns);

            using (StringWriter swAN = new StringWriter())
            {
                foreach (string s in sortArray)
                {
                    swAN.WriteLine(s);
       //             ProgressBar.Value += 1;
                }
                txtMain.Text = swAN.ToString();
                txtMain.Modified = true;
         //       ProgressBar.Visible = false;
                statusBarText.Text = "";
            }

            sortArray = null;

            if (Settings.Default.showLineCount == true)
                this.currentLineCountSSL.Text = getLineCount(txtMain, false).ToString();

            Cursor.Current = Cursors.Default;

        }

        private void linesByColumnToolStripMenuItem1_Click(object sender, EventArgs e)
        {
            
            statusBarText.Text = "Sorting by Column, Please be Patient.";
      //      ProgressBar.Visible = true;
            statusBar.Refresh();
       //     ProgressBar.Value = 0;
        //    ProgressBar.Maximum = getLineCount(txtMain.Text, false)*2;
            



            Cursor.Current = Cursors.WaitCursor;
            myundo.CheckPoint(txtMain.Text);

           List<string> mySortLines = new List<string>();
           List<string> mySortLines2 = new List<string>();


           // NumericComparer ns = new NumericComparer(); // create numeric sorting object, removed because not used

            //declare a variable to hold the lenght of the first line
            int int4length = 0;
            int X = 0;
           /* using (StringReader sr4Length = new StringReader(txtMain.Text))
            {
                try
                {
                    int4length = sr4Length.ReadLine().Length;
                }
                catch
                {
                    MessageBox.Show("No text to sort!", "Error", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
                    return;
                }
            }*/


            //do the sorting
            using (StringReader r3 = new StringReader(txtMain.Text))
            {

                string txtSortLine;
                int txtSelectedStart = txtMain.SelectionStart;
                int txtSelectedLength = txtMain.SelectionLength;
                string txtSortcharacter = string.Empty;
                if (txtSelectedLength == 0)
                {
                    MessageBox.Show("you must select characters from the first line of text", "Sort Error", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
                    Cursor.Current = Cursors.Default;
                    return;
                }



                while ((txtSortLine = r3.ReadLine()) != null)
                {

                    //set int4lenght var
                    
                    if (X < 1)
                    {
                        int4length = txtSortLine.Length;
                        X++;

                        //check if text is selected on the first line
                        if (int4length < txtSelectedStart)
                        {
                            MessageBox.Show("you must select characters from the first line of text", "Sort Error", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
                            Cursor.Current = Cursors.Default;
                            return;
                        }
                    }

                    //check if line contains entire selected coulmns 
                    if (txtSortLine.Length >= (txtSelectedStart + txtSelectedLength))
                    {

                        mySortLines.Add(txtSortLine);
                        mySortLines2.Add(txtSortLine.Substring(txtSelectedStart, txtSelectedLength));
                   //     ProgressBar.Value += 1;


                    }
                    //check if lines are shorter than selected columns if so sort as blank
                    if (txtSelectedStart + 1 > txtSortLine.Length)
                    {
                        mySortLines.Add(txtSortLine);
                        mySortLines2.Add("");
                  //      ProgressBar.Value += 1;
                    }
                    //need to sort for lines that contain part of selected columns
                    if (txtSortLine.Length >= (txtSelectedStart + 1) && txtSortLine.Length < (txtSelectedStart + txtSelectedLength))
                    {
                        mySortLines.Add(txtSortLine);
                        mySortLines2.Add(txtSortLine.Substring(txtSelectedStart, txtSortLine.Length - (txtSelectedStart)));
                  //      ProgressBar.Value += 1;
                    }


                }


                string[] sortArray = mySortLines.ToArray();
               string[] sortArray2 = mySortLines2.ToArray();
                Array.Sort(sortArray2, sortArray);

               //  Array.Sort(sortArray2, sortArray, ns); -- sort in natural order, works but results not prefered

               mySortLines = null;
               mySortLines2 = null;
               sortArray2 = null;

                using (StringWriter sw2 = new StringWriter())
                {
                    foreach (string s in sortArray)
                    {
                        sw2.WriteLine(s);
                 //       ProgressBar.Value += 1;
                    }
                    txtMain.Text = sw2.ToString();
                 //   ProgressBar.Visible = false;
                    statusBarText.Text = "";
                    txtMain.Modified = true;


                   sortArray = null;


                }

                if (Settings.Default.showLineCount == true)
                    this.currentLineCountSSL.Text = getLineCount(txtMain, false).ToString();
                
                Cursor.Current = Cursors.Default;
            }

            
        }

        //End of sort by column --

//method that counts lines in a string
      private static int getLineCount(RichTextBox mytxt ,bool showMessage)
        {
            int mylength = mytxt.Text.Split('\n').Length;
            if (showMessage == true)
                MessageBox.Show(mylength.ToString() + " Lines of text in this document", "Document Info");

            return mylength;
        }

this is the code for all three sorting commands, the txtMain refers to my richtextbox control. all three cause the exception error. the progress bar stuff has been commented out, and doesn't fix the error. but its all still there in its entirety.

[LizR]

Ah, if you're sticking the entire contents of a 75mb file in your richtext box, this may not help as this is yet another copy of your data.. Especially as you then manipulate that rather than an internal string..

I'll take a further look when I get to work (where Im heading now)