Cutting Down String Split Overhead

[nexocentric]

Before I state my problem, which is in bold text, I need to give you a little bit of background.

I'm working with 3 binary log files of about 35mb each. Each one holds approximately 65000 records. Each record, which is about 400 bytes, holds information containing the date, transmission type, what device it was transmitted from ect. Since each log file stores information in different places i am currently splitting each record, and inserting them into dynamic string arrays of the same structure. This is so that the same search tool can search all of the different records.

My problem is that I'm spending a lot of processing time on splitting each record using the Mid() function. I'm looking for a faster way to split non-delimited data.

I am currently searching the internet for ways to improve the performance of my program. Not finding much, but I'm still searching.

Thanks for your time.

This is a code block that splits one of the 3 types of log files I'm working with.

'/* Get the filesize and record size of the log to be read */
        GetFileData FileSize, HeaderSize, LinesToRead, DEVICE01_RECORD_FIELD_SIZE, 0, FilePath
        '/* allocate memory for the number of records to be read */
        ReDim RANDOM_LOG_FILE(RECORD_FIELD.TOTAL_FIELDS, LinesToRead)
        '/* RANDOM_LOG_FILE is the actual log file, with all search fields split, in memory */
        
        '/* load the log and split into records */
        '/* this takes approximately 1/2 a second */
        LoadAndSplitLogFile FilePath, DEVICE01_RECORD_FIELD_SIZE, False, TempLogFile
        
        '/* split each record into searchable fields */
        For LinesToRead = 0 To UBound(TempLogFile) - 1 Step 1
            
            '/* starting from here is where i am spending lots of processing time */
            ReDim RecordToCheck(RECORD_FIELD.TOTAL_FIELDS, 0)
            
            RecordToCheck(RECORD_FIELD.date, 0) = Mid(TempLogFile(LinesToRead), 1, 8)
            RecordToCheck(RECORD_FIELD.time, 0) = Mid(TempLogFile(LinesToRead), 9, 6)
            RecordToCheck(RECORD_FIELD.millsec, 0) = Mid(TempLogFile(LinesToRead), 15, 3)
            RecordToCheck(RECORD_FIELD.dateandtime, 0) = Mid(TempLogFile(LinesToRead), 1, 17)
            RecordToCheck(RECORD_FIELD.transtype, 0) = Mid(TempLogFile(LinesToRead), 18, 1)
            '/* find the end of text string */
            DataEnd = InStr(TempLogFile(LinesToRead), ETX)
            If DataEnd < DataStart Then
                DataEnd = DataStart
            End If
            RecordToCheck(RECORD_FIELD.data, 0) = Mid(TempLogFile(LinesToRead), DataStart, DataEnd - DataStart)
            RecordToCheck(RECORD_FIELD.textid, 0) = Mid(TempLogFile(LinesToRead), 33, 3)
            RecordToCheck(RECORD_FIELD.id, 0) = Mid(TempLogFile(LinesToRead), 37, 10)
            RecordToCheck(RECORD_FIELD.key, 0) = Mid(TempLogFile(LinesToRead), 48, 9)
            
            '/* check if the record is a match and then add it to the log file */
            If LogFilter(RecordToCheck, 0) Then
                '/* add record to log file */
                AddMatch RecordToCheck, RANDOM_LOG_FILE, MatchCount
            End If
            
        Next