I am completing the coding of a utility that reads data from a database and writes the returnset to a text file. The writing to the .txt file is performed by the client. The text file can range in size from 1 line to 20 million lines. When a file is less then 100K, the following code works fairly effieciently, however anything above moves very slow. Does anyone know how to optimize the writing of the output to a .txt file if the recordset count is greater then 100K?
Existing Code:
If Not RS.EOF Then
If bInteractive Then frmMain.StatusBar.Panels(1).Text = "Writing data..."
Set FSO = New FileSystemObject
Set TxtStream = FSO.OpenTextFile(OutputFileName, ForWriting, True)
'Write Column Headers
'Column headers below in ""
DataHeaders = "Field0" & "|" & "Field1" & "|" & "Field2" & "|" & "Field3" & "|" & "Field4 & "|" & "Field5" & "|" & "Field6" & "|" & "Field7" & "|" & "Field8" & "|" & "Field9" & "|" & "Field10" & "|" & "Field11" & "|" & "Field12" & "|" & "Field13" & "|" & "Field14"
TxtStream.WriteLine DataHeaders & vbCrLf
j = 0
k = 0
While Not RS.EOF
' "Field0 - 14 represent variables
Field0 = IIf(IsNull(RS.Fields(0)), "", RS.Fields(0))
Field1 = IIf(IsNull(RS.Fields(1)), "", RS.Fields(1))
Field2 = IIf(IsNull(RS.Fields(2)), "", RS.Fields(2))
Field3 = IIf(IsNull(RS.Fields(3)), "", RS.Fields(3))
Field4 = IIf(IsNull(RS.Fields(4)), "", RS.Fields(4))
Field5 = IIf(IsNull(RS.Fields(5)), "", RS.Fields(5))
Field6 = IIf(IsNull(RS.Fields(6)), "", RS.Fields(6))
Field7 = IIf(IsNull(RS.Fields(7)), "", RS.Fields(7))
Field8 = IIf(IsNull((RS.Fields(8) & "")), "", (RS.Fields(8) & ""))
Field9 = IIf(IsNull(RS.Fields(9)), "", RS.Fields(9))
Field10 = IIf(IsNull(RS.Fields(10)), "", RS.Fields(10))
Field11 = IIf(IsNull(RS.Fields(11)), "", RS.Fields(11))
Field12 = IIf(IsNull(RS.Fields(12)), "", RS.Fields(12))
Field13 = IIf(IsNull(RS.Fields(13)), "", RS.Fields(13))
Field14 = IIf(IsNull(RS.Fields(14)), "", RS.Fields(14))
'Write Column Data
Data = Field0 & "|" & Field1 & "|" & Field2 & "|" & Field3 & "|" & Field4 & "|" & Field5 & "|" & Field6 & "|" & Field7 & "|" & Field8 & "|" & Field9 & "|" & Field10 & "|" & Field11 & "|" & Field12 & "|" & Field13 & "|" & Field14
Buffer = Buffer & Data
j = j + 1
k = k + 1
If bInteractive Then
If k = frmMain.ProgressBar.Max Then
k = 0
frmMain.ProgressBar.Value = 0
frmMain.StatusBar.Panels(4).Text = Format(j, "#,##0") & " of " & Format(lngRecCount, "#,##0")
TxtStream.WriteLine Buffer
Buffer = ""
Else
Buffer = Buffer & vbCrLf
frmMain.ProgressBar.Value = frmMain.ProgressBar.Value + 1
End If
Else
If k = 1000 Then
k = 0
TxtStream.WriteLine Buffer
Buffer = ""
Else
Buffer = Buffer & vbCrLf
End If
End If
RS.MoveNext
DoEvents
Wend
Buffer = Left(Buffer, Len(Buffer) - 2) 'Remove the trailing crlf
TxtStream.WriteLine Buffer
TxtStream.Close