Click here to Skip to main content
15,914,165 members
Home / Discussions / Visual Basic
   

Visual Basic

 
AnswerRe: Clearing Textbox Control Pin
Kapil Thakur5-Dec-06 20:44
Kapil Thakur5-Dec-06 20:44 
QuestionVB 2005 code to a row and/or its cell values as variables Pin
Karma312515-Dec-06 10:10
Karma312515-Dec-06 10:10 
QuestionForm Resizing When Changing Text Size Thru FontDialog Pin
doveofpeace5-Dec-06 9:28
doveofpeace5-Dec-06 9:28 
QuestionPrinting a report Pin
dptalt5-Dec-06 8:48
dptalt5-Dec-06 8:48 
Questiondataset setup Pin
gorenekli5-Dec-06 6:03
gorenekli5-Dec-06 6:03 
AnswerRe: dataset setup Pin
Kenshino5-Dec-06 9:58
Kenshino5-Dec-06 9:58 
AnswerRe: dataset setup Pin
Polymorpher5-Dec-06 11:06
Polymorpher5-Dec-06 11:06 
Generalworking with large textfiles over the network Pin
milkyjoe5-Dec-06 5:56
milkyjoe5-Dec-06 5:56 
Hi all,

I have recently been trying to find the best method of determining how many lines a text file contains. The main issue has been that the files I am working with are usually above 100Mb and are located on network shares of varying connected speeds and reliability.

Just thought I would share my experiences and see if anyone else had any ideas or recomendations

So far I have tried the following:

1) Looping through the file using the IO.TextReader (both readline, and readtoend methods) but this uses a whole heap of memmory and is very very slow. It is also hard to implement a progress indicater.

<br />
Private Function LineCount(FileName) As Long<br />
      Dim TX As IO.TextReader = IO.File.OpenText(FileName)<br />
<br />
      Dim ALine As String = TX.ReadLine<br />
      Dim Counter As Long = 0<br />
<br />
      Do While Not ALine Is Nothing<br />
         Counter += 1<br />
         ALine = TX.ReadLine<br />
      Loop<br />
<br />
      TX.Close()<br />
      <br />
      Return Counter<br />
<br />
   End Function<br />


2) Looping through the file using the IO.FileStream byte at a time. This seems to perform better than the textreader method but is still a very slow option. It however does provide an easy way to show the progress by calculating the bytes done against the total file size


<br />
Public Function LineCount(ByVal fileName As String) As Long<br />
      <br />
      Dim lineBuffer As Byte()<br />
      lineBuffer = New Byte(4196 - 1) {}<br />
      Dim LineCount As Long<br />
      Dim FI As New IO.FileInfo(fileName)<br />
      Dim FileSize As Long = FI.Length<br />
      Dim BytesProcessed As Long = 0<br />
      Dim BufferRead As Integer = 0<br />
      Dim x As Integer = 0<br />
      LineCount = 0<br />
           <br />
      <br />
      Dim FS As New FileStream(fi.FullName, FileMode.Open, FileAccess.Read, _<br />
      FileShare.Read, lineBuffer.Length)<br />
<br />
      Do While BytesProcessed < FileSize<br />
         Try<br />
            BufferRead = FS.Read(lineBuffer, 0, lineBuffer.Length)<br />
<br />
            If BufferRead > 0 Then<br />
               For x = 0 To (BufferRead - 1)<br />
                  If lineBuffer(x) = 13 Then<br />
                     LineCount += 1<br />
                  End If<br />
<br />
                  ' Evaluate Percent done here i.e Percentdone = CInt((BytesProcessed / FileSize) * 100)<br />
<br />
                  BytesProcessed += 1<br />
               Next<br />
            End If<br />
      Loop<br />
<br />
      FS.Close()<br />
      Return LineCount<br />
<br />
End Function<br />


3) looping through a percentage of the file using the IO.FileStream byte at a time and estimating the number of lines based on the overall file length and the average line byte length. This was fairly fast but produced varying degrees of accuracy as the line lengths are mor often than not irregular.

<br />
Public Function LineCount(ByVal fileName As String, ByVal PercentageRead as Integer) As Long<br />
      <br />
      Dim lineBuffer As Byte()<br />
      lineBuffer = New Byte(4196 - 1) {}<br />
      Dim LineCount As Long<br />
      Dim FI As New IO.FileInfo(fileName)<br />
      Dim FileSize As Long = FI.Length<br />
      Dim BytesProcessed As Long = 0<br />
      Dim BufferRead As Integer = 0<br />
      Dim x As Integer = 0<br />
      Dim FileSizePercentage As Integer = Cint((FileSize / 100) * PercentageRead)<br />
      LineCount = 0<br />
           <br />
      <br />
      Dim FS As New FileStream(fi.FullName, FileMode.Open, FileAccess.Read, _<br />
      FileShare.Read, lineBuffer.Length)<br />
<br />
      Do While BytesProcessed < FileSizePercentage<br />
         Try<br />
            BufferRead = FS.Read(lineBuffer, 0, lineBuffer.Length)<br />
<br />
            If BufferRead > 0 Then<br />
               For x = 0 To (BufferRead - 1)<br />
                  If lineBuffer(x) = 13 Then<br />
                     LineCount += 1<br />
                  End If<br />
<br />
                  ' Evaluate Percent done here i.e Percentdone = CInt((BytesProcessed / FileSizePercentage) * 100)<br />
<br />
                  BytesProcessed += 1<br />
               Next<br />
            End If<br />
      Loop<br />
<br />
      FS.Close()<br />
<br />
      Return Clng((LineCount / FileSizePercentage) * 100)<br />
<br />
End Function<br />


4) Using LogParser Interfaces collection shipped with MS Logparser 2.2. This seems
to be the fastest method but will only work for files with a line count within the Int32 range. And again has no way to determine the percentage done.


<br />
Public Function LineCount(ByVal fileName As String) As Integer<br />
      <br />
      Dim InputClass As New MSUtil.COMTextLineInputContextClass<br />
      Dim LogParserClass As New MSUtil.LogQueryClass<br />
      Dim IRs As MSUtil.ILogRecordset   <br />
      Dim QueryString As String = "SELECT COUNT(Text) FROM '" & fileName & "'"<br />
      <br />
      InputClass.splitLongLines = False        <br />
      <br />
      IRs = LogParserClass.Execute(QueryString, InputClass)<br />
      Return IRs.inputUnitsProcessed<br />
<br />
End Function<br />



Ideas and thoughts appreciated
QuestionRe: working with large textfiles over the network Pin
Are Jay5-Dec-06 18:03
Are Jay5-Dec-06 18:03 
AnswerRe: working with large textfiles over the network Pin
milkyjoe5-Dec-06 23:06
milkyjoe5-Dec-06 23:06 
QuestionHelp me with weird datagrid checkbox problem Pin
Kenshino5-Dec-06 3:21
Kenshino5-Dec-06 3:21 
AnswerRe: Help me with weird datagrid checkbox problem Pin
Kevin Nicol5-Dec-06 3:53
Kevin Nicol5-Dec-06 3:53 
GeneralRe: Help me with weird datagrid checkbox problem Pin
Kenshino5-Dec-06 4:28
Kenshino5-Dec-06 4:28 
GeneralRe: Help me with weird datagrid checkbox problem Pin
Kevin Nicol5-Dec-06 5:18
Kevin Nicol5-Dec-06 5:18 
GeneralRe: Help me with weird datagrid checkbox problem Pin
Kenshino5-Dec-06 5:29
Kenshino5-Dec-06 5:29 
AnswerRe: Help me with weird datagrid checkbox problem Pin
Kevin Nicol5-Dec-06 6:17
Kevin Nicol5-Dec-06 6:17 
GeneralRe: Help me with weird datagrid checkbox problem Pin
Kenshino5-Dec-06 10:08
Kenshino5-Dec-06 10:08 
GeneralRe: Help me with weird datagrid checkbox problem Pin
Kapil Thakur6-Dec-06 1:06
Kapil Thakur6-Dec-06 1:06 
AnswerRe: Help me with weird datagrid checkbox problem Pin
Kapil Thakur5-Dec-06 20:53
Kapil Thakur5-Dec-06 20:53 
GeneralRe: Help me with weird datagrid checkbox problem Pin
Kenshino5-Dec-06 23:57
Kenshino5-Dec-06 23:57 
GeneralRe: Help me with weird datagrid checkbox problem Pin
Kapil Thakur6-Dec-06 1:01
Kapil Thakur6-Dec-06 1:01 
QuestionHelp me tune up this class, pls Pin
king gamo5-Dec-06 3:05
king gamo5-Dec-06 3:05 
AnswerRe: Help me tune up this class, pls Pin
Guffa5-Dec-06 5:15
Guffa5-Dec-06 5:15 
Questionclick in datagrid's cell Pin
Smithers-Jones5-Dec-06 2:17
Smithers-Jones5-Dec-06 2:17 
AnswerRe: click in datagrid's cell Pin
Kevin Nicol5-Dec-06 3:01
Kevin Nicol5-Dec-06 3:01 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.