Hi
I have been struggling to try and split a list of approx 200,000 List(Of Strings) into evenly split arrays for thread processing.
The user is specifying the number of threads (capped obviously) to determine best processing speed for their machine.
If i output the records in the main list: (left out surrounding code)
Console.Write(SQLStrings.Count)
It Gives me
Preperation of SQL Now Complete: 183096 Records to Process
Yet When i run the below code it gives me the following:
Pool: 18311
Pool: 18311
Pool: 18311
Pool: 18311
Pool: 18311
Pool: 18311
Pool: 18311
Pool: 18311
Pool: 18311
Pool: 18297
The Total of which is 18309
7 which is 1 records over, and for the life of me i cannot figure out why i am 1 over (tried several different loops and i keep getting the same result.
Is there a more structure way of splitting a List(Of String) into Array(10) for example to evenly balance it out as these will go into a thread pool for processing.
The reason for this is that is it currently processing much slower than it should be so i am tweaking the insertion of the records which could be up to 1mil records (data gathering from several sources) takes quite some time on a single connection.
Dim ThreadPoolSql(MaxThreadPoolSize - 1) As List(Of String)
For tmp As Integer = 0 To MaxThreadPoolSize - 1
ThreadPoolSql(tmp) = New List(Of String)
Next
Dim PerPoolCount As Integer = SQLStrings.Count / MaxThreadPoolSize + 1
Dim RunningLoop As Integer = 0, RunningCount As Integer = 0
For ListItm As Integer = 0 To SQLStrings.Count - 1
ThreadPoolSql(RunningLoop).Add(SQLStrings(ListItm))
RunningCount += 1
If (RunningCount = PerPoolCount) Then
RunningLoop += 1
RunningCount = 0
End If
Next