How to add rows to a datatable simultaneously (multithreading)

Question

0.00/5 (No votes)

See more:

I have made an IP Scanner, which contains an Backgroundworker for handling the Scanprocess. The Backgroundworker creates new Threads that were doing the real scan for every IP Address...
My Problem is that I need to collect the scanresults for each IP Address and this Collection needs to be Threadsafe especially in writing...

The Code for my Backgroundworker

Private Sub NwS_BackgroundWorker_DoWork(sender As Object, e As ComponentModel.DoWorkEventArgs) Handles NwS_BackgroundWorker.DoWork
        Dim toDoList As DataTable = e.Argument
        
        ' Progress Calculation
        Dim total As Int64 = NwS_ScanTotal(toDoList), current As Int64 = 0, progress As Int64 = 0

        ' Test for a Network Connection
        Dim netStatus As Boolean = False, netInterfaces As String = ""
        For Each networkCard As NetworkInterface In NetworkInterface.GetAllNetworkInterfaces
            If networkCard.Name = "Ethernet" Or networkCard.Name = "WLAN"
                netInterfaces = netInterfaces & $"
{networkCard.Name} : {networkCard.OperationalStatus.ToString}"
                If networkCard.OperationalStatus = OperationalStatus.Up Then netStatus = True
            End If
        Next
        If netStatus = False
            Dim messageBox = New Message_Box($"No Network Connection detected.{netInterfaces}", MetroColorStyle.Red)
            If messageBox.ShowDialog() = DialogResult.OK
                e.Cancel = True
                NwS_BackgroundWorker.CancelAsync()
            End If
        End If

        
        For Each ipRange As DataRow In toDoList.Rows
            
            Dim ipExport As IPAddress, endIp = Split(ipRange.Item(1), "."), startIp = Split(ipRange.Item(0), ".")

            ' Create a ThreadList
            Dim threadList As List(Of Thread) = New List(Of Thread)

            ' Scan all IPs
            For A As Int16 = startIp(0) To endIp(0)
                For B As Int16 = startIp(1) To endIp(1)
                    For C As Int16 = startIp(2) To endIp(2)
                        For D As Int16 = startIp(3) To endIp(3)
                            ipExport = Net.IPAddress.Parse(A & "." & B & "." & C & "." & D)

                            ' Report Progress
                            current += 1
                            progress = current/ total * 100
                            NwS_BackgroundWorker.ReportProgress(progress, $"Scanning {ipExport}")

                            ' Creating and starting new Background Thread
                            Dim oThread As New Thread(AddressOf Me.NwS_PingHost)
                            oThread.IsBackground = True
                            threadList.Add(oThread)
                            oThread.Start(ipExport)

                            ' End Conditions
                            If NwS_BackgroundWorker.CancellationPending
                                e.Cancel = True
                                Exit Sub
                            End If
                            Next

                        ' Handbrake for decreasing CPU Usage
                        NwS_BackgroundWorker.ReportProgress(progress, " ")
                        NwS_Handbrake(threadList, False)
                        threadList.Clear()
                    Next
                Next
            Next
        Next
        GC.Collect
    End Sub

Scan Host adding Datarows

Private Sub NwS_PingHost(ByVal ipExport As IPAddress)
' Set NetworkScan Variables for Ping Request
        Dim status As String = "Unknown", hostName As String = " ", macImport As String = " ", macAddress As String = " "
        Dim ipLong As Int64 = IPtoLong(ipExport)

        Dim result As Net.NetworkInformation.PingReply
        Dim sendPing As New Net.NetworkInformation.Ping

        Try
            ' Try to send Ping Request
            result = sendPing.Send(ipExport, 120)
            If result.Status = Net.NetworkInformation.IPStatus.Success Then

                ' Resolve Hostname
                status = "Used"
                hostName = Net.Dns.GetHostEntry(ipExport).HostName

                'Resolve Mac- Address
                Dim arp = New ArpRequest(ipExport)
                macImport = arp.GetResponse().ToString()
                If macImport <> Nothing Then
                    macAddress = Regex.Replace(macImport, "(.{2})(.{2})(.{2})(.{2})(.{2})(.{2})", "$1:$2:$3:$4:$5:$6")
                End If
            Else
                status = "Free"
                hostName = ""
            End If
            

            ' Ping Exception: Non valid IP
        Catch ex As PingException
            status = "Unknown"

            ' Hostname Exception: IP resolving Error
        Catch ex As SocketException
            status = "Unknown"

            ' Hostname Exception: Non valid IP
        Catch ex As System.ArgumentException
            status = "Unknown"

            ' Mac Exceptions: (Arp Request)
        Catch ex As System.ComponentModel.Win32Exception
            If ex.NativeErrorCode = 67 Then
                status = "Used"
                macAddress = "--:--:--:--:--:--"
            ElseIf ex.NativeErrorCode = 111 Then
                status = "Used"
                macAddress = "--:--:--:--:--:--"
            ElseIf ex.NativeErrorCode = 31 Then
                status = "Used"
                macAddress = "--:--:--:--:--:--"
            ElseIf ex.NativeErrorCode = 87 Then
                status = "Used"
                macAddress = "--:--:--:--:--:--"
            ElseIf ex.NativeErrorCode = 1784 Then
                status = "Used"
                macAddress = "--:--:--:--:--:--"
            ElseIf ex.NativeErrorCode = 1168 Then
                status = "Used"
                macAddress = "--:--:--:--:--:--"
            End If
        Catch ex As Exception

        End Try

        ' Add results to DataGrid and remove Thread from List
        If status = "Used" Then
            NwSData.Rows.Add(My.Resources.Host, ipExport.ToString, hostName, macAddress, status, ipLong.ToString("D10"))
        ElseIf status = "Free" Then
            NwSData.Rows.Add(My.Resources.Free, ipExport.ToString, hostName, macAddress, status, ipLong.ToString("D10"))
        Else
            NwSData.Rows.Add(My.Resources.Unresolvable, ipExport.ToString, hostName, macAddress, status, ipLong.ToString("D10"))
        End If

        ' CleanUP
        ipExport = Nothing
        status = ""
        hostName = ""
        macImport = ""
        macAddress = ""
    End Sub

What I have tried:

I created a DataTable Object = Unsafe for multithreading --> Exception in large Scans "Internal Index corrupted '5'"
I searched for threadsafe Methods to add the Rows, or avoiding this exception...
The DataSource needs to be bound to the DTGV, otherwise loading all results (e.g. 0.0.0.0 - 255.255.255.255) will cause a freeze or program cancellation...
Do you have any Ideas?

My DataSources:

Public NwSData As DataTable = New DataTable("NwS_Log")
    Public NwSBinding As BindingSource = New BindingSource()
    ' Create NwS DataSource
    Private Sub NwS_DataSource()
        NwSData.Columns.Add("NwS_Column1", GetType(System.Drawing.Image))
        NwSData.Columns.Add("NwS_Column2", GetType(String))
        NwSData.Columns.Add("NwS_Column3", GetType(String))
        NwSData.Columns.Add("NwS_Column4", GetType(String))
        NwSData.Columns.Add("NwS_Column5", GetType(String))
        NwSData.Columns.Add("NwS_Column6", GetType(String))
        NwSBinding.DataSource = NwSdata
        NwS_Log.DataSource = NwSBinding
        NwS_Log.Columns(0).AutoSizeMode = DataGridViewAutoSizeColumnMode.None
        NwS_Log.Columns(0).HeaderText = "Status"
        NwS_Log.Columns(1).AutoSizeMode = DataGridViewAutoSizeColumnMode.None
        NwS_Log.Columns(1).HeaderText = "IP Address"
        NwS_Log.Columns(2).AutoSizeMode = DataGridViewAutoSizeColumnMode.Fill
        NwS_Log.Columns(2).HeaderText = "Hostname"
        NwS_Log.Columns(3).AutoSizeMode = DataGridViewAutoSizeColumnMode.Fill
        NwS_Log.Columns(3).HeaderText = "Mac - Address"
        NwS_Log.Columns(4).Visible = False
        NwS_Log.Columns(4).HeaderText = "Status String"
        NwS_Log.Columns(5).Visible = False
        NwS_Log.Columns(5).HeaderText = "Bytes"

        ' Put each of the columns into programmatic sort mode.
        For Each column As DataGridViewColumn In NwS_Log.Columns
            column.SortMode = DataGridViewColumnSortMode.Programmatic
        Next
    End Sub

Posted 18-May-20 6:21am

User 14766787

Updated 18-May-20 6:47am

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Richard Deeming · Accepted Answer · 2020-05-18T06:48:00

Solution 1

As you've discovered, the DataTable class is not thread-safe. If you try to add rows from multiple threads at the same time, the internal state will be corrupted.

The simple solution is to wrap the row addition in a SyncLock block:

VB.NET

' Add results to DataGrid and remove Thread from List
SyncLock NwSData
    If status = "Used" Then
        NwSData.Rows.Add(My.Resources.Host, ipExport.ToString, hostName, macAddress, status, ipLong.ToString("D10"))
    ElseIf status = "Free" Then
        NwSData.Rows.Add(My.Resources.Free, ipExport.ToString, hostName, macAddress, status, ipLong.ToString("D10"))
    Else
        NwSData.Rows.Add(My.Resources.Unresolvable, ipExport.ToString, hostName, macAddress, status, ipLong.ToString("D10"))
    End If
End SyncLock

SyncLock Statement - Visual Basic | Microsoft Docs[^]

You will need to measure what performance impact that has on your code before deciding whether to explore a more complicated solution.

EDIT:
Based on the stack-trace of the exception, I suspect the grid doesn't like having the data source updated from a background thread. You may need to use BeginInvoke to update the table from the UI thread, which would remove the need for the SyncLock block. Try something like this:

VB.NET

' Add results to DataGrid and remove Thread from List
If status = "Used" Then
    BeginInvoke(Sub()
        NwSData.Rows.Add(My.Resources.Host, ipExport.ToString, hostName, macAddress, status, ipLong.ToString("D10"))
    End Sub)
ElseIf status = "Free" Then
    BeginInvoke(Sub()
        NwSData.Rows.Add(My.Resources.Free, ipExport.ToString, hostName, macAddress, status, ipLong.ToString("D10"))
    End Sub)
Else
    BeginInvoke(Sub()
        NwSData.Rows.Add(My.Resources.Unresolvable, ipExport.ToString, hostName, macAddress, status, ipLong.ToString("D10"))
    End Sub)
End If

Posted 18-May-20 6:48am

Richard Deeming

Updated 18-May-20 7:44am

v2

Comments

[no name] 18-May-20 13:00pm

Ok, I tried Synclock yesterday, but maybe I was to tired... Because I used it for the whole ScanHost ---> Very Big Impact, lol
Now with only at the DataRow adding the impact is almost not measurable :)
So thanks for that first Idea :)

But at some point on my test scan (which always leaded to the Exception above "0.0.0.0 - 254 + 100.0.0.1 - 192.0.0.254) it throws me a new Exception "System.NullReferenceException: Object reference was not set to an object"
Marked with that x is:
NwSData.Rows.Add(My.Resources.Free, ipExport.ToString, hostName, macAddress, status, ipLong.ToString("D10"))

Have you an idea where that is coming from?

Richard Deeming 18-May-20 13:06pm

Does it tell you which variable on that line is Nothing?

[no name] 18-May-20 13:18pm

This is the detailslist for the Exception, the debuger doesn't mention the varname
ipExport = {110.0.0.219}
hostName = ""
maxAddress = " "
status = "Free"
My.Ressources.Free = {System.Drawing.Bitmap}
ipLong = 1845493979
The thing I'm wondering about is that NwSData = {NwS_Log} (My DTGV)

System.NullReferenceException
HResult=0x80004003
Message = Object Reference was not set to an object
Source = System.Windows.Forms
Batch Monitoring:
at System.Windows.Forms.DataGridView.FlushDisplayedChanged()
at System.Windows.Forms.DataGridView.PerformLayoutPrivate(Boolean useRowShortcut, Boolean computeVisibleRows, Boolean invalidInAdjustFillingColumns, Boolean repositionEditingControl)
at System.Windows.Forms.DataGridView.ResetUIState(Boolean useRowShortcut, Boolean computeVisibleRows)
at System.Windows.Forms.DataGridViewRowCollection.OnCollectionChanged_PreNotification(CollectionChangeAction cca, Int32 rowIndex, Int32 rowCount, DataGridViewRow& dataGridViewRow, Boolean changeIsInsertion)
at System.Windows.Forms.DataGridViewRowCollection.OnCollectionChanged(CollectionChangeEventArgs e, Int32 rowIndex, Int32 rowCount, Boolean changeIsDeletion, Boolean changeIsInsertion, Boolean recreateNewRow, Point newCurrentCell)
at System.Windows.Forms.DataGridViewRowCollection.InsertInternal(Int32 rowIndex, DataGridViewRow dataGridViewRow, Boolean force)
at System.Windows.Forms.DataGridView.DataGridViewDataConnection.ProcessListChanged(ListChangedEventArgs e)
at System.Windows.Forms.DataGridView.DataGridViewDataConnection.currencyManager_ListChanged(Object sender, ListChangedEventArgs e)
at System.Windows.Forms.CurrencyManager.OnListChanged(ListChangedEventArgs e)
at System.Windows.Forms.CurrencyManager.List_ListChanged(Object sender, ListChangedEventArgs e)
at System.Windows.Forms.BindingSource.OnListChanged(ListChangedEventArgs e)
at System.Windows.Forms.BindingSource.InnerList_ListChanged(Object sender, ListChangedEventArgs e)
at System.Data.DataView.OnListChanged(ListChangedEventArgs e)
at System.Data.DataView.IndexListChanged(Object sender, ListChangedEventArgs e)
at System.Data.Listeners`1.Notify[T1,T2,T3](T1 arg1, T2 arg2, T3 arg3, Action`4 action)
at System.Data.Index.OnListChanged(ListChangedEventArgs e)
at System.Data.Index.OnListChanged(ListChangedType changedType, Int32 index)
at System.Data.Index.InsertRecord(Int32 record, Boolean fireEvent)
at System.Data.DataTable.RecordStateChanged(Int32 record1, DataViewRowState oldState1, DataViewRowState newState1, Int32 record2, DataViewRowState oldState2, DataViewRowState newState2)
at System.Data.DataTable.SetNewRecordWorker(DataRow row, Int32 proposedRecord, DataRowAction action, Boolean isInMerge, Boolean suppressEnsurePropertyChanged, Int32 position, Boolean fireEvent, Exception& deferredException)
at System.Data.DataTable.InsertRow(DataRow row, Int64 proposedID, Int32 pos, Boolean fireEvent)
at System.Data.DataRowCollection.Add(Object[] values)
at IP_Tool.Main.NwS_PingHost(IPAddress ipExport) in C:\Users\Electro Attacks\source\repos\Advanced IP Tool\Main.vb:line 535
at IP_Tool.Main._Lambda$__R260-16(Object a0)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)

This exception was originally raised by this call list:
[External Code]
IP_Tool.Main.NwS_PingHost(System.Net.IPAddress) in Main.vb
[External Code]

Richard Deeming 18-May-20 13:45pm

Based on the stack trace, I suspect the grid view doesn't like having its data source updated from a background thread. You could try using BeginInvoke to update the table from the UI thread. I've added a sample to my solution - although I'm a bit rusty on the precise VB.NET syntax. :)

[no name] 18-May-20 13:51pm

Well, but when I'm starting with Invokes again doesn't make it freeze my UI when ~ 200 Threads want's to update the DataGridView via Invokes?
Wouldn't be it better when I use NwSBinding.SuspendBinding() at the Beginning of the Scan and than NwSBidning.ResumeBinding() it on the BackgroundWorker.ReportProgress Event?

And it's wondering me, that the DTGV recognizes it so late, that the updates were coming from different Threads than the UI... That were about 2.805 rows before it was checking it... And Before the Exception was thrown it was updating very fast so I could see every new entry :/

I think it could be a problem made by the BindingSource...

Richard Deeming 18-May-20 13:59pm

I'd hope that the Invoked code is small enough that it wouldn't completely freeze the UI, but you'd have to test it to see.

Otherwise, assuming the NwS_Handbrake method is waiting for the pending threads to complete, you could resume and then re-suspend the binding when that's done. You'd have to make sure to always resume the binding in the RunWorkerCompleted event to cope with cancellation and errors.

[no name] 18-May-20 14:08pm

So now I tried the Scan again without any changes and it completed the scan without any issues... (Well the Scrollbar of the TGV is still... I think you could call that destroyed...)

So because I tried the same ranges and with the same conditions (Handbrake disabled), it may be a racecondition again... The question is than: Where did it come from? xD
So reading of public vars shouldn't be a problem, but what is about a performance Counter?
And than I have to figure out how I can fix the scrollbar on large Scans... Maybe redraw the DTGV or the bar?

Private Sub NwS_Handbrake(ByVal threadList As List(Of Thread), ByVal Mode As Boolean)
        Do
            Dim pause As Boolean = False

            ' Handbrake: Wait for Threads
            Dim tAlive As Integer = 0
            If Mode Or My.Settings.Handbrake
                For Each t In threadList
                    If t.IsAlive() <> False Then
                        tAlive += 1
                    End If
                Next t

                If tAlive > 0 Then pause = True
            End If

            ' Handbrake: CPU Limit
            If My.Settings.Handbrake = False And Mode = False
                Dim cpuLoad = Main_cpuWatcher.NextValue()
                If cpuLoad > 80 Then pause = True
            End IF

            If pause = False Then Exit Do
            Thread.Sleep(1000)
        Loop
    End Sub

[no name] 18-May-20 14:47pm

So last update:

Thanks a lot for your Time, without you I would still for Thread- Safe dataTables, just because I was using using Synclock at the wrong point xD

The Program works fine now, ScrollBars were fixed now too by just redrawing them with:

NwS_Log.ScrollBars = ScrollBars.None
        NwS_Log.ScrollBars = ScrollBars.Vertical

Todd Howard would say: "It just works" xD

Maciej Los 18-May-20 17:16pm

5ed!