what kind of algorithm to Comparing Images using vb.net

Strictly speaking, the question as it is formulated does not make sense. A comparison is a category which needs definitions for every class of objects. It is defined for numbers, strings, but not for images. The only apparent natural part of definition is equivalence, but comparison also implies <=, >=…

In practice, equivalence of images pixel-to-pixel is rarely used by applications. Extensively discussed is the idea of recognition of "similar" images, which is extremely difficult even to formulate in a practically sensible way.

—SA

Try this one :

Private Sub btnCheck_Click(ByVal sender As System.Object, _
    ByVal e As System.EventArgs) Handles btnGo.Click
    Me.Cursor = Cursors.WaitCursor
    Application.DoEvents()
 
    ' Get the threshold.
    Dim threshold As Integer = _
        Integer.Parse(txtThreshold.Text)
 
    ' Load the images.
    Dim bmp1 As Bitmap = Image.FromFile(txtFile1.Text)
    Dim bmp2 As Bitmap = Image.FromFile(txtFile2.Text)
 
    ' Make a difference image.
    Dim wid As Integer = Math.Min(bmp1.Width, bmp2.Width)
    Dim hgt As Integer = Math.Min(bmp1.Height, bmp2.Height)
    Dim bmp3 As New Bitmap(wid, hgt)
 
    ' Create the difference image.
    Dim are_identical As Boolean = True
    Dim r1, g1, b1, r2, g2, b2, r3, g3, b3 As Integer
    Dim color1, color2 As Color
    Dim eq_color As Color = Color.White
    Dim ne_color As Color = Color.Red
    Dim dr, dg, db, diff As Integer
    For x As Integer = 0 To wid - 1
        For y As Integer = 0 To hgt - 1
            color1 = bmp1.GetPixel(x, y)
            color2 = bmp2.GetPixel(x, y)
            dr = CInt(color1.R) - color2.R
            dg = CInt(color1.G) - color2.G
            db = CInt(color1.B) - color2.B
            diff = dr * dr + dg * dg + db * db
            If diff <= threshold Then
                bmp3.SetPixel(x, y, eq_color)
            Else
                bmp3.SetPixel(x, y, ne_color)
                are_identical = False
            End If
        Next y
    Next x
 
    ' Display the result.
    picResult.Image = bmp3
 
    Me.Cursor = Cursors.Default
    If (bmp1.Width <> bmp2.Width) OrElse (bmp1.Height <> _
        bmp2.Height) Then are_identical = False
    If are_identical Then
        MessageBox.Show("The images are identical")
    Else
        MessageBox.Show("The images are different")
    End If
 
    bmp1.Dispose()
    bmp2.Dispose()
End Sub