Here is a very simple solution using the
SelectedIndexChanged event handler for the ListBox control.
I'm checking to make sure the index is greater than -1 as that is an invalid index (-1 means nothing is selected). You can modify this as needed but as it stands the snippet makes it so any listbox item clicked on with the value "I can't be selected" will then clear all items selected. Use this event handler as you wish.
Hint: Change the backcolor of items :p
Private Sub ListBox1_SelectedIndexChanged(sender As Object, e As EventArgs) Handles ListBox1.SelectedIndexChanged
If ListBox1.SelectedIndex > -1 And
ListBox1.Items(ListBox1.SelectedIndex) = "I can't be selected" Then
ListBox1.ClearSelected()
End If
End Sub