Thanks for your answer Dave. I have tried to investigate this issue but I haven't had much luck, especially as the behavior of the port is not constant as far as I could try.
The problem that I posted was greatly reduced after restarting my PC, it seems it was something related to memory (I do not know really).
I completely avoid this error with the following extension method, I don't know if it's the best option, but I hope that helps.
<extension()>
Public Sub OpenPort(ByVal port As SerialPort, Optional ByVal millisecondsSleep As Integer = 20)
port.Open()
System.Threading.Thread.Sleep(millisecondsSleep)
End Sub