In the documentation of the
Write[
^] method is a remark about the encoding used by the function.
By default, SerialPort uses ASCIIEncoding to encode the characters. ASCIIEncoding encodes all characters greater then 127 as (char)63 or '?'. To support additional characters in that range, set Encoding to UTF8Encoding, UTF32Encoding, or UnicodeEncoding.
So the encoding is messing up your bytes, as stated any byte larger than 127 is replaced by char(63) '?'.
Try using a different encoding.