Click here to Skip to main content
14,838,541 members
Articles / Desktop Programming / Windows Forms
Posted 25 Jul 2010


40 bookmarked

Detect Encoding from ByteOrderMarks (BOM)

Rate me:
Please Sign up or sign in to vote.
4.80/5 (15 votes)
15 Aug 2010CPOL4 min read
Avoid foot-traps while using StreamReader / StreamWriter


Each of you knows, to convert raw Bytes to readable Strings and vice-versa, you need a conversion-algorithm, namely an "Encoding". There are several Encodings available, and problems come up, when you read a file using another Encoding than the one, which wrote the file:


Very usual encode-mismatch: write with Utf8, then read with Ansi - most of you will recognize the typical 3 leading chars - this is how Ansi tries to display the ByteOrderMarks of Utf8


My favourite:
"Hello World", written with Ansi-encoding, and read with Utf16

In the "Windows-world", there is a very common technique to ship information about the applied encoding together with the file itself, called "ByteOrderMarking" (BOM):
The first 2, 3 or 4 bytes of the file are not meant to be readable chars, but to indicate the used encoding.
Unfortunately not every encoding has a BOM defined. But the most important encodings do so, namely Utf8, Utf16, Utf32. In the "Windows-world" there is one more "most important" encoding: Ansi - its Webname: "Windows-1252". For instance, Visual Studio saves its files with Ansi. And Ansi has no BOM defined. :(

So an unsafe, heuristic way to guess the encoding is to check for BOM, and if there is none, assume Encoding.Ansi. I admit: This is "quick and dirty at its best" - and I only recommend this approach in cases where you can strongly expect files to be written in one of the common Encodings.

(To learn about a real sophisticated way of guessing the encoding directly from the raw Bytes, refer to [^] Carsten Zeumers article).

How Not To Do It

using(var r = new StreamReader(filename)) {
   richtextBox1.Text = r.ReadToEnd();

Although StreamReader is smart enough to detect BOMs by default, this will cause 2 disadvantages:

  1. If no BOM is detected, StreamReader guesses Utf8 by default. IMO a suboptimal choice, since Utf8 can be detected. As I said, in cases of BOM-absence, better assume Encoding.Ansi.
  2. The detected/guessed Encoding gets lost. Now when you save the file, StreamWriter will choose by default Utf8, so your file-encoding may become changed, but you haven't noticed.

How To Do

Tell StreamReader to guess Encoding.Ansi by default, and store the detected/guessed encoding, to re-use it when writing the file. So the file-encoding will be left as it was before.

private Encoding _Detected;


using(var r = new StreamReader(filename, Encoding.Default)) {
   richtextBox1.Text = r.ReadToEnd();
   _Detected = r.CurrentEncoding;


File.WriteAllText(filename, richtextBox1.Text, _Detected);


Caution! Because the BOM is checked while reading, StreamReader.CurrentEncoding only is meaningful after minimum one Char was tried to be read.

About Foot-traps

May be the main foot-trap is the suboptimal documentation of StreamReaders smartness. Checking out StreamReaders-constructor in Objectbrowser:

public StreamReader(string path, System.Text.Encoding encoding)

Member of System.IO.StreamReader

Summary: Initializes a new instance of the StreamReader class for the specified file name, with the specified character encoding.

Would you now expect, that byte order mark detection is enabled, and the specified encoding may be ignored?
In a way, the summary lies to us: Yes, it instantiates a StreamReader with the given encoding, but it will not use it, in case of detecting a BOM.
To get a more correct description, you must refer to the [^] online-MSDN, since your offline-MSDN may be not up to date. (For instance: on my system, the following hint is missing):


This constructor initializes the encoding as specified by the encoding parameter, and the internal buffer to the default size. The StreamReader object attempts to detect the encoding by looking at the first three bytes of the stream. It automatically recognizes UTF-8, little-endian Unicode, and big-endian Unicode text if the file starts with the appropriate byte order marks. Otherwise, the user-provided encoding is used. See the Encoding.GetPreamble method for more information.

Encodings in XML/HTML

For XML, there is a simple and common standard about the used encoding: It should be specified in the document definition tag, for example:

<?xml version="1.0" encoding="utf-8" ?>

.NET-XmlReader, XmlDocument and XDocument handle such data properly, so if the encoding is specified, you can rely on that the data is read correct.
It's difficult, but still possible to save XML with one encoding, and specify a wrong encoding in the definition-tag, but that's not yours, but the responsibility of the XML author.

The Sample-app

I just built a little thing, with that you can read and write files with different encodings. You can detect encodings, change them, and reproduce mismatches.


  • 7/15/2010: First post
  • 8/15/2010: Add section "Encodings in XML/HTML", added corresponding code to sample-app


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

Germany Germany
No Biography provided

Comments and Discussions

PraiseGreat Tutorial Pin
Burhan Joukhadar9-Aug-19 23:35
MemberBurhan Joukhadar9-Aug-19 23:35 
QuestionTraditional Chinese characters aren’t being read from network stream Pin
Balaji198221-Mar-12 19:44
professionalBalaji198221-Mar-12 19:44 
SuggestionRe: Traditional Chinese characters aren’t being read from network stream Pin
yetibrain16-Sep-13 22:43
Memberyetibrain16-Sep-13 22:43 
QuestionI vote for 5 Pin
sundareswaran.senthilvel5-Feb-12 7:10
professionalsundareswaran.senthilvel5-Feb-12 7:10 
GeneralMy vote of 1 Pin
kiquenet.com25-Aug-10 22:27
professionalkiquenet.com25-Aug-10 22:27 
General[My vote of 2] tips on defaulting to ANSI Pin
Aldo .NET2-Aug-10 2:41
MemberAldo .NET2-Aug-10 2:41 
SuggestionRe: [My vote of 2] tips on defaulting to ANSI Pin
yetibrain16-Sep-13 22:47
Memberyetibrain16-Sep-13 22:47 
GeneralLooks like a tip Pin
PIEBALDconsult26-Jul-10 8:43
professionalPIEBALDconsult26-Jul-10 8:43 
GeneralRe: Looks like a tip Pin
Mr.PoorEnglish26-Jul-10 18:54
MemberMr.PoorEnglish26-Jul-10 18:54 
GeneralRe: Looks like a tip Pin
Paul C Smith27-Jul-10 7:37
MemberPaul C Smith27-Jul-10 7:37 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.