Click here to Skip to main content
15,867,308 members
Articles / Programming Languages / C++
Article

Unicode, MBCS and Generic text mappings

Rate me:
Please Sign up or sign in to vote.
4.94/5 (38 votes)
28 Mar 2000CPOL 584K   152   77
A guide to using generic text functions to make the transition between character sets simple and painless

Introduction

In order to allow your programs to be used in international markets it is worth making your application Unicode or MBCS aware. The Unicode character set is a "wide character" (2 bytes per character) set that contains every character available in every language, including all technical symbols and special publishing characters. Multibyte character set (MBCS) uses either 1 or 2 bytes per character and is used for character sets that contain large numbers of different characters (eg Asian language character sets).

Which character set you use depends on the language and the operating system. Unicode requires more space than MBCS since each character is 2 bytes. It is also faster than MBCS and is used by Windows NT as standard, so non-Unicode strings passed to and from the operating system must be translated, incurring overhead. However, Unicode is not supported on Win95 and so MBCS may be a better choice in this situation. Note that if you wish to develop applications in the Windows CE environment then all applications must be compiled in Unicode.

Using MBCS or Unicode

The best way to use Unicode or MBCS - or indeed even ASCII - in your programs is to use the generic text mapping macros provided by Visual C++. That way you can simply use a single define to swap between Unicode, MBCS and ASCII without having to do any recoding.

To use MBCS or Unicode you need only define either _MBCS or _UNICODE in your project. For Unicode you will also need to specify the entry point symbol in your Project settings as wWinMainCRTStartup. Please note that if both _MBCS and _UNICODE are defined then the result will be unpredictable.

Adjusting your project settings

Generic Text mappings and portable functions

The generic text mappings replace the standard char or LPSTR types with generic TCHAR or LPTSTR macros. These macros will map to different types and functions depending on whether you have compiled with Unicode or MBCS (or neither) defined. The simplest way to use the TCHAR type is to use the CString class - it is extremely flexible and does most of the work for you.

In conjunction with the generic character type, there is a set of generic string manipulation functions prefixed by _tcs. For instance, instead of using the strrev function in your code, you should use the _tcsrev function which will map to the correct function depending on which character set you have compiled for. The table below demonstrates:

#define Compiled Version Example
_UNICODE Unicode (wide-character) _tcsrev maps to _wcsrev
_MBCS Multibyte-character _tcsrev maps to _mbsrev
None (the default: neither _UNICODE nor _MBCS defined) SBCS (ASCII) _tcsrev maps to strrev

Each str* function has a corresponding tcs* function that should be used instead. See the TCHAR.H file for all the mapping and macros that are available. Just look up the online help for the string function in question in order to find the equivalent portable function.

Note: Do not use the str* family of functions with Unicode strings, since Unicode strings are likely to contain embedded null bytes.

The next important point is that each literal string should be enclosed by the TEXT() (or _T()) macro. This macro prepends a "L" in front of literal strings if the project is being compiled in Unicode, or does nothing if MBCS or ASCII is being used. For instance, the string _T("Hello") will be interpreted as "Hello" in MBCS or ASCII, and L"Hello" in Unicode. If you are working in Unicode and do not use the _T() macro, you may get compiler warnings.

Note that you can use ASCII and Unicode within the same program, but not within the same string.

All MFC functions except for database class member functions are Unicode aware. This is because many database drivers themselves do not handle Unicode, and so there was no point in writing Unicode aware MFC classes to wrap these drivers.

Converting between Generic types and ASCII

ATL provides a bunch of very useful macros for converting between different character format. The basic form of these macros is X2Y(), where X is the source format. Possible conversion formats are shown in the following table.

String Type Abbreviation
ASCII (LPSTR) A
WIDE (LPWSTR) W
OLE (LPOLESTR) OLE
Generic (LPTSTR) T
Const C

Thus, A2W converts an LPSTR to an LPWSTR, OLE2T converts an LPOLESTR to an LPTSTR, and so on.

There are also const forms (denoted by a C) that convert to a const string. For instance, A2CT converts from LPSTR to LPCTSTR.

When using the string conversion macros you need to include the USES_CONVERSION macro at the beginning of your function:

void foo(LPSTR lpsz)
{
   USES_CONVERSION;
   
   ...
   LPTSTR szGeneric = A2T(lpsz)
   // Do something with szGeneric
   ...
}

Two caveats on using the conversion macros:

  1. Never use the conversion macros inside a tight loop. This will cause a lot of memory to be allocated each time the conversion is performed, and will result in slow code. Better to perform the conversion outside the loop and pass the converted value into the loop.

  2. Never return the result of the macros directly from a function, unless the return value implies making a copy of the data before returning. For instance, if you have a function that returns an LPOLESTR, then do not do the following:
    LPTSTR BadReturn(LPSTR lpsz)
    {
        USES_CONVERSION;
        // do something
        return A2T(lpsz);
    }

    Instead, you should return the value as a CString, which would imply a copy of the string would be made before the function returns:

    CString GoodReturn(LPSTR lpsz)
    {
        USES_CONVERSION;
        // do something
        return A2T(lpsz);
    }

Tips and Traps

The TRACE statement

The TRACE macros have a few cousins - namely the TRACE0, TRACE1, TRACE2 and TRACE3 macros. These macros allow you to specify a format string (as in the normal TRACE macro), and either 0,1,2 or 3 parameters, without the need to enclose your literal format string in the _T() macro. For instance,

TRACE(_T("This is trace statement number %d\n"), 1);

can be written

TRACE1("This is trace statement number %d\n", 1);

Viewing Unicode strings in the debugger

If you are using Unicode in your applciation and wish to view Unicode strings in the debugger, then you will need to go to Tools | Options | Debug and click on "Display Unicode Strings".

The Length of strings

Be careful when performing operations that depend on the size or length of a string. For instance, CString::GetLength returns the number of characters in a string, NOT the size in bytes. If you were to write the string to a CArchive object, then you would need to multiply the length of the string by the size of each character in the string to get the number of bytes to write:

CString str = _T("Hello, World");
archive.Write( str, str.GetLength( ) * sizeof( TCHAR ) ); 

Reading and Writing ASCII text files

If you are using Unicode or MBCS then you need to be careful when writing ASCII files. The safest and easiest way to write text files is to use the CStdioFile class provided with MFC. Just use the CString class and the ReadString and WriteString member functions and nothing should go wrong. However, if you need to use the CFile class and it's associated Read and Write functions, then if you use the following code:

CFile file(...); 
CString str = _T("This is some text"); 
file.Write( str, (str.GetLength()+1) * sizeof( TCHAR ) ); 

instead of

CStdioFile file(...); 
CString str = _T("This is some text"); 
file.WriteString(str); 

then the results will be Significantly different. The two lines of text below are from a file created using the first and second code snippets respectively:

Unicode strings

(This text was viewed using WordPad)

Not all structures use the generic text mappings

For instance, the CHARFORMAT structure, if the RichEditControl version is less than 2.0, uses a char[] for the szFaceName field, instead of a TCHAR as would be expected. You must be careful not to blindly change "..." to _T("...") without first checking. In this case, you would probably need to convert from TCHAR to char before copying any data to the szFaceName field.

Copying text to the Clipboard

This is one area where you may need to use ASCII and Unicode in the same program, since the CF_TEXT format for the clipboard uses ASCII only. NT systems have the option of the CF_UNICODETEXT if you wish to use Unicode on the clipboard.

Installing the Unicode MFC libraries

The Unicode versions of the MFC libraries are not copied to your hard drive unless you select them during a Custom installation. They are not copied during other types of installation. If you attempt to build or run an MFC Unicode application without the MFC Unicode files, you may get errors.

(From the online docs) To copy the files to your hard drive, rerun Setup, choose Custom installation, clear all other components except "Microsoft Foundation Class Libraries," click the Details button, and select both "Static Library for Unicode" and "Shared Library for Unicode."

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Founder CodeProject
Canada Canada
Chris Maunder is the co-founder of CodeProject and ContentLab.com, and has been a prominent figure in the software development community for nearly 30 years. Hailing from Australia, Chris has a background in Mathematics, Astrophysics, Environmental Engineering and Defence Research. His programming endeavours span everything from FORTRAN on Super Computers, C++/MFC on Windows, through to to high-load .NET web applications and Python AI applications on everything from macOS to a Raspberry Pi. Chris is a full-stack developer who is as comfortable with SQL as he is with CSS.

In the late 1990s, he and his business partner David Cunningham recognized the need for a platform that would facilitate knowledge-sharing among developers, leading to the establishment of CodeProject.com in 1999. Chris's expertise in programming and his passion for fostering a collaborative environment have played a pivotal role in the success of CodeProject.com. Over the years, the website has grown into a vibrant community where programmers worldwide can connect, exchange ideas, and find solutions to coding challenges. Chris is a prolific contributor to the developer community through his articles and tutorials, and his latest passion project, CodeProject.AI.

In addition to his work with CodeProject.com, Chris co-founded ContentLab and DeveloperMedia, two projects focussed on helping companies make their Software Projects a success. Chris's roles included Product Development, Content Creation, Client Satisfaction and Systems Automation.

Comments and Discussions

 
QuestionI cannot define _UNICODE, why? Pin
Abin6-Jun-02 20:27
Abin6-Jun-02 20:27 
AnswerRe: I cannot define _UNICODE, why? Pin
10-Jun-02 6:05
suss10-Jun-02 6:05 
GeneralConfirming MBCS Pin
26-Oct-01 19:07
suss26-Oct-01 19:07 
GeneralMultibyte Character Set (MBCS) Survival Guide Pin
Chris Maunder15-Jul-01 19:37
cofounderChris Maunder15-Jul-01 19:37 
GeneralRe: Multibyte Character Set (MBCS) Survival Guide Pin
User 2694211-Sep-02 3:03
professionalUser 2694211-Sep-02 3:03 
GeneralRe: Multibyte Character Set (MBCS) Survival Guide Pin
Nick Parker9-Jan-03 10:34
protectorNick Parker9-Jan-03 10:34 
QuestionDoes CStdioFile work with Unicode? Pin
Robert Pittenger, MCPD-EAD12-Apr-01 7:49
Robert Pittenger, MCPD-EAD12-Apr-01 7:49 
AnswerRe: Does CStdioFile work with Unicode? Pin
Mike Parks28-Jun-01 11:47
Mike Parks28-Jun-01 11:47 
GeneralGeneric coding at runtime for Unicode ans ANSI Pin
mmacrobert27-Sep-00 13:39
mmacrobert27-Sep-00 13:39 
GeneralWin32 Conversion Pin
Kannan5-Jul-00 2:29
Kannan5-Jul-00 2:29 
GeneralRe: Win32 Conversion Pin
Gowhar4-Sep-02 5:02
Gowhar4-Sep-02 5:02 
GeneralRe: USES_CONVERSION based macros vs. exceptions Pin
Joe O'Leary18-May-00 11:16
Joe O'Leary18-May-00 11:16 
GeneralRe: USES_CONVERSION based macros vs. exceptions Pin
Kim Gräsman12-Aug-02 10:17
Kim Gräsman12-Aug-02 10:17 
GeneralKim is absolutely right Pin
Joe O'Leary12-Aug-02 12:19
Joe O'Leary12-Aug-02 12:19 
GeneralBug getting the number of bytes Pin
Matt Gradwohl27-Apr-00 11:16
sussMatt Gradwohl27-Apr-00 11:16 
GeneralUSES_CONVERSION based macros vs. exceptions Pin
Juraj Rojko5-Apr-00 8:56
Juraj Rojko5-Apr-00 8:56 
GeneralCString.GetLength() with MBCS Pin
M.Vo.3-Apr-00 21:25
M.Vo.3-Apr-00 21:25 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.