Click here to Skip to main content
15,860,972 members
Articles / Programming Languages / C++

Convert Between std::string and std::wstring, UTF-8 and UTF-16

Rate me:
Please Sign up or sign in to vote.
3.60/5 (16 votes)
20 May 2007CPOL1 min read 329.9K   2.8K   63   43
How to convert safely STL strings between Unicode formats

Introduction

I needed to convert between UTF-8 coded std::string and UTF-16 coded std::wstring. I found some converting functions for native C strings, but these leave the memory handling to the caller. Not nice in modern times.

The best converter is probably the one from unicode.org. Here is a wrapper around this one which converts the STL strings.

Unlike other articles, this one has no other dependencies, does not introduce yet another string class, only converts the STL strings, and that's it. And it's better than the widely found...

C++
std::wstring widestring(sourcestring.begin(), sourcestring.end()); 

... which only works for ASCII text.

Source

The header goes like this:

C++
#ifndef UTFCONVERTER__H__
#define UTFCONVERTER__H__

namespace UtfConverter
{
    std::wstring FromUtf8(const std::string& utf8string);
    std::string ToUtf8(const std::wstring& widestring);
}

#endif

I guess this is simple and easy enough to use.

Here is the source code:

C++
#include "stdafx.h"
#include "UtfConverter.h"
#include "ConvertUTF.h"

namespace UtfConverter
{
    std::wstring FromUtf8(const std::string& utf8string)
    {
        size_t widesize = utf8string.length();
        if (sizeof(wchar_t) == 2)
        {
            wchar_t* widestringnative = new wchar_t[widesize+1];
            const UTF8* sourcestart = reinterpret_cast<const UTF8*>(utf8string.c_str());
            const UTF8* sourceend = sourcestart + widesize;
            UTF16* targetstart = reinterpret_cast<UTF16*>(widestringnative);
            UTF16* targetend = targetstart + widesize+1;
            ConversionResult res = ConvertUTF8toUTF16
		(&sourcestart, sourceend, &targetstart, targetend, strictConversion);
            if (res != conversionOK)
            {
                delete [] widestringnative;
                throw std::exception("La falla!");
            }
            *targetstart = 0;
            std::wstring resultstring(widestringnative);
            delete [] widestringnative;
            return resultstring;
        }
        else if (sizeof(wchar_t) == 4)
        {
            wchar_t* widestringnative = new wchar_t[widesize];
            const UTF8* sourcestart = reinterpret_cast<const UTF8*>(utf8string.c_str());
            const UTF8* sourceend = sourcestart + widesize;
            UTF32* targetstart = reinterpret_cast<UTF32*>(widestringnative);
            UTF32* targetend = targetstart + widesize;
            ConversionResult res = ConvertUTF8toUTF32
		(&sourcestart, sourceend, &targetstart, targetend, strictConversion);
            if (res != conversionOK)
            {
                delete [] widestringnative;
                throw std::exception("La falla!");
            }
            *targetstart = 0;
            std::wstring resultstring(widestringnative);
            delete [] widestringnative;
            return resultstring;
        }
        else
        {
            throw std::exception("La falla!");
        }
        return L"";
    }

    std::string ToUtf8(const std::wstring& widestring)
    {
        size_t widesize = widestring.length();

        if (sizeof(wchar_t) == 2)
        {
            size_t utf8size = 3 * widesize + 1;
            char* utf8stringnative = new char[utf8size];
            const UTF16* sourcestart = 
		reinterpret_cast<const UTF16*>(widestring.c_str());
            const UTF16* sourceend = sourcestart + widesize;
            UTF8* targetstart = reinterpret_cast<UTF8*>(utf8stringnative);
            UTF8* targetend = targetstart + utf8size;
            ConversionResult res = ConvertUTF16toUTF8
		(&sourcestart, sourceend, &targetstart, targetend, strictConversion);
            if (res != conversionOK)
            {
                delete [] utf8stringnative;
                throw std::exception("La falla!");
            }
            *targetstart = 0;
            std::string resultstring(utf8stringnative);
            delete [] utf8stringnative;
            return resultstring;
        }
        else if (sizeof(wchar_t) == 4)
        {
            size_t utf8size = 4 * widesize + 1;
            char* utf8stringnative = new char[utf8size];
            const UTF32* sourcestart = 
		reinterpret_cast<const UTF32*>(widestring.c_str());
            const UTF32* sourceend = sourcestart + widesize;
            UTF8* targetstart = reinterpret_cast<UTF8*>(utf8stringnative);
            UTF8* targetend = targetstart + utf8size;
            ConversionResult res = ConvertUTF32toUTF8
		(&sourcestart, sourceend, &targetstart, targetend, strictConversion);
            if (res != conversionOK)
            {
                delete [] utf8stringnative;
                throw std::exception("La falla!");
            }
            *targetstart = 0;
            std::string resultstring(utf8stringnative);
            delete [] utf8stringnative;
            return resultstring;
        }
        else
        {
            throw std::exception("La falla!");
        }
        return "";
    }
} 

How To Do It Better

Here's another version that avoids using new and delete, by writing directly into the string buffer. Does anyone know whether this is okay?

C++
#include "stdafx.h"
#include "UtfConverter.h"
#include "ConvertUTF.h"

namespace UtfConverter
{
    std::wstring FromUtf8(const std::string& utf8string)
    {
        size_t widesize = utf8string.length();
        if (sizeof(wchar_t) == 2)
        {
            std::wstring resultstring;
            resultstring.resize(widesize+1, L'\0');
            const UTF8* sourcestart = reinterpret_cast<const UTF8*>(utf8string.c_str());
            const UTF8* sourceend = sourcestart + widesize;
            UTF16* targetstart = reinterpret_cast<UTF16*>(&resultstring[0]);
            UTF16* targetend = targetstart + widesize;
            ConversionResult res = ConvertUTF8toUTF16
		(&sourcestart, sourceend, &targetstart, targetend, strictConversion);
            if (res != conversionOK)
            {
                throw std::exception("La falla!");
            }
            *targetstart = 0;
            return resultstring;
        }
        else if (sizeof(wchar_t) == 4)
        {
            std::wstring resultstring;
            resultstring.resize(widesize+1, L'\0');
            const UTF8* sourcestart = reinterpret_cast<const UTF8*>(utf8string.c_str());
            const UTF8* sourceend = sourcestart + widesize;
            UTF32* targetstart = reinterpret_cast<UTF32*>(&resultstring[0]);
            UTF32* targetend = targetstart + widesize;
            ConversionResult res = ConvertUTF8toUTF32
		(&sourcestart, sourceend, &targetstart, targetend, strictConversion);
            if (res != conversionOK)
            {
                throw std::exception("La falla!");
            }
            *targetstart = 0;
            return resultstring;
        }
        else
        {
            throw std::exception("La falla!");
        }
        return L"";
    }

    std::string ToUtf8(const std::wstring& widestring)
    {
        size_t widesize = widestring.length();

        if (sizeof(wchar_t) == 2)
        {
            size_t utf8size = 3 * widesize + 1;
            std::string resultstring;
            resultstring.resize(utf8size, '\0');
            const UTF16* sourcestart = 
		reinterpret_cast<const UTF16*>(widestring.c_str());
            const UTF16* sourceend = sourcestart + widesize;
            UTF8* targetstart = reinterpret_cast<UTF8*>(&resultstring[0]);
            UTF8* targetend = targetstart + utf8size;
            ConversionResult res = ConvertUTF16toUTF8
		(&sourcestart, sourceend, &targetstart, targetend, strictConversion);
            if (res != conversionOK)
            {
                throw std::exception("La falla!");
            }
            *targetstart = 0;
            return resultstring;
        }
        else if (sizeof(wchar_t) == 4)
        {
            size_t utf8size = 4 * widesize + 1;
            std::string resultstring;
            resultstring.resize(utf8size, '\0');
            const UTF32* sourcestart = 
		reinterpret_cast<const UTF32*>(widestring.c_str());
            const UTF32* sourceend = sourcestart + widesize;
            UTF8* targetstart = reinterpret_cast<UTF8*>(&resultstring[0]);
            UTF8* targetend = targetstart + utf8size;
            ConversionResult res = ConvertUTF32toUTF8
		(&sourcestart, sourceend, &targetstart, targetend, strictConversion);
            if (res != conversionOK)
            {
                throw std::exception("La falla!");
            }
            *targetstart = 0;
            return resultstring;
        }
        else
        {
            throw std::exception("La falla!");
        }
        return "";
    }
}

How to Use It

Just add it to your project. Download the Unicode converter from here and add that to the project, too. It should just work.

Of course, you can throw whatever exceptions you like upon failure.

I must admit I tried it only for 2-byte wchar_t.

Comments are welcome.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
rh_
Germany Germany
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

 
QuestionDo The Same Thing With C++11 STL Pin
Theo Buys8-Mar-15 22:45
Theo Buys8-Mar-15 22:45 
GeneralMy vote of 5 Pin
Stephan(Russland)27-Dec-11 19:44
Stephan(Russland)27-Dec-11 19:44 
Generalworking code for this article Pin
assaf raman13-Mar-11 14:15
assaf raman13-Mar-11 14:15 
AnswerOr The Same Thing In Four Lines ! Pin
megaadam24-Oct-10 23:55
professionalmegaadam24-Oct-10 23:55 
GeneralRe: Or The Same Thing In Four Lines ! Pin
Michael B Pliam7-Dec-11 14:46
Michael B Pliam7-Dec-11 14:46 
GeneralRe: Or The Same Thing In Four Lines ! Pin
megaadam7-Dec-11 22:49
professionalmegaadam7-Dec-11 22:49 
GeneralRe: Or The Same Thing In Four Lines ! Pin
Michael B Pliam8-Dec-11 7:49
Michael B Pliam8-Dec-11 7:49 
GeneralRe: Or The Same Thing In Four Lines ! Pin
megaadam9-Dec-11 2:14
professionalmegaadam9-Dec-11 2:14 
GeneralRe: Or The Same Thing In Four Lines ! Pin
Michael B Pliam9-Dec-11 10:54
Michael B Pliam9-Dec-11 10:54 
GeneralNO No No! Re: Or The Same Thing In Four Lines ! Pin
Theo Buys18-Feb-15 3:42
Theo Buys18-Feb-15 3:42 
GeneralZipfile code not working (later code does work, though) Pin
babzog5-May-10 10:26
babzog5-May-10 10:26 
GeneralThe easiest way to do the same conversion Pin
steveb23-Oct-08 9:21
mvesteveb23-Oct-08 9:21 
GeneralRe: The easiest way to do the same conversion Pin
kurt.griffiths9-Nov-11 9:09
kurt.griffiths9-Nov-11 9:09 
Generaltrouble appending string to conversion Pin
Jonathan Davies18-Oct-08 2:53
Jonathan Davies18-Oct-08 2:53 
GeneralIncorrect size set in ToUTF8 Pin
DEmberton19-Jun-08 2:55
DEmberton19-Jun-08 2:55 
GeneralRe: Incorrect size set in ToUTF8 Pin
peterchen21-Aug-08 4:47
peterchen21-Aug-08 4:47 
GeneralRe: Incorrect size set in ToUTF8 Pin
Vite Falcon22-Apr-11 10:19
Vite Falcon22-Apr-11 10:19 
Generalthank you Pin
wipehindy15-Feb-08 12:59
wipehindy15-Feb-08 12:59 
Questionwhat is “L” in:resultstring.resize(widesize+1, L'\0'); Pin
Eva ranee6-Jan-08 20:34
Eva ranee6-Jan-08 20:34 
GeneralRe: what is “L” in:resultstring.resize(widesize+1, L'\0'); Pin
Mircea Puiu6-Jan-08 23:41
Mircea Puiu6-Jan-08 23:41 
GeneralRe: what is “L” in:resultstring.resize(widesize+1, L'\0'); Pin
Member 1305757813-Mar-17 23:21
Member 1305757813-Mar-17 23:21 
GeneralUNICODE is not the same as UTF16 PinPopular
christophe.hermier@quickfds.com30-Sep-07 21:55
christophe.hermier@quickfds.com30-Sep-07 21:55 
GeneralRe: UNICODE is not the same as UTF16 Pin
Theo Buys19-Feb-15 1:04
Theo Buys19-Feb-15 1:04 
GeneralCA2T, CA2W Pin
kpnut30-Aug-07 4:35
kpnut30-Aug-07 4:35 
If this is Windows, wouldn't CA2W, CA2T etc do what you need?

Ken Davis
QuestionIs this OK? Pin
Stephen Hewitt20-May-07 21:12
Stephen Hewitt20-May-07 21:12 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.