Locale support in the Visual Component Framework

• Download VCF source (tar.gz) - 6.4 MB
• Download VCF Windows Installer - 29 MB
• Download LocalesUI example source - 16.4 Kb
• Download localization example source - 25.6 Kb

Introduction

You may find that as your application progresses, you want to add support for additional languages other than your native language that you're developing the application in. This can be somewhat tricky to manage, depending on what you want to accomplish, and how deep you want to support localization within your application.

Fortunately, the Visual Component Framework makes it easy to add locale support to your application by using the VCF's Locale class. The Locale class is used to represent a specific country, language, and region. It has a series of functions that deal with string handling, conversion, and various other locale-centric services.

The Locale class is designed to use the underlying operating system's locale features and services rather than re-invent the wheel. This, in turn, makes your application more closely conform to what users expect on the platform. Both Windows and OSX provide solid locale and internationalization features in their OS APIs so this is not a problem. For operating systems that lack this level of support, we will use the ICU library from IBM. In other words, for Win32 code, we simply call directly to the National Language Support (NLS) APIs like GetNumberFormat(), GetDateFormat(), and so on.

The reason for developing the Locale class instead of relying on C++'s std::locale is that C++'s locale support is not very effective, and not entirely implemented, since parts of the STL specification for it leaves it up to vendors to implement. For example, the std::locale messages category, which is intended for translating a source message to a string in a different language, is useless (i.e., non-functional) on Win32, OS X, and, the last time I checked, Linux as well. This makes it unpredictable to use, and difficult to count on any functionality from platform to platform or even from vendor to vendor within a platform. In addition, one could make a convincing argument that the C++ locale classes are not exactly intuitive or easy to use.

Features

The VCF's Locale class supports string collation, both case sensitive and case insensitive, various types of string conversions (such as converting an integer to a string, and converting a string back to a number), and date/time conversion to a string. The Locale class also makes use of the current OS User settings for locales. In addition to various comparison and conversion/parsing routines, the Locale class also supports translating a source string to that of a specific locale.

The translation feature has default support for a specific kind of text based translation file, but it also has support for custom translation routines, so that one could write code for re-using existing translation file formats, such as the .PO format, a standard for storing localized translations.

The Locale class is also integrated into the VCF's UI classes, which allow for automatic translation of UI strings, such as a menu item's caption, or a command button's text. It's also relatively easy to add support for this in your own custom controls as well.

String Handling

The Locale class makes use of the VCF String class for dealing with text. The VCF's String class is a very thin wrapper around std::basic_string<> and stores the string data as unsigned shorts, in UTF-16 format, the same format that both Win32 and Mac OS X store their Unicode strings in. When transforming between the native UTF-16 and ANSI string formats, the String class, by default on Win32, uses the native MultiByteToWideChar and WideCharToMultiByte functions. It's also possible to use your own text codecs but this requires an explicit call to do so. Since the internals of the VCF use strings for all string handling, any VCF program will call the native wide string API functions. If the VCF detects it is running on a non NT based system (i.e., Win9X), it will convert the string to ANSI and call the appropriate ANSI function. This means you don't have to fuss with worrying about whether UNICODE or MBCS is defined, and the same executable will run correctly on either system. This means that when running on NT, you're effectively running with out the ANSI -> Unicode conversion penalty, just as if you'd made a UNICODE build of your program.

Usage

The Locale class is meant to be easy to use. You simply create a new instance using standard country and language codes, or you get the current thread's locale. The current thread's locale will have the same settings that the user has chosen for his or her language of choice.

Locale locEnUS( "en", "US" );
//use the locale...

The language code is one of the standard ISO-639 codes. The country code is one of the standard ISO-3166 codes.

Alternately, you can create a locale from one of the various language/country enumerations:

Locale locEnUS( Locale::lcEnglish, Locale::ccUnitedStates );
//use the locale...

Getting the current thread's locale:

Locale* locale = <A href="http://vcf-online.org/docs/src_manual/classVCF_1_1System.html#63cdc0fbffbffc6438035fe21422e308" target=_blank>System::getCurrentThreadLocale()</A>;

Collation

Once you have a valid locale instance, we can start to perform operations with it. Let's look into collating strings. Collating strings simply means sorting them in a specific order. Typically, if you had a list of strings, "hello", "goodbye", "123", and "ascrobotic", and wanted to sort them alphabetically, then you could do this in one of two ways:

• use the built in "<" and ">" operators for strings.
• use the locale's collate functions.

The former might be OK for English text, but it won't order things correctly for other languages, so we need to resort to the second method. The Locale class allows you to sort with or without case sensitivity factored in. To sort two strings with a locale taking case into consideration, we would use:

String s1 = "Hello";
String s2 = "hello";
Locale* locale = System::getCurrentThreadLocale();
int res = locale->collate( s1, s2 );

Locale::collate will return -1 if s1 is "less" than s2 in sort order, 0 if they are considered equivalent, and 1 if s1 is considered "greater" than s2.

You might sort a list of strings like so (using some STL to help us out):

class MySort {
public:
 MySort(Locale* l) :locale(l){}

 bool operator() ( const String& x, const String& y ) {
       return locale->collate( x,y) >= 0;
 }

 Locale* locale;
};

std::vector<String> strings(4);
strings[0] = "Hello";
strings[1] = "Asdf";
strings[2] = "1233";
strings[3] = "VCF";

std::sort( strings.begin(), strings.end(), 
           MySort( System::getCurrentThreadLocale() ) );

The same thing can be done using case insensitive collation, just call the Locale::collateCaseInsensitive() function.

For those wondering how this works, the locale peer implementation, on Win32 platforms, ends up calling the CompareString API function.

Symbols

You can get access to various symbols, such as the symbol for currency, the symbol for separating numbers, and so on. Each symbol is represented by a string and may contain Unicode characters. Note that when trying to display these characters on a console, you may see a strange symbol and not what you expect. Let's look at how we might access the symbols:

Locale locItalian( Locale::lcItalian, Locale::ccItaly );
System::println( "100's separator: " +  locItalian.getNumberThousandsSeparator() );
System::println( "Decimal point: " +  locItalian.getNumberDecimalPoint() );
System::println( "Decimal point: " +  locItalian.getCurrencySymbol() );

Conversion

We can convert various data types to a string using the Locale class by calling the toString() function. The basic types supported are:

• int
• unsigned int
• long
• unsigned long
• double
• float
• double evaluated as currency

All of these conversion functions will take into consideration the locale symbols, and various number groupings that may apply. This includes any specific user settings for the locale. For example:

Locale* locale = System::getCurrentThreadLocale();
String s = locale->toString( 123456789 );
System::println( s );

This should output (assuming default settings) the string "123,456,789". To convert a currency value to a string, you would use the Locale::toStringFromCurrency() function. This will format the string according to the currency rules for the locale. For example:

Locale* locale = System::getCurrentThreadLocale();
String s = locale->toStringFromCurrency( 567432645.9883 );
System::println( s );

Assuming a locale of "en-US" with default user settings, this will output "$567,432,645.99" on Windows.

We can convert a string to all upper case or all lower case using the formatting rules of the locale. To do so, we just call the Locale::toLowerCase() or Locale::toUpperCase(). Both functions will return a new string that's been converted as necessary.

It's also possible to parse a string and convert this to a primitive type. Locale supports converting to int, unsigned int, float, or double types. The parsing is done according to locale rules enforced by the OS. For example:

Locale* locale = System::getCurrentThreadLocale();
int num = locale->toInt( "1,000,023" );

If the string value is a currency, this can also be parsed:

Locale* locale = System::getCurrentThreadLocale();
double moneyVal = locale->toInt( "$ 1,000,023.89" );

Locale Identification

Locales can be identified by four different values. The Locale name is the combination of the language and country code, such as "en_US" or "it_IT". You can retrieve this value by calling Locale::getName(). The functions Locale::getLanguageCodeString() and Locale::getCountryCodeString() return the locale's language and country codes, respectively. To retrieve the locale's human readable language name, you can call Locale::getLanguageName(), which will return things like "English" or "Italian". At the moment, this name is pulled by calling the GetLocaleInfo using the LOCALE_SLANGUAGE parameter. This is supposed to return a localized string for the language name.

Message Translation

The last key feature of the Locale class is translating a string ID into its localized version. You might have the string "Hello" and wish it to be translated (or localized) to an appropriate string for the locale, for example:

Locale loc1("fr", "FR");
String s1 = loc1.translate( "Hello" );

Locale loc2("it", "IT");
String s2 = loc2.translate( "Hello" );

where s1 will be "Salute" and s2 will be "Ciao". The translation "magic" happens because a special file (or files) exists, one for each locale supported, that maps the ID, or "key" (in this case, "Hello"), to a corresponding equivalent value for the specific locale. This file is loaded by a special class called a MessageLoader which understands the specific file format and can parse and extract the value for the specific key.

Multiple MessageLoader instances may be registered for different extensions allowing you to add custom support for other translation file types. For example, if you had translation files in the PO format, you could write a custom MessageLoader for this format and re-use your existing translation files. Currently, the default translation format is the ".strings" format, which is the same as Mac OS X uses, and is very easy to read, write, and parse.

The ".strings" format is quite simple. It is a plain text format that consists of a key and a value. Each key name is unique, and is enclosed in double quotes ('"'). The value may be any string you wish so long as it is enclosed in quotes. Unicode characters may be represented by a "\" character and a "U" character followed by a four digit number in hexadecimal format (such as "\U010F"). Comments are written using C style "/*" to start and "*/" to end. Comments may be nested.

A sample file:

/*
Spanish localization test file
/**
nesting comments
*/

*/

"Hello" = "Hola" /*this is Hello in spanish*/

"I understand" = "Yo comprendo"

.strings format in BNF notation (roughly):

strings-file ::=
       (string-entry)*
string-entry ::=
       key '=' value
key ::=
       '"' (char | uni-char)+ '"'
value ::=
       '"' (char | uni-char)+ '"'
uni-char ::=
       '\' 'U' (0-9,A-F,a-f)+4

The mechanism for performing the translation is as follows:

• The call to Locale::translate() is made.
• The framework determines the resource directory for the executable.
• The framework uses the locale name as the subdirectory to look in.
• The translation file name is determined.
• If the translation file exists, the framework determines the MessageLoader to use.
• The MessageLoader instance is used to load the translation file (see MessageLoader::loadMessageFile()).
• The message for the given ID is extracted by the message loader instance (see MessageLoader::getMessageFromID()).
• If the result for the translation is still an empty string at this point, then the result returned is the original ID value passed in.

To actually translate a string is trivial:

Locale loc("it", "IT");
String s = loc.translate( "Hello" );

Assuming you have the following entry in your polish .strings file: "The file %s cannot be found." = "Plik %s nie znaleziony."

You can use format symbols in conjunction with the Format class like so:

String fileName = "Income2005.xls";
Locale loc("pl", "PL");
String s = Format( loc.translate( "The file %s cannot be found." ) ) % fileName;

And you should end up with the string: "Plik Income2005.xls nie znaleziony."

Locale Usage in the User Interface

While the Locale class can be used independently of the UI classes, since it's part of the FoundationKit library, there is support for locale sensitive text rendering, and support for string translation in various UI elements.

To support locales in the GraphicsKit (the library that handles all the core drawing functionality), you can specify a specific locale instance for a given Font instance. By default, the Font has a null locale. On Windows, prior to drawing text, the current font's locale value is checked; if it is NULL then the current thread's locale is used. The LCID is extracted, and based on this, the LOGFONT's character set is determined. If no character set can be determined from the LCID, the DEFAULT_CHARSET value is used. This ensures that the font will be rendered correctly. This does assume that the user has correct fonts on their system that can support Unicode characters.

All of this allows us to either use the default locale, or to override this and dynamically control what locale to use. For example:

<A href="http://vcf-online.org/docs/src_manual/classVCF_1_1GraphicsContext.html">GraphicsContext</A>* ctx = ....//get graphics context
Locale loc("pl", "PL");
ctx->getCurrentFont()->setLocale( &loc );
ctx->textAt( 100, 100, "Czecs" );  //"Hello" in Polish

This will draw the text "Czecs" correctly, including the accented characters.

The ApplicationKit (the library that provides UI functionality) uses the Locale class to provide translation support for various UI classes like controls, window captions, and menu items. This means that you can set the caption of a control (like a CommandButton, or a Label), and the framework will lookup the translation for the text at runtime. You can optionally turn off this automatic behavior as well, if it's not something you want to happen. This means that all you need to do is supply a locale specific translation file and, for most of the UI elements, the display of the localized text will happen without you writing any extra code.

Adding Locale Support to Custom Controls or Components

If you write a custom control, you may want to provide built-in support for locales like other controls do. Doing so is quite easy. Let's build a little control that displays the date and time, and make sure it's localized for display. First, let's create the control class:

class DateTimeLabel : public <A href="http://vcf-online.org/docs/src_manual/classVCF_1_1CustomControl.html" target=_blank>CustomControl</A> {
protected:
       <A href="http://vcf-online.org/docs/src_manual/classVCF_1_1TimerComponent.html">TimerComponent</A>* timer;
       String extraTxt;
       Locale* locale;
       void onTimer( Event* e ) {
               repaint();
       }
public:

       DateTimeLabel() : CustomControl(false),locale(NULL){

       setBorder( new <A href="http://vcf-online.org/docs/src_manual/classVCF_1_1TitledBorder.html">TitledBorder</A>() );

       timer = new TimerComponent(this);
       timer->setTimeoutInterval ( 1000 );
       timer->TimerPulse +=
              new GenericEventHandler<DateTimeLabel>(this, 
              &DateTimeLabel::onTimer, 
              "DateTimeLabel::onTimer" );
       }

       virtual ~DateTimeLabel() {
               delete locale;
       }

       void setLocale( Locale* loc ) {
               if ( NULL != locale ) {
                       delete locale;
               }

               locale = new Locale( loc->getLanguageCode(), 
                                    loc->getCountryCode() );

               TitledBorder* border = (TitledBorder*)getBorder();
               border->setCaption( locale->getLanguageName() );
               border->getFont()->setLocale( locale );
       }

       void setLocale( const String& lang, const String& country ) {
               if ( NULL != locale ) {
                       delete locale;
               }

               locale = new Locale( lang, country );

               TitledBorder* border = (TitledBorder*)getBorder();
               border->setCaption( locale->getLanguageName() );
               border->getFont()->setLocale( locale );
       }

       void start() {
               timer->setActivated ( true );
       }

       void stop() {
               timer->setActivated ( false );
       }

       void setExtraTxt( const String& val ) {
               extraTxt = val;
               repaint();
       }

       String getExtraTxt() {
               return extraTxt;
       }

};

This is all you need to create your control. We've added a member String that holds some extra text. We've also added a member that points to a custom Locale instance that we keep track of and allow the user to change at will. Finally, we have a timer component that fires off every second and repaints the control so that we can display the current, localized, date and time.

We add two functions to let us control the timer - stop() stops the timer component from firing, and start() starts the timer component timer events.

However, it doesn't yet draw itself. For that, we need to override the Control's paint() function. Once we've done that, then we can just create a window and add the control to the window.

Let's look at our custom paint function:

class DateTimeLabel : public CustomControl {
protected:
       TimerComponent* timer;
       String extraTxt;
       Locale* locale;
       //rest omitted

       virtual void <A href="http://vcf-online.org/docs/src_manual/classVCF_1_1Control.html#7f2cce53b6821bed9cefbe03b71cc7ef">paint</A>( GraphicsContext* ctx ) {

       <A href="http://vcf-online.org/docs/src_manual/classVCF_1_1Control.html#7f2cce53b6821bed9cefbe03b71cc7ef">CustomControl::paint( ctx )<A>;

       //get the current locale
       Locale* currentLocale = System::getCurrentThreadLocale();

       if ( NULL != locale ) {
               currentLocale = locale;
       }

       String localizedExtra = extraTxt;

       //check if we can localize the string
       if ( getUseLocaleStrings() ) {
               //Yep! Let's get the localized version. 
               //Worst case scenario is that
               //no translation exists, which means 
               //localizedExtra will be the same
               //as extraTxt
               localizedExtra = currentLocale->translate( extraTxt );
       }


       DateTime dt = DateTime::now();
       //localize the date/time value into a string
       String dateStr = <A href="http://vcf-online.org/docs/src_manual/classVCF_1_1Locale.html#52812049ad3abdec2f1bcc9e74e4bb40">currentLocale->toStringFromDate</A>( dt, 
                        "dddd, MMM d yyyy" );
       String timeStr = <A href="http://vcf-online.org/docs/src_manual/classVCF_1_1Locale.html#50f6ce4dd4109307e8e565b6bebb5be0">currentLocale->toStringFromTime</A> ( dt );


       ctx->getCurrentFont()->setName( "Times New Roman" );
       ctx->getCurrentFont()->setPointSize( 16 );

       Rect r = getClientBounds();

       Rect xtraRect = r;
       xtraRect.bottom_ = xtraRect.top_ + 
                          ctx->getTextHeight( localizedExtra );

       long textDrawOptions = GraphicsContext::tdoCenterHorzAlign;

       ctx->textBoundedBy( &xtraRect, 
                              localizedExtra, textDrawOptions );

       Rect textRect = r;

       textRect.inflate( -10, -10 );
       textRect.top_ = xtraRect.bottom_;

       ctx->getCurrentFont()->setBold( true );

       textDrawOptions = GraphicsContext::tdoWordWrap | 
                         GraphicsContext::tdoCenterHorzAlign;
       ctx->textBoundedBy( &textRect, dateStr + 
                              "\n" + timeStr, textDrawOptions );
   }
};

We first call the super class' paint (CustomControl::paint(ctx)) to ensure that the basic paint operations take place (like painting the background). Then, we determine the current locale we are going to use. Next, we translate our "extra" string and get the localized string for the current date and time. We then calculate two rectangles: one for the extra text to be drawn in, and another for the date/time text to be drawn in. The actual drawing of the text takes place by calling the GraphicsContext::textBoundedBy() function.

Now that we have our control, we can make use of it by creating a simple top level frame window, and then adding multiple instances of it. Let's look at that code:

Window* window = new Window();

DateTimeLabel* label;

label = new DateTimeLabel();

label->setLocale( "en", "US" );

label->setExtraTxt( "Hello it's:" );

label->setHeight( 100 );

window->add( label, AlignTop );

label->start();


label = new DateTimeLabel();

label->setLocale( "it", "IT" );

label->setExtraTxt( "Hello it's:" );

label->setHeight( 100 );

window->add( label, AlignTop );

label->start();


label = new DateTimeLabel();

label->setLocale( "pl", "PL" );

label->setExtraTxt( "Hello it's:" );

label->setHeight( 100 );

window->add( label, AlignTop );

label->start();


label = new DateTimeLabel();

label->setLocale( "de", "DE" );

label->setExtraTxt( "Hello it's:" );

label->setHeight( 100 );

window->add( label, AlignTop );

label->start();


label = new DateTimeLabel();

Locale loc( Locale::lcJapanese, Locale::ccJapan );
label->setLocale( &loc );

label->setExtraTxt( "Hello it's:" );

label->setHeight( 100 );

window->add( label, AlignTop );

label->start();

label = new DateTimeLabel();

Locale loc2( Locale::lcRussian, Locale::ccRussianFederation );
label->setLocale( &loc2 );

label->setExtraTxt( "Hello it's:" );

label->setHeight( 100 );

add( label, AlignTop );

window->label->start();

window->setBounds( 100.0, 100.0, 350.0, 620.0 );
window->show();

We now have six different instances of our custom control, looking something like this:

You'll notice that our extra text, "Hello it's:", is not being translated yet. We need to add our translation files to the application. For that, we simply create a directory named "Resources" at the same level as our executable. Then, we create subdirectories under the "Resources" directory, one for each locale we want to support. In this case, we'll create directories named "de_DE", "it_IT", "pl_PL", and "ru_RU" for German, Italian, Polish, and Russian locales.

We then create a .strings file, one per locale, and place it in each sub directory. The name of the file must be the name of the executable plus the ".strings" extension. The contents of the files will look something like this:

file name: Resources/de_DE/LocaleUI.strings
/*German*/
"Hello it's:" = "Hallo ist es:"

file nam lang=texte: Resources/ru_RU/LocaleUI.strings
/*Russian*/
"Hello it's:" = "\U0417\U0434\U0440\U0430\U0432\U0441\
                  U0442\U0432\U0443\U043B\U0442\
                  U0435! \U043E\U043D\U043E:"

file name: Resources/it_IT/LocaleUI.strings
/*Italian*/
"Hello it's:" = "Ciao è:"

file name: Resources/pl_PL/LocaleUI.strings
/*Polish*/
"Hello it's:" = "Cze\U0107\U015B to jest:"

Note that the translations I am providing here may not be that accurate - I got the German and Russian translations from Babelfish. My apologies to the German and Russian speakers out there.

With these files in place, we now see the effects of the translated text!

Notes on Building the Examples

You'll need to have the most recent version of the VCF installed (at least 0-9-0 or better), and you'll need to make sure you have built the static libraries for the VCF (as opposed to the DLL version). The examples are configured to link to the VCF statically. For more documentation on building the VCF, see: Building the VCF, at the VCF online documentation.

Conclusion

We've covered pretty much all the basics for working with locales in the VCF, and the various features and functions of the Locale class. There are some advanced locale issues that we haven't covered, such as custom numerical string parsing or formatting. That may be added in future releases of the VCF, but is not currently supported.

However, we did see the ability to extract and convert basic types to and from a string, get locale information, translating strings, and then making use of all of this in the user interface.

Questions about the framework are welcome, and you can post them either here, or in our forums. If you have suggestions on how to make any of this better, we'd love to hear them!