How to Create a Handwriting to Text Control (Ink Recognizer)

When building a (multi)touch application, you may need one nice feature: translate hand-written text to real words. This opens a whole new world full of possibilities like starting some actions when keywords are recognized or simply allow the users to write some text for later use.

In this post, we'll see all the steps to create a handwriting to text control and how to tune it.

Specifications

The HandWritingToText controls translate the text written with its hand by the user into "computer text". The final purpose is to trigger some actions when a specific keyword is recognized and it'll so be able to recognize only one word at a time and not a whole text.

1. The recognized text will be published via an event and available trough a DependencyProperty
2. The control will be fully customizable via properties
3. It is delivered into an assembly which can be used in any WPF project

We will follow all the tips and tricks given in this previous post: How to create your own control library.

Creating the Hand Writing to Text Control

Requirements

The first thing we need is the engine which will translate our hand written text to a real text. This tool is made available to us into the “IAWinFx dlls” of the Windows SDK. Starting from here, I had a hard time finding all of them because they are placed in a lot of different folders. You have to install the FULL windows SDK with all the samples for all the languages or you won’t be able to find them all. The files that are seeked are:

1. IACore.dll
2. IALoader.dll
3. IAWinFX.dll
4. Microsoft.Ink.Analysis.dll
5. Microsoft.Ink.JournalReader.dll

After adding these DLLs as a reference to your project, you have to do a little configuration. As they are compiled for an old version of the runtime and that we are using the 4.0, we need to add this snippet in our App.config file:

<startup useLegacyV2RuntimeActivationPolicy="true">
<supportedRuntime version="v4.0" />
</startup>

Build Up the Visual

The template of our control is pretty simple because it will be constituted of one single element: an InkCanvas. This control is a Template part that we name PART_theInkCanvas. An InkCanvas is simply an area that receives and displays ink strokes.

Here is the content of our control theme file:

<ResourceDictionary 
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:handWriting="clr-namespace:AmazingsWPFControls.HandWritingToText">
<!-- ================================================================= -->
<!-- HandWritingToText                                                               -->
<!-- ================================================================= -->
<Style x:Key="{x:Type handWriting:HandWritingToText}"
TargetType="{x:Type handWriting:HandWritingToText}">
<Setter Property="Template">
<Setter.Value>
<ControlTemplate TargetType="{x:Type handWriting:HandWritingToText}">
<InkCanvas x:Name="PART_theInkCanvas" Background="Transparent" />
</ControlTemplate>
</Setter.Value>
</Setter>
</Style>
</ResourceDictionary>

You may only want to recognize the words and not display what the user writes: To do so, you’ll only have to set the strokes color to Transparent. But this is not as simple because you have to set the property on the DrawingAttributes of the inkCanvas which is not bindable because this is not a DependencyObject. So the solution I found is to add a dependency property “StrokeColor” on the control, and to set via the code-behind the color on each change of this property (the color must also be set when the control is loaded). The code is very classic and I let you dig in the source if you want to see it !

Recognize Hand Writing

Then comes the funny part: analyze the drawn strokes and find the real words behind them! It’s here that the IAWinFX DLLs take all their meaning by providing the InkAnalyser class. This is a very nice tool which analyzes the strokes you feed it with and returns you what it recognize.

It’s pretty straightforward to instantiate an InkAnalyser because no parameters are needed. But we need to set some parameters for its algorithm to make it work better:

• The AnalysesMode which is how the ink analyzer behaves before, after and during the analysis. In our case, we want it to clear the cache automatically and to start the reconciliation process right after it had finished his recognition.
• The analysis hint which is information we give to the analyzer to facilitate its guess. Each information is called a Factoid. In our case, the hint will be applied to the whole inkCanvas (a hint can be defined to a restricted zone if wanted), we’ll be looking for words only (no full sentence) and the analyzer will limit its analysis of ink within the hint's area to conform to the hint's Factoid property. A complete list of the different factoids available can be found here.

//Create the ink analyzer
theInkAnalyzer = new InkAnalyzer();
theInkAnalyzer.AnalysisModes = 
AnalysisModes.StrokeCacheAutoCleanupEnabled | 
AnalysisModes.AutomaticReconciliationEnabled;
theInkAnalyzer.ResultsUpdated += 
	new ResultsUpdatedEventHandler(theInkAnalyzer_ResultsUpdated);
var hint = theInkAnalyzer.CreateAnalysisHint();
hint.Location.MakeInfinite();
hint.WordMode = true;
hint.CoerceToFactoid = true;

Feed the Ink Analyzer with the Strokes

To get the strokes entered by the use, we only have to get the InkCanvas from the template in the OnApplyTemplate methods and subscribe to its StrokeCollected event. Then in its handler, we’ll be able to feed the ink analyser.

But wait… If we let the user play with the component and write whatever she/he wants to, the ink analyzer won’t be able to recognize any pattern: it will only be a child drawing for her/him!

So the solution I found is to define a maximum time for the user to write the keyword. This time is fully customizable and exposed as a DependencyProperty named TimeToEnterText. If this duration is elapsed, we clear the ink analyzer of all the strokes it possessed and start feeding it with the new ones:

public override void OnApplyTemplate()
{
base.OnApplyTemplate();
theInkCanvas = 
      base.Template.FindName("PART_theInkCanvas", this) as InkCanvas;
if (theInkCanvas == null)


    throw new ArgumentException("Cannot find the PART_theInkCanvas InkCanvas.");
theInkCanvas.StrokeCollected += 
      new InkCanvasStrokeCollectedEventHandler(theInkCanvas_StrokeCollected);
}
void theInkCanvas_StrokeCollected(object sender,
       InkCanvasStrokeCollectedEventArgs e)
{
//Check if the time has passes before reset
if (DateTime.Now – lastResetOfInkCanvas 
             > TimeSpan.FromMilliseconds(TimeToEnterText))
{
try
{
theInkAnalyzer.RemoveStrokes(theInkCanvas.Strokes);
}
catch (Exception)
{
Debug.WriteLine(e);
throw;
}
//Remove old strokes
this.theInkCanvas.Strokes.Clear();
//Add the last one added
theInkCanvas.Strokes.Add(e.Stroke);
lastResetOfInkCanvas = DateTime.Now;
}
theInkAnalyzer.AddStroke(e.Stroke);
//Launch the analysis
theInkAnalyzer.BackgroundAnalyze();
}

How to Get the Results of the Analysis ?

Everything takes place in the handler of the InkAnalyzer ResultsUpdated event. The argument of type ResultsUpdatedEventArgs is first used to find if the analysis is successful by using its e.Status.Successful property. If there are some results, we then have to get the inkAnalyzer to get them.

The results are available in the form of a collection of a base class: ContextNode. Here is a list of the different classes inheriting from the ContextNode:

What they represent is each time quite clear and the most important are maybe these three:

• InkWordNode: Represents a collection of strokes that make up a logical grouping that forms a recognizable word. The text can be found via the GetRecognizedString method.
• InkDrawingNode: Represents a ContextNode for a collection of strokes that make up a drawing. For example, it can represent a Rectangle, a Circle or any geometric shape. The shape can be found via the GetShape method on an instance.
• LineNode: Represents a ContextNode for a line of words.

In the event handler, we iterate through the results and raise an event for each word found. The event is a routed one that we have declared in the control named TextEnteredEvent:

void theInkAnalyzer_ResultsUpdated(object sender, ResultsUpdatedEventArgs e)
{
if (e.Status.Successful)
{
ContextNodeCollection nodes = ((InkAnalyzer)sender).FindLeafNodes();
foreach (ContextNode node in nodes)
{
if (node is InkWordNode)
{
InkWordNode t = node as InkWordNode;
string recognizedString = t.GetRecognizedString();
RaiseTextEnteredEvent(recognizedString);
}
//For demonstration purpose only
else if (node is InkDrawingNode)
{
InkDrawingNode d = node as InkDrawingNode;
Shape shape = d.GetShape();
//Shape may be null here...
}
}
}
}

Additional Thoughts

One more thing I faced in testing this control: it worked very nicely on my laptop but it was not recognizing a word on another of my home computers. It took me a little time to figure out what was the problem: one computer was in English and the other was in French.

But don’t worry, you can solve it and very easily. When you add a stroke in the analyzer, you can customize it a little and one of the customizations is to set the language code of the draw! So if you face the same problem as me, you can use this snippet (which is also specifying that the strokes are forming a word and not anything else):

theInkAnalyzer.AddStroke(e.Stroke);
theInkAnalyzer.SetStrokeType(e.Stroke, StrokeType.Writing);
theInkAnalyzer.SetStrokeLanguageId(e.Stroke, 0x09);//Use EN-US languageID

Clean Up Your Room!

Using the ink analyzer can lead to memory leaks and freeze when exiting the application so you have to be careful to clean-up all the used resources. To do so, I make the control implement the IDisposable interface and call in it a method named ClearResources.

I also subscribe to the Application.Exit event if it exists to call the same method in the handler:

/// <summary>
/// Clears all resources : abort the current analyses, call dispose , ...
/// </summary>
private void ClearAllRessources()
{
lock (_locker)
{
if (!_isDisposed)
{
theInkAnalyzer.Abort();
theInkAnalyzer.Dispose();
_isDisposed = true;
}
}
}

How to Use It In Your Application?

Here are the XXX steps to use it in your application:

1. Add the amazing control libs to your project as a reference
2. Define the XML Namespace amazingControls: http://blog.lexique-du-net.com/wpf/AmazingWPFControls
3. Add the control in your visual tree
4. Register to the TextEnteredEvent or bind yourself to the LastRecognizedWord property of the control

<AmazingControls:HandWritingToText
xmlns:AmazingControls="http://blog.lexique-du-net.com/wpf/AmazingWPFControls"
TextEntered="HandWritingToText_TextEntered" StrokeColor="Red"
Height="450" Background="White" />

Where to Find the Resulting Control ?

You can find it on codeplex in the amazing WPF controls project:

• http://amazingwpfcontrols.codeplex.com/

(The post announcing the creation of this codeplex project can be found here.)

Some Links You May Find Useful

• Find New Meaning In Your Ink With Tablet PC APIs In Windows Vista
• List of available Factoid for you analysis hints
• MSDN page on digital ink