# SBlackboard

By , 12 Nov 2013
Votes of 3 or less require a comment

Introduction

SBlackboard (Smart Blackboard) is an app to teach your kids the alphabet and simple math operations by touching the screen or using a pen. The app will fall into Tablet: Education category of the contest.

It seems that kids are operating electronic devices such as smartphones or tablets practically moths after being born. Let's use the technology as a source of learning and entertainment. We don't have pencils and paper everywhere so why don't we use our tablet to teach our kids the letters of the alphabet and perform some simple math operations. That sounds good but... you may be wondering where the innovation is.

The innovation will be on using a uni/multi stroke recognition algorithm for detecting patterns and use a gesture based user interface.

## Functionality and Background

The application will teach kids the letters of the alphabet by doing exercises like:

• Repeat the pattern shown in animation for letter b. Try to follow the same gesture with the finger/pen. Just like Simon Says but with letters!!
• Lets listen a riddle about a letter. Guess the letter by drawing it with finger/pen. If your kid has problems with the riddle, let them listen the letter directly or see the gesture animation.  After evaluating the functionality, text-to-speech api (selected to scale the exercises) sounds awkward for children. Drop it.
The app will teach some basic math operations (Do the sum, do the rest, multiply these numbers...). It will present in big letters the quest and the kid must draw/touch the solution. For example, the app will show 2 + 2 and the kid must draw the number 4 to pass the exercise.

To sum up, the app will have this functionalities:

• Touch gestures recognition (Uni/multi stroke)
• Alphabet letter gesture log
• Numbers gesture log
• Math exercises
• Calligraphy exercises

## The app

The app is very simple and designed for children. The main window allow us to select between the different types of exercises: Letters, Numbers and Math

The exercises are pretty straightforward. For letters, the kids have to guess the hidden letter by drawing it on the canvas. There is a hint (image) on the upper right part of the screen.

For Numbers, the children must count the number of elements in the image and draw/touch the answer.

Basic Math operations is very easy as well. Do the operation, draw the answer and continue if you did well

The children select the exercise/game and try to guess the letters and the numbers by drawing them on the canvas. The app will try to recognize the gesture as a valid answer and in case of success the children will hear an fanfare and the app will let them continue with the next challenge. If they fail, a buzzer will sound and they can keep trying to guess the letter/number.

## Show me the code

The main sections for the app (regarding technical stuff) are:

• The gesture recognizer
• The exercises backend

The exercises backend is based on a simple text file where I describe the different exercises that will be available for the app. It is in json format. This way I can Serialize/Deserialize easyly.

```{Exercises:[{"Type":"Math",
"Image":"",
"Quest":"3 + 4 = __",
"Result":"7"
},
{"Type":"Letters",
"Image":"Exercises/Letters/bat.png",
"Quest":"B __ T",
"Result":"A"
},
{"Type":"Numbers",
"Image":"Exercises/Numbers/candies_6.png",
"Quest":"How many candies?",
"Result":"6"
}]}  ```

JSON exercise set example

```    public class ExerciseSet
{
public List<Exercise> Exercises { get; set; }
}

public class Exercise
{
public string Type { get; set; }
public string Image { get; set; }
public string Quest { get; set; }
public string Result { get; set; }
}

```

C# classes for exercise set

Once we have the exercise set defined, we need to load it when the application starts.

```        protected override void OnStartup(StartupEventArgs e)
{
base.OnStartup(e);

ExerciseSet ExeSet = JsonConvert.DeserializeObject<ExerciseSet>(contentFile);
}
```

Whenever we start a new exercise, we just select a random one from the array created on the startup (no save progress yet, see roadmap section) and populate the controls on the layout.

```private void CreateNewExercise()
{
Canvas.Children.Clear();
if (randNum == -1)
{
Random rnd = new Random();
randNum = rnd.Next(0, App.LettersExercises.Letters.Count);
}
else
{
randNum++;
if (randNum&amp;gt;=App.LettersExercises.Letters.Count)
{
randNum =0;
}
}
_exe = App.LettersExercises.Letters[randNum];
Quest.Content = _exe.Quest;
BitmapImage src = new BitmapImage();
src.BeginInit();
src.UriSource = new Uri(_exe.Image, UriKind.Relative);
src.EndInit();
ImageQuest.Source = src;
}```

The gesture recognizer is based on the \$N Multistroke Recognizer I created a portable class library so it can be used for Desktop applications, Windows 8 apps or even Windows Phone apps (see the roadmap section).

To recognize the gestures, I have created a timer. After more than 350 milliseconds with no touch input, I try to recognize the letter or the number. Letters and numbers models are stored in xml files. The user input (like the model) is stored as an array of points with their coordinates and the number of strokes. Those xml files are compared with the user input to classify the gesture.

```<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<Gesture Name = "C">
<Stroke>
<Point X = "865" Y = "200" T = "0" Pressure = "0" />
<Point X = "865" Y = "200" T = "0" Pressure = "0" />
<Point X = "865" Y = "200" T = "0" Pressure = "0" />
<Point X = "865" Y = "200" T = "0" Pressure = "0" />
<Point X = "865" Y = "197" T = "0" Pressure = "0" />
<Point X = "865" Y = "196" T = "0" Pressure = "0" />
<Point X = "865" Y = "196" T = "0" Pressure = "0" />
<Point X = "863" Y = "196" T = "0" Pressure = "0" />
<Point X = "863" Y = "195" T = "0" Pressure = "0" />
<Point X = "863" Y = "195" T = "0" Pressure = "0" />
<Point X = "861" Y = "194" T = "0" Pressure = "0" />
<Point X = "860" Y = "194" T = "0" Pressure = "0" />
<Point X = "860" Y = "193" T = "0" Pressure = "0" />
<Point X = "858" Y = "193" T = "0" Pressure = "0" />
<Point X = "856" Y = "193" T = "0" Pressure = "0" />
<Point X = "855" Y = "192" T = "0" Pressure = "0" />
<Point X = "854" Y = "191" T = "0" Pressure = "0" />
<Point X = "851" Y = "191" T = "0" Pressure = "0" />
<Point X = "849" Y = "191" T = "0" Pressure = "0" />
<Point X = "847" Y = "190" T = "0" Pressure = "0" />
<Point X = "842" Y = "189" T = "0" Pressure = "0" />
<Point X = "827" Y = "187" T = "0" Pressure = "0" />
<Point X = "818" Y = "185" T = "0" Pressure = "0" />
<Point X = "812" Y = "184" T = "0" Pressure = "0" />
<Point X = "809" Y = "184" T = "0" Pressure = "0" />
<Point X = "806" Y = "184" T = "0" Pressure = "0" />
<Point X = "804" Y = "184" T = "0" Pressure = "0" />
<Point X = "800" Y = "184" T = "0" Pressure = "0" />
<Point X = "796" Y = "184" T = "0" Pressure = "0" />
<Point X = "793" Y = "185" T = "0" Pressure = "0" />
<Point X = "789" Y = "186" T = "0" Pressure = "0" />
<Point X = "787" Y = "187" T = "0" Pressure = "0" />
<Point X = "783" Y = "188" T = "0" Pressure = "0" />
<Point X = "780" Y = "188" T = "0" Pressure = "0" />
<Point X = "778" Y = "189" T = "0" Pressure = "0" />
<Point X = "771" Y = "190" T = "0" Pressure = "0" />
<Point X = "767" Y = "191" T = "0" Pressure = "0" />
<Point X = "763" Y = "193" T = "0" Pressure = "0" />
<Point X = "759" Y = "194" T = "0" Pressure = "0" />
<Point X = "756" Y = "196" T = "0" Pressure = "0" />
<Point X = "753" Y = "197" T = "0" Pressure = "0" />
<Point X = "749" Y = "200" T = "0" Pressure = "0" />
<Point X = "744" Y = "202" T = "0" Pressure = "0" />
<Point X = "736" Y = "207" T = "0" Pressure = "0" />
<Point X = "733" Y = "210" T = "0" Pressure = "0" />
<Point X = "731" Y = "213" T = "0" Pressure = "0" />
<Point X = "728" Y = "216" T = "0" Pressure = "0" />
<Point X = "726" Y = "218" T = "0" Pressure = "0" />
<Point X = "724" Y = "221" T = "0" Pressure = "0" />
<Point X = "721" Y = "224" T = "0" Pressure = "0" />
<Point X = "717" Y = "229" T = "0" Pressure = "0" />
<Point X = "710" Y = "236" T = "0" Pressure = "0" />
<Point X = "708" Y = "240" T = "0" Pressure = "0" />
<Point X = "704" Y = "244" T = "0" Pressure = "0" />
<Point X = "703" Y = "247" T = "0" Pressure = "0" />
<Point X = "699" Y = "255" T = "0" Pressure = "0" />
<Point X = "696" Y = "260" T = "0" Pressure = "0" />
<Point X = "695" Y = "265" T = "0" Pressure = "0" />
<Point X = "693" Y = "270" T = "0" Pressure = "0" />
<Point X = "691" Y = "274" T = "0" Pressure = "0" />
<Point X = "689" Y = "280" T = "0" Pressure = "0" />
<Point X = "687" Y = "285" T = "0" Pressure = "0" />
<Point X = "686" Y = "290" T = "0" Pressure = "0" />
<Point X = "683" Y = "297" T = "0" Pressure = "0" />
<Point X = "681" Y = "301" T = "0" Pressure = "0" />
<Point X = "681" Y = "306" T = "0" Pressure = "0" />
<Point X = "681" Y = "311" T = "0" Pressure = "0" />
<Point X = "680" Y = "317" T = "0" Pressure = "0" />
<Point X = "679" Y = "322" T = "0" Pressure = "0" />
<Point X = "679" Y = "327" T = "0" Pressure = "0" />
<Point X = "679" Y = "331" T = "0" Pressure = "0" />
<Point X = "679" Y = "342" T = "0" Pressure = "0" />
<Point X = "679" Y = "347" T = "0" Pressure = "0" />
<Point X = "679" Y = "351" T = "0" Pressure = "0" />
<Point X = "679" Y = "354" T = "0" Pressure = "0" />
<Point X = "681" Y = "358" T = "0" Pressure = "0" />
<Point X = "683" Y = "364" T = "0" Pressure = "0" />
<Point X = "684" Y = "369" T = "0" Pressure = "0" />
<Point X = "686" Y = "373" T = "0" Pressure = "0" />
<Point X = "689" Y = "380" T = "0" Pressure = "0" />
<Point X = "691" Y = "382" T = "0" Pressure = "0" />
<Point X = "692" Y = "385" T = "0" Pressure = "0" />
<Point X = "694" Y = "389" T = "0" Pressure = "0" />
<Point X = "697" Y = "396" T = "0" Pressure = "0" />
<Point X = "700" Y = "401" T = "0" Pressure = "0" />
<Point X = "702" Y = "404" T = "0" Pressure = "0" />
<Point X = "705" Y = "407" T = "0" Pressure = "0" />
<Point X = "707" Y = "410" T = "0" Pressure = "0" />
<Point X = "710" Y = "413" T = "0" Pressure = "0" />
<Point X = "714" Y = "416" T = "0" Pressure = "0" />
<Point X = "718" Y = "420" T = "0" Pressure = "0" />
<Point X = "726" Y = "424" T = "0" Pressure = "0" />
<Point X = "729" Y = "427" T = "0" Pressure = "0" />
<Point X = "734" Y = "429" T = "0" Pressure = "0" />
<Point X = "739" Y = "431" T = "0" Pressure = "0" />
<Point X = "744" Y = "433" T = "0" Pressure = "0" />
<Point X = "749" Y = "435" T = "0" Pressure = "0" />
<Point X = "758" Y = "437" T = "0" Pressure = "0" />
<Point X = "762" Y = "438" T = "0" Pressure = "0" />
<Point X = "766" Y = "438" T = "0" Pressure = "0" />
<Point X = "771" Y = "438" T = "0" Pressure = "0" />
<Point X = "776" Y = "439" T = "0" Pressure = "0" />
<Point X = "781" Y = "440" T = "0" Pressure = "0" />
<Point X = "785" Y = "440" T = "0" Pressure = "0" />
<Point X = "790" Y = "440" T = "0" Pressure = "0" />
<Point X = "799" Y = "438" T = "0" Pressure = "0" />
<Point X = "803" Y = "438" T = "0" Pressure = "0" />
<Point X = "808" Y = "438" T = "0" Pressure = "0" />
<Point X = "812" Y = "438" T = "0" Pressure = "0" />
<Point X = "816" Y = "438" T = "0" Pressure = "0" />
<Point X = "820" Y = "437" T = "0" Pressure = "0" />
<Point X = "824" Y = "436" T = "0" Pressure = "0" />
<Point X = "827" Y = "435" T = "0" Pressure = "0" />
<Point X = "833" Y = "434" T = "0" Pressure = "0" />
<Point X = "835" Y = "434" T = "0" Pressure = "0" />
<Point X = "835" Y = "434" T = "0" Pressure = "0" />
<Point X = "835" Y = "434" T = "0" Pressure = "0" />
<Point X = "836" Y = "434" T = "0" Pressure = "0" />
<Point X = "836" Y = "434" T = "0" Pressure = "0" />
<Point X = "836" Y = "434" T = "0" Pressure = "0" />
<Point X = "836" Y = "434" T = "0" Pressure = "0" />
<Point X = "836" Y = "434" T = "0" Pressure = "0" />
</Stroke>
</Gesture>

```

XML file for letter 'C'

```        private void RecognizeGesture()
{
if (_points.Count > 31)
{
var candidate = new Gesture(_points.ToArray());
GestureAnalysis gestureData = PointCloudRecognizer.Classify(candidate, _trainingSet);
if ((gestureData.Mindistance <= 0.5) && (_exe.Result == gestureData.GestureName))
{
Result.Text = "GOOD! Next level";
Result.Foreground = Brushes.DarkBlue;
BtnNext.Visibility = System.Windows.Visibility.Visible;
SoundPlayer tadaSound = new SoundPlayer(@"Sounds/Tada.wav"); //Initialize a new SoundPlayer of name wowSound
Storyboard sb = (Storyboard)this.FindResource("StoryboardOk");
sb.Begin();
}
else
{
Result.Foreground = Brushes.Red;
Result.Text = "Try again!!";
BtnNext.Visibility = System.Windows.Visibility.Hidden;
SoundPlayer errorSound = new SoundPlayer(@"Sounds/Buzzer.wav"); //Initialize a new SoundPlayer of name wowSound
errorSound.PlaySync();
Storyboard sb = (Storyboard)this.FindResource("StoryboardError");
sb.Begin();
}
_recognized = true;
}
else
{
Canvas.Children.Clear();
_strokeIndex = 0;
_points.Clear();
}
}

```

Recognizer function

This is the future of Smart Blackboard:
• Extend exercises markup to include different sounds for error and success
• Save progress
• New exercise follow the letters sequence. SBlackboard says...
• New exercise follow the numbers sequence. SBlackboard says...
• New set of math operations exercises. More complex with 2 or more digits
• New sets for animals, fruits, etc within letters exercises
• New exercise. Fill in the gaps with more than 1 letter recognition.
• More complex count exercises (mixing fruits for example...)
• New Windows 8 app
• New Windows Phone 8 app

Software Developer (Senior)
Spain

Passionate about technology in general and .NET development in particular. As a chief software engineer in S&C I manage the technical work-packages on some projects partially funded under the 7th Framework Programme by the European Commission.

I love coding and heading up technical implementation of cloud based applications. Right now, I am involved on M2M solutions interconnecting web applications, smartphones and tablets. All of them acquiring data from a Web API implementation on the cloud that relies on a database pumped in by a driver getting info from the real world.

I like simplicity. Something that you cannot explain is not worth it. Some of my skills are mashing up ideas and thinking outside the box in order to develop innovative solutions.

In my spare time you can find me snowboarding deep in the Pyrenees mountain range, improving my capoeira skills or simply coding an app inspired by some crazy idea.

 First Prev Next
 Thanks! Kevin Priddle 22-Aug-13 5:44
 Last Visit: 31-Dec-99 18:00     Last Update: 8-Dec-13 11:03 Refresh 1