How Scribblify Delivers a Cross-Platform GUI with Node.js* and the Chromium* Embedded Framework

Introduction

For Wisconsin-born developer Matt Pilz, the third time’s the charm. Pilz is founder of LinkedPIXEL, and his creativity and hard work have found favor with judges in a trio of consecutive developer challenges that Intel conducted. Most recently, his Scribblify drawing app with its 10-point touch capability and Google Chrome*-based GUI took the grand prize in the Intel® App Innovation Contest 2013 using resources from the Intel® Developer Zone. In spring 2013, Pilz won the Intel® Perceptual Computing Challenge with Magic Doodle Pad, a drawing app that’s controlled with hand gestures. And in 2012, he was recognized for Ballastic, an action game that’s optimized for Ultrabook™ devices.

Like his prior contest experiences, the idea for Scribblify (Figure 1) pre-dated the contest, but Scribblify actually began life running on handheld devices. As one of 200 contestants to receive a Lenovo Horizon* All-in-One (AIO) PC from contest sponsor Lenovo, Pilz was excited at the prospect of growing his doodling app to a full-size screen. "When I read about the Lenovo Horizon All-in-One with its 1920 HD resolution and huge 27-inch screen with touch capabilities, I realized how fun it would be to create a gigantic canvas-sized version of this app so you can draw on a party table or another large surface," he said. "That's where my vision took me." To make that happen, Pilz would have to modify the program to work well on a screen that’s four to six times wider than the original version and accept input from 10 simultaneous touch points instead of one.

Origins of Scribblify

Pilz first began dabbling in mobile-app development in 2010, about the time he got hold of his first Apple iPod touch* mobile device. LinkedPIXEL released an early version of Scribblify the following year. "The first interface was designed for 480x320 resolution. Everything was made minimalistic to support the very limited hardware at the time," he recalled. Scribblify for iOS* hit the App Store in January 2011. "It actually took off and has been doing better every month," said Pilz. So about a year ago he followed up with a Google Android* version.

Pilz describes Scribblify as a casual drawing and doodling program with some special capabilities. "What sets it apart are its brushes and color effects," he said. The brushes (Figure 2), which he designed himself, incorporate textures not seen in many drawing apps of this type. "I like to believe most of them are pretty unique," he said, but he quickly added that Scribblify isn’t meant to compete with high-end illustration apps. "The iPad equivalent of an app like Photoshop* has a pretty steep learning curve," said Pilz. "Then you also have some limited drawing apps that come with just a round brush and a few options. There wasn't anything up to the level I envisioned for Scribblify, which is made for all ages to pick up and start using."

Challenges

Pilz’s first major obstacle was to decide which technology he would use to build a version of Scribblify for Microsoft Windows*. He started with TGC’s App Game Kit* (AGK), the engine he used to make Ballastic. "That allowed me to create a rough prototype, [but] it didn't support a lot of the functionality that I wanted in this app," he said. To qualify for the contest, apps must be built for Lenovo’s Aura* multi-user interface and must not rely on major portions of the underlying operating system. One stipulation of the guidelines was that developers could not use any native Windows dialog boxes. "It would've been easy through my app to just have users browse a standard file dialog box to save their artwork or open up a different image," when interfacing with the file system, Pilz said, for example. "But that would have taken away from the experience that Intel was trying to achieve…to make each app almost a sandbox."

Pilz’s remedy was to use Node.js*, an event-driven, server-side JavaScript* framework for building network services for V8, the JavaScript engine inside the Chrome browser. According to Node creator Ryan Dahl, the Node framework makes I/O non-blocking by using an event loop and facilitates asynchronous file I/O, thanks to a built-in event pool.

"One main purpose of Node.js is to develop web apps and use them natively on a desktop regardless of operating system," Pilz said. "You don't have to rely on having Windows installed; Node.js is the back-end that powers the entire app." Using the framework and a library called Node-Webkit, Pilz saved and loaded art files easily using an interface and gallery system he custom built using standard web technologies. The application runs on top of the Chromium* Embedded Framework (CEF), which provides the same powerful HTML and JavaScript renderer as used by Chrome.

How Scribblify Saves and Deletes Images
When the Scribblify app first loads, the user’s public Documents directory is retrieved via Node.js:

// Initialize Node.js file system
var fs = require('fs');

// Get User's Documents Folder (Win 7+)
var galleryPath = process.env['USERPROFILE'] + "\Documents\Scribblify\";
galleryPath = galleryPath.replace(/\/g, "/"); // Use forward slashes

// See if folder exists, otherwise create it
if(!fs.existsSync(galleryPath))
{
    fs.mkdir(galleryPath);  
}

When the user requests the image be saved, the HTML5 canvas is converted into image data:

var saveImg = canvas.toDataURL();

A unique filename is generated based on timestamp, and Node.js saves the image data to the system.
// Write image data buffer to file
var data = saveImg.replace(/^data:image/w+;base64,/, "");
var buf = new Buffer(data, 'base64');
var fName = galleryPath + destFilename;
fs.writeFile(fName, buf, function(err) {
    if(err)
    {
        console.log(err);
    }
});

If the user wishes to delete an image from the gallery, Node.js achieves this simply:

fs.unlinkSync(galleryPath + targetImg);

Getting a list of existing files to display in the gallery is slightly more complex, as I also check to ensure that only PNG files are retrieved (not directories or other files that may exist).

var fileList = fs.readdirSync(currentPath);
var result = [];
for (var i in fileList) {
    var currentFileFull = currentPath + '/' + fileList[i];
    var currentFile = fileList[i];
    var fileExt = currentFile.split(".").pop().toUpperCase();
    var stats = fs.statSync(currentFileFull);
    if (stats.isFile() && fileExt == "PNG") {
       result.push(currentFile);
    }
    else if (stats.isDirectory()) {
         // Directory instead...
    }
}

Once the list is generated, the image paths stored in the array can be loaded using standard HTML and JavaScript techniques. This is used for both the gallery display and for importing a particular piece of art back into the canvas.

While Pilz was able to prototype most of the app using technologies already familiar to him, including HTML5’s native canvas element for the front-end and Node.js on the back, some of the app’s most groundbreaking features were just plain breaking. "I was trying to do something that hasn't often been attempted in a web app—creating an entire creative drawing app with complex textured brushes instead of procedurally generated shapes," he said.

The program initially worked fine, but Pilz soon discovered a problem. "When using more than a few touch points or drawing too rapidly, the app would lag for a few seconds due to the performance of the canvas element, and then it would go on," he said of his new app’s responsiveness. His search for a solution this time led him to Cocos2d-x, an open source, cross-platform framework available under the MIT License which recently added WebGL support to its JavaScript branch.

Cocos2d-Javascript allowed Pilz to port his code much easier to WebGL, which he said performed much better and was well supported by the video card inside his AIO development system. "Now, Cocos2d-Javascript renders the drawings to the screen using WebGL…and it runs at 60 frames a second. That optimized the experience and made it very fun to use," Pilz said. Scribblify uses standard HTML for the interface elements and user interactions; Pilz describes it as a hybrid of standard HTML and JavaScript, mixed in with those other engines including Node.js for back-end processing and Cocos2d-Javascript for simplified WebGL rendering.

For help with general issues, Pilz turned to Stack Overflow, a free question-and-answer site for development pros and hobbyists. "I went there a few times when researching a particular task or if I got stuck on something," he said. "It’s always a quick and easy reference to see what other people recommend." For help specific to Cocos2d-x and Node.js, Pilz referenced APIs for those products quite extensively; and, of course, he used CodeProject* for not only this project, but also for others through the years. "I go there whenever I want to do a deeper investigation of certain programming topics," he said. "They had very helpful discussion forums related to this competition and were able to answer questions quickly."

Pilz also identified a bug with the way Chromium handles touches in conjunction with the Lenovo Horizon driver set, which caused some touches not to be released, especially on rapid use. His solution was to build a small "check-and-balance system" that issues an alert if an excessive number of touches have been recorded erroneously. "It will automatically save the user’s art to the gallery and alert them that they should restart the app," he said. Then it presents guidelines on how to prevent such a problem in the future. "If the user touches it with their whole palm [for example], it would display the alert, ‘You shouldn't do it that way, just use your fingertips.’ It would [also] recommend that you restart," he said. "I submitted this as a formal bug report to the Chromium developers during the competition."

var canvas = document.getElementById('canvas');
canvas.addEventListener('touchstart', touchStartFunction, false);
canvas.addEventListener('touchend', touchEndFunction, false);
canvas.addEventListener('touchcancel', touchCancelFunction, false);
canvas.addEventListener('touchmove', touchMoveFunction, false);
canvas.addEventListener('touchleave', touchLeaveFunction, false);

The Chromium*Embedded Framework (on which Google Chrome* is built) has supported native touch controls for some time and is a breeze to implement. Various forms of touch can be bound using simple event handlers, including touchstart, touchend, touchcancel, touchmove, and touchleave.

Time constraints of the contest—and limitations of the framework with which Pilz was working—caused him to delay or forego implementation of some Scribblify functions. "One big feature I wanted to add was a full undo and redo system, but I was in unfamiliar territory on how to make that work without bogging down the system," Pilz said. So for now, there's an eraser and a button for clearing the canvas. He also wanted to include a few more drawing modes and more brushes.

Debugging

Among the benefits of developing a web-based application is that most modern browsers come with comprehensive debugging tools, which Pilz said proved invaluable during the entire development process.

"Perhaps the most beneficial to me is that the Chrome developer tools support a full-featured JavaScript debugger. This allowed me to create breakpoints and walk through the code step-by-step and to monitor specific variables and properties throughout, especially when encountering problems," he said. At the same time, Pilz was able to analyze specific Document Object Model (DOM) elements and update styles on-the-fly to more rapidly achieve the desired result. Pilz also found helpful the developer resources provided by Lenovo on lenovodev.com, particularly the Integration Requirements for Horizon Apps (login required).

Multi-User

Scribblify for the AIO supports 10 simultaneous touch points, which means that as many as 10 users can be drawing on the canvas at once. Given the time constraints of the contest, a top consideration for Pilz was the ease with which he could integrate multi-touch capabilities. "When I made the final determination to develop a web-based application using HTML5 and related technologies, I knew that the complexities of handling multi-touch could be alleviated if I used a web foundation that already had desktop multi-touch support," he said.

Thankfully, the CEF has had such support on its desktop and mobile releases. With the entire application being powered through the CEF, Pilz said it was easy to store and retrieve touch data for as many touches as the AIO would support.

Additional articles

• All-in-One PC: What are the developer possibilities?
• Developer's Guide for Intel® Processor Graphics for 4th Generation Intel® Core™ Processors
• Developing Immersive Apps For A New Breed Of All-In-One Computer

More Innovation

Until you’ve used Scribblify or seen one of LinkedPIXEL’s demo videos, it’s difficult to appreciate how truly unusual its brushes are. "I would consider the brushes themselves to be the most highly innovative aspect of Scribblify," said Pilz. "All of the designs and brushes were my own invention, [and] each one has many different properties that control how it looks and how it behaves as you draw along the screen. They're pretty unique."

Pilz is also proud of the user interface, which he built mostly from scratch in just six weeks. "To be able to draw with all ten fingers at once using 10-point multi-touch, or with one person on one side and one on the other drawing at once, I consider that an innovative experience," he said. Pilz admits to taking a few shortcuts, where possible. "There were some libraries I used, like for the advanced color picker," he said. "Some MIT licensed libraries sped up that process so I didn't have to reinvent the wheel."

Scribblify also provides an innovative way to pick and blend colors. "With the many different color effects that you can add, you don't just pick a base color," Pilz said. "You can actually blend two colors together or use a plasma color or even add what I call ‘color variance,’ which mixes the shades of light and dark as you draw to create more natural-looking artwork."

Recently, Pilz has been working hard to convert Scribblify into a native Windows 8 app so that it can be distributed through the Windows Store—including through the specialized Lenovo Windows 8 Store. To address the requirements in the most streamlined fashion, the forthcoming version of Scribblify will be powered by Internet Explorer* 11 instead of Chromium and will use new methods for handling file input/output since Node.js is not supported. The Windows Store version of Scribblify is expected to be released in the April to May 2014 time frame.

About the Contest

The Intel App Innovation Contest 2013 called on app builders from Code Project, Habrahabr, ThinkDigit, and CSDN, to submit app ideas for Windows-based tablets and AIO PCs across the finance, healthcare, retail, education, gaming, and entertainment industries. Winning ideas were given a development system—one of 200 tablets and 300 AIO PCs from sponsor Lenovo—and six weeks to turn their idea into a demo app. The winning apps are listed here on the Intel® Developer Zone.

Of the 500 ideas, 276 demo apps were created, five of which were selected as the best in each of the six categories—making 30 winners in all. Judging criteria covered how well each app demonstrated the capabilities of its host platform, how well it addressed its market segment, and whether it solved the needs of the intended end user. Of course, winning apps must look professional and employ good graphics, perform well, be glitch-free, and take advantage of sensors, where appropriate.