Web Ink/Drawing Control

• Download demo project - 20.2 KB
• Download source for the C# control - 20.0 KB
• Download Microsoft Ink version 1.7 custom MSI installer - 2 MB
• Download & view the demo video (Windows Media Format) - 1 MB

Introduction

This article discusses how to create and use a web browser (IE only) compatible ink or drawing control using the Microsoft Tablet PC SDK version 1.7. A sample pre-built control is provided for use and demonstration. Also, please read the disclaimer at the end in case you have any issues with this article.

Background

Having seen Microsoft's Tablet PC ink capabilities in action, I found it sad they didn't provide web or ASP.NET support of their ink libraries. That is until recently — with the release of the Tablet PC SDK version 1.7, Microsoft provides a way to create an ink user control that can be embedded in a webpage and interact with the web server via web scripting, etc.

There are several articles on the web (and a sample in the SDK) that talk about this new feature and how to build such a control. However, I found the documentation/articles to be lacking in some respect due to confusing presentation of how to deploy such an application. Furthermore, no complete source code (other than the sample in the SDK) is provided for evaluation. Only minor snippets are shown in the articles.

Thus, this article skips over the part on creating the control (you can refer to the reference websites below for that information) and instead focuses on:

1. the deployment part using a sample demo control project, and
2. possible (creative?) uses plus pros and cons for such a control.

The sample control is also pre-built with common drawing and inking features included so that you can use it as is, saving you time from having to write one up from scratch. Or you can also extend the control with additional properties & methods or modify the existing methods.

Reference Websites

Main References

Code Magazine article: Ink on the Web
Mobile Ink Jots 3: Ink on the Web
Microsoft Tablet PC - Creating a Redistributable Setup

Additional References

Creating Interactive Hand-Written Web Pages on Tablet PC
Code Magazine article: Introduction to Tablet PC Development
Code Magazine article: Ink Recognition and Ink Analysis

Ink Control Overview

We'll begin the article from the point after creation of the user control. First, some brief points to mention:

1. The control is a .NET windows form control (not an ASP.NET web control) that you can embed onto a webpage and have it run like a Java applet on the client side. Interaction with the server can be achieved through posting back to the server via JavaScript & a web form. You can also alternatively use the control in a windows form application. The control could also be extended to provide methods for communicating with a server by making use of the .NET Web Request & Response classes, or networking classes, etc.
2. The control can be implemented with interaction between itself and the external environment by (1) placing code for interaction as components/controls within the control or (2) placing code for interaction as public methods that are a part of the control that can be called from the external environment. In terms of making the control web usable, option two provides better customization and extendability, and better resizing of the control. Thus, the sample control project uses option two.

Using the Control

Requirements

The following is required to run or execute the control on a browser client:

• .NET Framework 1.1 Runtime (or SDK) or greater.
• Internet Explorer (IE) 5.5 or greater
• Javascript enabled in browser (for interactivity features)
• Microsoft Ink Library version 1.7^* or Windows XP Tablet PC w/ SP2 installed
• Mouse or other pointing device. A Windows XP Tablet PC system is recommended.

^* You can get the Microsoft Ink Library version 1.7 by installing a custom Microsoft Ink version 1.7 library installer or from installing the Microsoft Tablet PC SDK version 1.7. Windows XP Tablet PC with Service Pack 2 also includes the library by default. Sadly, deploying the control over the web does not allow loading the reference libraries along with the control, unlike a windows form application where the Ink library can reside (without installation) in the same directory as the application. For developing & modifying the control, you will need to use the SDK (even if you have Windows XP Tablet PC w/ SP2).

Embedding & Loading the Control

To embed & load the control in a browser, you simply insert a special block of code utilizing the HTML object tag into a webpage. The tag should be similar to the following format (it worked for me, while some examples in the other articles didn't).

<object id="NameForObject" height="aValue" width="aValue" 
classid="ControlDLLName.dll#ControlNamespaceName.ControlClassName"></object>
<!--// Example here assumes that the DLL is in the same directory as the 
webpage. If not, you will have to modify the path in the classid attribute.
//-->

With this code, the object will run upon the browser loading the webpage. You can't do much with it at this point, so you can go to the next step.

Interacting with the Control on the Client Side

To provide functionality between the control & the user, you can call the control's built-in methods. This, however, requires writing JavaScript wrapper functions that call the control's methods as in the following example:

function SetPenColor(pColor){
//common colors only, unrecognized colors results in no change or black ink

//good idea to access control object by having it be a part of a form

//then access object's methods through the form

document.forms[0].NameForObject.SetPenColor(pColor);
}

The following code listing shows the available methods in the demo control. Not all are available for use over the web due to security restrictions, unless you add the website as a fully trusted site in Internet Explorer. NOTE: Not all methods were fully tested and some may not behave as you would expect.

/* Get & Set Methods */
// Get or set whether anti-alias is used. Default is true

public bool AntiAlias
// Get or set whether to smooth out the drawn ink or not.

public bool FitToCurve
// Get or set whether the pen recognizer detects pressure from the pen

// to adjust the width, height, and intensity of the ink drawn, etc. 

// Default is false.

public bool IgnorePressure
// Set the pen color using common (.NET) color names.

public void SetPenColor(string usrColor)
// Set the background color using common (.NET) color names.

public void SetBackgroundColor(string usrColor)
// Get or set the pen tip style. 0 for ball point, 1 for rectangle.

public int PenTipStyle
// Get or set the pen width in pixels.

public float PenWidth
// Get or set the pen height in pixels.

public float PenHeight
// Get or set transparency level of ink drawn. Values range 

// from 0 to 255. 0 is opague, 255 is transparent.

public int Transparency
/* Editing Mode Methods */
// For set modes, returns true on success in changing modes. Else false.

// Set the control to inking or drawing mode.

private bool SetToInkMode()
// Set the control to ink/object selection mode.

private bool SetToSelectMode()
// Set the control to erasing mode.

private bool SetToDeleteMode()
// Clears the ink in the control.

public void ClearInk()
// For clipboard functions, returns true on success, else false

// Copy ink to clipboard as serialized ink. Not usable when used on the web.

public bool CopyAsSerialInk()
// Copy ink to clipboard as a bitmap. Not usable when used on the web.

public bool CopyAsBitmap()
// Cut ink from control to clipboard as serialized ink. Not usable when used 

// on web.

public bool CutAsSerialInk()
// Cut ink from control to clipboard as a bitmap. Not usable when used on web.

public bool CutAsBitmap()
// Paste ink in clipboard to control. Not usable when used on the web.

public bool Paste()
/* Transformation Methods */
// Create a shear effect on the selected ink based on a set of coordinates

// x = Horizontal shear factor

// y = Vertical shear factor

public void Shear(float x, float y)
// Scale selected ink based on a set of coordinates

// x = Horizontal scale factor

// y = Vertical scale factor

public void ScaleInk(float x, float y)
// Rotate the selected ink for some degrees based on a set of coordinates

// degrees = Number of degrees to rotate

// x = Horizontal coordinate

// y = Vertical coordinate

public void Rotate(float degrees, int x, int y)
// Move the selected ink by an offset coordinate

// x = Horizontal offset

// y = Vertical offset

public void MoveInk(float x, float y)
/* Text Recognition Methods */
// Perform text recognition on ink and return best guess. Only works with 

// Windows XP 

// Tablet PC systems.

public string InkToTextBestGuess()
// Perform text recognition on ink and return the set of best guesses as a 

// newline delimited string. Only works with Windows XP Tablet PC systems.

public string InkToTextResults()
/* I/O Methods */
// Get ink in the control as Base 64 encoded string of the ink in GIF format.

public string ReturnInkAsBase64Gif()
// Get ink in the control as a GIF (byte array)

public byte[] ReturnInkAsGif()
// Get ink in the control as Base 64 encoded string of the serialized ink.

public string ReturnInkAsBase64SerializedInk()
// Get ink in the control as serialized ink

public byte[] ReturnInkAsSerializedInk()
// Load a byte array of ink into the control. Returns true on success.

// inkData = the byte array of ink data.

public bool LoadInk(byte[] inkData)
// Load a base 64 encoded string (of a byte array) of ink/GIF into the 

// control. Returns true on success.

// b64InkData = the base 64 encoded string (of byte data).

public bool LoadInkFromBase64(string b64InkData)
// Load background image to ink control. NOTE: This is meant for windows 

// forms applications only, or requires

// no security restrictions on client side file access in a web application.

// filePath = the file path of background image to load.

public bool LoadBackgroundImage(string filePath)
// Dispose of the control when not needed, like on a webpage unload

//you can use the control's native dispose method, something like:

protected void Dispose()

Interacting with the Control on the Server Side

To communicate with the web server, you can make a request to the server, sending the ink data along with it. And you may be able to retrieve information back to the control from the server reply (this part has not been tested nor implemented). A simple way to do so would be to make the control object part of a form and use JavaScript to submit the ink data (stored to a hidden form field) to the server. To pass the data over the web, it needs to be encoded as a Base 64 string since HTML does not handle binary data. Then you can decode and process the data on the server side. Unless you need to work in native Microsoft Ink format, you would generally pass the ink data as a GIF byte array since GIF is a standard image format, easy to process, and still be loaded back into an ink control. The following code snippets show how to send the control's ink data to the server as a GIF byte array.

First, implement the technique to retrieve the control's data and store it in a hidden form field.

<!--// This can be done with HTML and a javascript onclick event, for example.
//-->
<input type="button" value="Save as GIF" name="SaveCmd" onclick="SaveAsGif()">
<input type=hidden name="SavedInkData" value="empty">

Then, implement the technique to send the data to the server as a Base 64 encoded string. The demo project includes a simple server processing sample that just decodes & outputs the GIF image to the browser. In practice, you could further manipulate the image, store to a database, write to file, or send somewhere via email.

function SaveAsGif(){
//store control's ink data in base 64 encoding to hidden form field

document.forms[0].SavedInkData.value = 
               document.forms[0].InkBoxCtrl.ReturnInkAsBase64Gif();
//send the data using a form submit, generally via HTTP POST method

document.forms[0].submit();
}

Notes on Deploying the Demo Application and About the Source Files

The demo application was created in the simplest fashion, and is not a standard Visual Studio .NET web application. To deploy the demo, you need only copy it to your web server. No special Visual Studio .NET or IIS virtual directory setup is required. The server side script is available in ASP.NET or PHP, which means you can deploy this application to IIS with ASP.NET (or PHP) or any web server that supports PHP. Only the browser clients need to support the .NET Framework.

For any ASP.NET configuration files and settings that are needed in your case, you may use your existing configuration files or create them yourself by manually creating them or using Visual Studio .NET to generate them.

The source is a Visual Studio .NET 2003 C# control project.

NOTE: The control works fine when used in a web application like the one described here but there may be problems when using it in a Windows Form/Desktop application. I tried dropping the control into a Windows forms project and just compiling it and I got errors (or maybe I didn't add the control to the application correctly). So if you plan to use it in such projects, you may have to tweak the control to get it to work.

IIS Deployment Issues

You may encounter some problems when deploying to IIS, particularly IIS 6 on Windows 2003 Server. Here are some tips on resolving such problems. Try them in the specified order:

1. Verify in IIS, that your application has "Script only" execute permissions under your Virtual Directory, Home Directory, or Directory settings. If you have it set to "Scripts and Executables" you will not be able to download the web ink control DLL. If you must use "Scripts and Executables" because you are using CGI, you may need to rethink how you plan to deploy this application. This method solved my IIS problem. Click here for more info.
2. Verify that you are not using URLSCAN (from Microsoft) or something similar on IIS. If you are using such a tool, you must configure it to allow downloads of DLL files. Click here for Using URLScan on IIS.
3. Try lowering your IIS server security settings, if necessary. Probably not a good idea though, and I don't think you will really need to do this.
4. When all else fails & you really like using this control & you are not required to use IIS/ASP/ASP.NET, consider hosting the application on a non-IIS web server like Apache.

Apache Deployment Issues

Alessandro Torrisi pointed out a potential issue on Apache (whether running server on Windows or Linux, to be safe) where the ink control DLL was not loaded into the application. According to Alessandro, to fix this, just add "application/x-msdownload dll" in the "/etc/apache2/mime.types" file and restart apache.

Points of Interest

Summary of Pros & Cons

I think such a control has potential in some applications. The negative aspects are that the control runs in a Microsoft client environment only, it requires downloading & installing the Microsoft Ink library, and text recognition is available only for Tablet PCs. These aspects might deter people from using & deploying the control. However, there are some positive aspects of this control too, particularly in the possible uses that the control can provide as well as the option of using any server side platform (it need not be Microsoft). In summary, while the control may not be feasible for general public & corporate use, it may prove useful in certain specialized applications. Plus it's a fun thing to toy around with in your spare time.

Possible (Creative?) Use Cases

• Inking or drawing on the web, as an alternative to using a Java applet or Macromedia Shockwave/Flash implementation. A good example of a Shockwave version can be seen at Tutor.com's Live Homework Help information site. I used to be a Tutor.com tutor and they do have a very nifty web whiteboard interface.
• Provide ink & drawing support (via the control on the client side) to ASP.NET applications or web services running on the Mono open source .NET platform (and Microsoft .NET too).
• Provide drawing capabilities to non-Microsoft and legacy web server platforms:
  • Java, J2EE, EJB, servlets, JSP
  • PHP (sample provided in demo project), Perl, Python, Ruby
  • Classic ASP, ColdFusion
  • Standard HTML/DHTML web pages only, Microsoft HTA applications
  • Interact/integrate with Macromedia Shockwave/Flash plugin or Java applet for more capabilities?
• Provide ink recognition processing on the server side using the control on web browser clients & a Tablet PC as the web or application server (even better if the clients were running on Tablet PC too of course).
• Use the control as inkable form fields:
  • Each instance of the control replaces a textbox in a form & on submit, the server processes the ink data along with the other form data.
  • Or use the control like a Tablet PC Input Panel & on click, the ink is converted to text & can then be copied & pasted to the appropriate fields. NOTE: this option is only available on Tablet PC systems.
• Support digital signatures over the web (as GIF image or MS Ink format). Though this might present some legality, security, & privacy issues.
• Implementing a non-realtime digital whiteboard for conferencing--after drawing on the board, a user submits the data to the server for intermediate storage. Then either periodically/automatically by the application or manually performed by receiving users, the users can get an updated snapshot of the shared board (or the submitted user's board) loaded into their boards from the server. This is just my thoughts & hasn't been tested. It may be tricky to implement, if possible at all. But if it is possible, this is a neat thing to do.
• Use this control along with AJAX to make nifty dynamic web applications?
• Customized ink applications using the control
• Extend control to return ink in bitmap, JPEG, and other formats
• Finally, can you provide other ideas?

Updated Features and Information - as of Oct. 27, 2007

• Added new methods to the ink control to load ink from base 64 encoded data and to load a background image (primarily for Windows forms applications only).
• Added client side code for backup & restore of ink functionality. Backup feature only backs up current ink and overwrites previous backup. Backup uses the same hidden form field SavedInkData, so when you "Save" or send ink to server, current backup gets overwritten as well. You can modify it so that backup uses a different form field instead, etc. Restore simply loads ink from the current backup. NOTE: You may have to click "Restore" twice to get it to load the backup, that's what happens on my system for some reason.
• For debugging purposes, I added a textbox with the same name as hidden form field "SavedInkData" so that you can see and modify the base 64 encoded version of ink data. By default, this new field is commented out. You can switch to this one instead of the hidden form field for debugging.
• I added a new web server script that takes a supplied image URL, downloads it (to the server), converts it to Base 64 text, and returns that to the browser/web client for use by the web ink control. Unfortunately, this new script is currently available for ASP.NET only, no PHP.
• I added a new client side feature of using XmlHttpRequest to pass a URL of an image to the server and get back its Base 64 encoded version so that you can load it into the web ink control. Unfortunately, I couldn't get it to load in the ink control. In any case, this is a good sample of how you can use XmlHttpRequest to fetch data from server to load into your ink control, etc. I tried to use fetch from image URL, but you can fetch from database or file on server, etc. Also to note, I made use of my custom XmlHttpRequest library for this (but you can use XmlHttpRequest natively, for those of you who are proficient enough with it). See my other Code Project article titled "Web/HTTP Automation Libraries" for more information.
• See also my comment/forum post to this article for additional updates on web ink, web ink recognition, and IE and Firefox support.

Bugs

• Scrolling causes problems with accessing form elements & selecting text, etc. This might be a browser or .NET rendering problem. Therefore, if you use a lot of form controls or have a lot of content below or above the control, etc., it is recommended that the webpage be viewed in full screen browser mode OR in portrait orientation (for Tablet PCs only) to minimize scrolling. Otherwise, consider having less controls or content on the page.
• Not all implemented control methods are supported when running the control over the web & some don't necessarily work as intended, so be aware of that.
• Restore feature may require you to click twice to load ink from backup.
• Load Ink from URL feature currently not working with ink control. Kept in demo project as an example of how to attempt this.

Release History

• 06/27/2006 - Initial Release. Some future enhancements may include additional output formats & built-in web request/response methods for input/output of data between control and a server.
• 07/04/2006 - Updated article & download files to clarify some things.
• 11/11/2006 - Updated article with information on deployment issues with IIS.
• 10/27/2007 - Updated article and code for issues and new features.

Disclaimer

NOTE: This article and accompanying code are not entirely original. Rather, this article is intended to clarify and elaborate on the subject where the original articles seem to be lacking in detail. Additionally, source code (a compilation of code snippets from the original articles integrated together with my own code and extra functionality) and a demo application are provided for you to use freely. The original articles only provided code snippets, no complete code that you could use off the bat. Last, this article is intended to suggest and foster creative use of such an ink control beyond the standard Microsoft convention/platform.