Click here to Skip to main content
Click here to Skip to main content
Go to top

Accessing WPF apps over a Web browser via WebSockets and HTML5 Canvas

, 11 Apr 2011
Rate this:
Please Sign up or sign in to vote.
Shows how to push your WPF apps to users via WebSockets and HTML5 Canvas.

Introduction

This is a hack for pumping WPF windows over WebSockets and rendering it using HTML5 canvas - also with basic mouse input support. For each user connecting to the server, a new window instance is created, and the screen will be pumped over web sockets to the browser as base64 encoded images. These are rendered on an HTML5 canvas, almost real time.

browserwpf.png

What does this mean?

Once the HTML5 specification is final - it is almost certain that it'll be the most widely adopted platform for delivering applications to end users. With WebSockets in HTML5, we are going beyond the unidirectional, stateless HTTP to a full duplex, high speed connection infrastructure between the server and the client. This will definitely bring up lot of interesting possibilities, including exciting and feature rich user experiences. WPF is already a widely adopted, feature rich desktop application development platform, and this hack is a quick POC to bring the WPF experience over the web to the end user using HTML5 web sockets and canvas.

You can see that I'm accessing my WPF application over multiple browsers in the below video. Just to clarify, it is rendered in the browser purely using HTML5 canvas - No Silverlight or Flash.

Rendering the screen in the browser

First, let me explain a bit about WebSockets. The HTML5 draft specification includes WebSockets - here:

WebSocket is a technology providing for bi-directional, full-duplex communications channels, over a single Transmission Control Protocol (TCP) socket. It is designed to be implemented in web browsers and web servers but it can be used by any client or server application. The WebSocket API is being standardized by the W3C and the WebSocket protocol is being standardized by the IETF.

Here is a pretty minimized implementation of the page that opens a web socket, retrieves the image data, and draws it on a canvas:

var socket = new WebSocket("ws://localhost:8181/main");
socket.onopen = function () {

}

socket.onmessage = function (msg) {
    var canvas = document.getElementById("canvas");
    var ctx = canvas.getContext("2d");
    var img = new Image;
    img.onload = function () {
        ctx.clearRect(0, 0, canvas.width, canvas.height)
        ctx.drawImage(img, 0, 0); // Or at whatever offset you like
    };
    img.src = "data:image/gif;base64," + msg.data; ;
}

The server

I was planning to implement a quick socket server based on draft specifications - but I found that there are already a couple of .NET based implementations out there. One among them is Nugget - http://nugget.codeplex.com/ - I have a thin layer on top of Nugget for creating a window instance whenever a client socket connects to the server.

var srv = new WebSocketServer(port, origin, "ws://" + server + ":" + port);
srv.RegisterHandler<ElementSocket<T>>("/" + name);
srv.Start();
return srv;

Most of the logic required for creating window instances and pumping them back to the connected client socket goes in my ElementSocket implementation.

Once connected, the client WebSocket can send string commands to our server, requesting operations like start, stop, create etc. Have a look at the ProcessCommand method in the ElementSocket class, which actually does the command processing. When we receive a 'create' request from the WebSocket, we'll create a WPF Window instance based on the input key, and will setup a timer to pump the window as base64 encoded images to the WebSockets. This will be rendered on an HTML5 canvas as shown above.

Here is the method that encodes the WPF screens as GIF images. Have a look at this method:

public static string FrameworkElementToBase64(FrameworkElement c)
{
    using (MemoryStream ms = new MemoryStream())
    {
        int Height = (int)c.ActualHeight;
        int Width = (int)c.ActualWidth;

        RenderTargetBitmap bmp = new RenderTargetBitmap(
                 Width, Height, 96, 96, PixelFormats.Pbgra32);
        bmp.Render(c);
        var encoder = new GifBitmapEncoder();
        encoder.Frames.Add(BitmapFrame.Create(bmp));
        encoder.Save(ms);
        byte[] imageBytes = ms.ToArray();
        // Convert byte[] to Base64 String
        string base64String = Convert.ToBase64String(imageBytes);
        return base64String;
    }
}

Now, a WebSocket can subscribe to a visual element, and from that point onwards we'll send the Visual Element to the WebSocket based on the interval, as a base64 encoded image.

void SetupTimer(int interval)
{
    t = new Timer { Interval = interval };
    t.Elapsed += (e, s) =>
    {
        try
        {
            Send(element.GetContextAsBase64());
        }
        catch (Exception ex)
        {
            Debug.WriteLine(ex.Message);
        }
    };

    t.Start();
}

Capturing user inputs

Presently, the support for input devices (mouse, keyboard) etc., is minimal. Only the mouse-button-down and mouse-button-up events are supported. To do this, we capture the mouse inputs using jQuery, and then pump it back to the server.

$("#canvas").mousedown(function (e) {
    var point = getXY(e);
    socket.send("mousedown;" + point.X + ";" + point.Y);
});

$("#canvas").mouseup(function (e) {
    var point = getXY(e);
    socket.send("mouseup;" + point.X + ";" + point.Y);
});

On the server side, the incoming messages are used to simulate the events on the WPF window instance that corresponds to the socket, like:

var pnt=this.TranslatePoint(new Point(x, y), this);
var res=VisualTreeHelper.HitTest(this, pnt);

if (res != null && res.VisualHit != null && res.VisualHit is UIElement)
{
    SimulateEvent(res, action);
}

In SimulateEvent, we'll invoke the corresponding action on the target element after performing a hit test.

Challenges

There are a number of challenges - ideally, we should push only the region copies once it is modified. As of now, I'm pushing the entire screen image (bad) as that's pretty easy (this is a weekend hack, remember). Now, another challenge is proper user input simulation on the instance windows.

I still need to find a good way to trigger user inputs to the specific windows, without affecting the actual input devices. For example, when a user moves the mouse, the move message should be triggered to the corresponding server side window instance, without affecting the actual mouse cursor. I may need to have a look at the WM_ messages, not sure to what extend it can be applied on WPF top level windows.

Related reads and hacks

  • HTML5 Labs Interop Web Sockets implementation: Read
  • A nice GTK + HTML5 Hack - More mature: Read
  • Enabling WebSockets in Firefox 4 - Read. It is disabled by default in Firefox, though it is available in Chrome. IE9 has a draft version.
  • You can simulate WebSockets using the Flash proxy, if your browser won't support WebSockets - Read. Most browsers do support Canvas though.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author


Comments and Discussions

 
QuestionIs this application works with the latest chrome browser ? Pinmemberkirrik31-Oct-13 1:27 
QuestionProject for converting WPF to HTML5 Pinmembercc626331-May-12 3:23 
GeneralMy vote of 5 PinmvpMarcelo Ricardo de Oliveira25-Apr-11 12:35 
Great article, Anoop, 5 points from me Thumbs Up | :thumbsup:
 
cheers,
marcelo
Take a look at MVC Bricks for ASP.net here in The Code Project.

GeneralMy vote of 5 Pinmvpdefwebserver12-Apr-11 5:48 
GeneralMy vote of 5 Pinmemberwebmaster44212-Apr-11 3:16 
GeneralGreat stuff as usual Anoop PinmvpSacha Barber12-Apr-11 0:47 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web04 | 2.8.140926.1 | Last Updated 12 Apr 2011
Article Copyright 2011 by Anoop Madhusudanan
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid