Click here to Skip to main content
11,718,016 members (86,379 online)
Rate this: bad
Please Sign up or sign in to vote.

I am trying to use HTML5 for and websockets to send a canvas image to a server in C++ using websockets, then manipulate the image with OpenCV and send it back to the webpage. I am able to successfully send and receive data from C++ using websockets to the webpage. However, when i use any of the following all i get is garbage in OpenCV:

var img = canvas.getImageData(0, 0, 320, 240);
var binary = new Uint8Array(;
for (var i = 0; i <; i++) {
  binary[i] =[i];

Also have tried:
var data = canvas.get()[0].toDataURL('image/jpeg', 1.0);
newblob = dataURItoBlob(data);
using the function:
function dataURItoBlob(dataURI) {
// convert base64 to raw binary data held in a string
// doesn't handle URLEncoded DataURIs
// convert base64/URLEncoded data component to raw binary data held in a string
	var byteString;
	if (dataURI.split(',')[0].indexOf('base64') >= 0)
		byteString = atob(dataURI.split(',')[1]);
		byteString = unescape(dataURI.split(',')[1]);
	// separate out the mime component
	var mimeString = dataURI.split(',')[0].split(':')[1].split(';')[0]
        // write the bytes of the string to an ArrayBuffer
	var ab = new ArrayBuffer(byteString.length);
	var ia = new Uint8Array(ab);
	for (var i = 0; i < byteString.length; i++) {
		ia[i] = byteString.charCodeAt(i);
	// write the ArrayBuffer to a blob, and you're done
	var bb;
	try {
		bb = new BlobBuilder();
	} catch(e) {
		try {
			bb = new WebKitBlobBuilder();
		} catch(e) {
			bb = new MozBlobBuilder();
	return bb.getBlob(mimeString);

Server side I am using:
CvMat mat1 = cvMat(320, 240,CV_8UC3, &buffer);
IplImage *frame = cvDecodeImage(&mat1);
cv::imshow("Input Image", frame);

Also tried the hard way:
CvSize size = cvSize(320, 240); 
int depth  = IPL_DEPTH_8U; 
int channels = 3; 
IplImage *frame =  cvCreateImageHeader(size, depth,channels);
frame->imageData = (char*)buffer;
cv::imshow("Input Image", frame);

And still, all i get in OpenCV is garbage recieved from the HTML5 socket send. It is not a valid jpeg image I am receiving. The image I get is only capable of being displayed on another Canvas itself and not useable in OpenCV.

What I am doing wrong? Any suggestions? Anyone familiar with this problem? And help?
Posted 26-May-12 10:00am
enhzflep at 26-May-12 22:30pm
Haven't done much at all with html5, really just some cosmetic stuff.
Having a bit of a look around at the Canvas element, I think there's 1 thing that your code appears to overlook - the bit-depth of the pixel data returned by getImageData.

The examples I can see all indicate that this is 32bit data - this will of cours throw your server side code out of whack.

The other thing that attracts my attention is the use of toDataUrl. I just saw something that discusses why the format is bitmap32 and not png. I didn't read into it much, perhaps you should change the requested format to image/bmp
_Maxime_D at 27-Jul-12 5:36am

I am trying to do the same thing, using HTML5 and websockets to send an image to a C++ server and manipulate it with openCV.

I would like to know if you manage to make it work?
If so, can you tell me wich library did you use for your C++ websockets? How did you include openCV with the websockets? And how did you solve your problem?

Thanks for your reply,


Fred Ackers at 28-Jul-12 21:28pm
I am still working on this project so I am not 100% there yet. I have had limited success with using a C++ library called libwebsockets to communicate with the HTML5 page. The HTML5 page loads the camera, gets the image from the canvas and transmits it to through the websockets to the C++ program. The C++ program grabs the image from the libwebsocket library sockets and then creates an image using OpenCV. I have had success with it, but it is not easy and does not always give me the data I seek for some unknown reason. I am using a C++ custom program that makes use of the OpenCV and libwebsockets for the processing, then sends the updated image back to the HTML5 page. One item worthy of mention, is that I am copying the pixels from the canvas into a buffer and sending that buffer of raw pixels. I then receive the raw buffer and add an image header in the C++ program for OpenCV to use.

Hope this helps!
_Maxime_D at 3-Aug-12 3:22am
Thanks a lot for your answer!

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
0 OriginalGriff 215
1 F-ES Sitecore 210
2 Maciej Los 140
3 Jochen Arndt 120
4 Andy Lanng 115
0 Sergey Alexandrovich Kryukov 1,649
1 OriginalGriff 535
2 Maciej Los 529
3 Richard MacCutchan 440
4 CHill60 415

Advertise | Privacy | Mobile
Web04 | 2.8.150901.1 | Last Updated 26 May 2012
Copyright © CodeProject, 1999-2015
All Rights Reserved. Terms of Service
Layout: fixed | fluid

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100