Click here to Skip to main content
15,867,568 members
Please Sign up or sign in to vote.
4.00/5 (1 vote)
Hello,

I am trying to use HTML5 for and websockets to send a canvas image to a server in C++ using websockets, then manipulate the image with OpenCV and send it back to the webpage. I am able to successfully send and receive data from C++ using websockets to the webpage. However, when i use any of the following all i get is garbage in OpenCV:

JavaScript
var img = canvas.getImageData(0, 0, 320, 240);
var binary = new Uint8Array(img.data.length);
for (var i = 0; i < img.data.length; i++) {
  binary[i] = img.data[i];
}
socket.send(binary.buffer);


Also have tried:
JavaScript
var data = canvas.get()[0].toDataURL('image/jpeg', 1.0);
newblob = dataURItoBlob(data);
socket.send(newblob);

using the function:
function dataURItoBlob(dataURI) {
// convert base64 to raw binary data held in a string
// doesn't handle URLEncoded DataURIs
// convert base64/URLEncoded data component to raw binary data held in a string
	var byteString;
	if (dataURI.split(',')[0].indexOf('base64') >= 0)
		byteString = atob(dataURI.split(',')[1]);
	else
		byteString = unescape(dataURI.split(',')[1]);
	// separate out the mime component
	var mimeString = dataURI.split(',')[0].split(':')[1].split(';')[0]
	
        // write the bytes of the string to an ArrayBuffer
	var ab = new ArrayBuffer(byteString.length);
	var ia = new Uint8Array(ab);
	for (var i = 0; i < byteString.length; i++) {
		ia[i] = byteString.charCodeAt(i);
	}
	// write the ArrayBuffer to a blob, and you're done
	var bb;
	try {
		bb = new BlobBuilder();
	} catch(e) {
		try {
			bb = new WebKitBlobBuilder();
		} catch(e) {
			bb = new MozBlobBuilder();
		}
	}
	bb.append(ab);
	return bb.getBlob(mimeString);
};


Server side I am using:
C++
CvMat mat1 = cvMat(320, 240,CV_8UC3, &buffer);
IplImage *frame = cvDecodeImage(&mat1);
cv::imshow("Input Image", frame);


Also tried the hard way:
C++
CvSize size = cvSize(320, 240); 
int depth  = IPL_DEPTH_8U; 
int channels = 3; 
IplImage *frame =  cvCreateImageHeader(size, depth,channels);
frame->imageData = (char*)buffer;
cv::imshow("Input Image", frame);


And still, all i get in OpenCV is garbage recieved from the HTML5 socket send. It is not a valid jpeg image I am receiving. The image I get is only capable of being displayed on another Canvas itself and not useable in OpenCV.

What I am doing wrong? Any suggestions? Anyone familiar with this problem? And help?
Posted
Comments
enhzflep 26-May-12 22:30pm    
Haven't done much at all with html5, really just some cosmetic stuff.
Having a bit of a look around at the Canvas element, I think there's 1 thing that your code appears to overlook - the bit-depth of the pixel data returned by getImageData.

The examples I can see all indicate that this is 32bit data - this will of cours throw your server side code out of whack.


The other thing that attracts my attention is the use of toDataUrl. I just saw something that discusses why the format is bitmap32 and not png. I didn't read into it much, perhaps you should change the requested format to image/bmp

http://beej.us/blog/data/html5s-canvas-2-pixel/
http://stackoverflow.com/questions/4121142/javascript-getimagedata-for-canvas-html5
_Maxime_D 27-Jul-12 5:36am    
Hello,

I am trying to do the same thing, using HTML5 and websockets to send an image to a C++ server and manipulate it with openCV.

I would like to know if you manage to make it work?
If so, can you tell me wich library did you use for your C++ websockets? How did you include openCV with the websockets? And how did you solve your problem?

Thanks for your reply,

Greeting,

Maxime.
Fred Ackers 28-Jul-12 21:28pm    
I am still working on this project so I am not 100% there yet. I have had limited success with using a C++ library called libwebsockets to communicate with the HTML5 page. The HTML5 page loads the camera, gets the image from the canvas and transmits it to through the websockets to the C++ program. The C++ program grabs the image from the libwebsocket library sockets and then creates an image using OpenCV. I have had success with it, but it is not easy and does not always give me the data I seek for some unknown reason. I am using a C++ custom program that makes use of the OpenCV and libwebsockets for the processing, then sends the updated image back to the HTML5 page. One item worthy of mention, is that I am copying the pixels from the canvas into a buffer and sending that buffer of raw pixels. I then receive the raw buffer and add an image header in the C++ program for OpenCV to use.

Hope this helps!
_Maxime_D 3-Aug-12 3:22am    
Thanks a lot for your answer!

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900