Click here to Skip to main content
Rate this: bad
good
Please Sign up or sign in to vote.
Hello,
 
I am trying to use HTML5 for and websockets to send a canvas image to a server in C++ using websockets, then manipulate the image with OpenCV and send it back to the webpage. I am able to successfully send and receive data from C++ using websockets to the webpage. However, when i use any of the following all i get is garbage in OpenCV:
 
var img = canvas.getImageData(0, 0, 320, 240);
var binary = new Uint8Array(img.data.length);
for (var i = 0; i < img.data.length; i++) {
  binary[i] = img.data[i];
}
socket.send(binary.buffer);
 
Also have tried:
var data = canvas.get()[0].toDataURL('image/jpeg', 1.0);
newblob = dataURItoBlob(data);
socket.send(newblob);
 
using the function:
function dataURItoBlob(dataURI) {
// convert base64 to raw binary data held in a string
// doesn't handle URLEncoded DataURIs
// convert base64/URLEncoded data component to raw binary data held in a string
	var byteString;
	if (dataURI.split(',')[0].indexOf('base64') >= 0)
		byteString = atob(dataURI.split(',')[1]);
	else
		byteString = unescape(dataURI.split(',')[1]);
	// separate out the mime component
	var mimeString = dataURI.split(',')[0].split(':')[1].split(';')[0]
	
        // write the bytes of the string to an ArrayBuffer
	var ab = new ArrayBuffer(byteString.length);
	var ia = new Uint8Array(ab);
	for (var i = 0; i < byteString.length; i++) {
		ia[i] = byteString.charCodeAt(i);
	}
	// write the ArrayBuffer to a blob, and you're done
	var bb;
	try {
		bb = new BlobBuilder();
	} catch(e) {
		try {
			bb = new WebKitBlobBuilder();
		} catch(e) {
			bb = new MozBlobBuilder();
		}
	}
	bb.append(ab);
	return bb.getBlob(mimeString);
};
 
Server side I am using:
CvMat mat1 = cvMat(320, 240,CV_8UC3, &buffer);
IplImage *frame = cvDecodeImage(&mat1);
cv::imshow("Input Image", frame);
 
Also tried the hard way:
CvSize size = cvSize(320, 240); 
int depth  = IPL_DEPTH_8U; 
int channels = 3; 
IplImage *frame =  cvCreateImageHeader(size, depth,channels);
frame->imageData = (char*)buffer;
cv::imshow("Input Image", frame);
 
And still, all i get in OpenCV is garbage recieved from the HTML5 socket send. It is not a valid jpeg image I am receiving. The image I get is only capable of being displayed on another Canvas itself and not useable in OpenCV.
 
What I am doing wrong? Any suggestions? Anyone familiar with this problem? And help?
Posted 26-May-12 11:00am
Comments
enhzflep at 26-May-12 22:30pm
   
Haven't done much at all with html5, really just some cosmetic stuff.
Having a bit of a look around at the Canvas element, I think there's 1 thing that your code appears to overlook - the bit-depth of the pixel data returned by getImageData.
 
The examples I can see all indicate that this is 32bit data - this will of cours throw your server side code out of whack.
 

The other thing that attracts my attention is the use of toDataUrl. I just saw something that discusses why the format is bitmap32 and not png. I didn't read into it much, perhaps you should change the requested format to image/bmp
 
http://beej.us/blog/data/html5s-canvas-2-pixel/
http://stackoverflow.com/questions/4121142/javascript-getimagedata-for-canvas-html5
_Maxime_D at 27-Jul-12 5:36am
   
Hello,
 
I am trying to do the same thing, using HTML5 and websockets to send an image to a C++ server and manipulate it with openCV.
 
I would like to know if you manage to make it work?
If so, can you tell me wich library did you use for your C++ websockets? How did you include openCV with the websockets? And how did you solve your problem?
 
Thanks for your reply,
 
Greeting,
 
Maxime.
Fred Ackers at 28-Jul-12 21:28pm
   
I am still working on this project so I am not 100% there yet. I have had limited success with using a C++ library called libwebsockets to communicate with the HTML5 page. The HTML5 page loads the camera, gets the image from the canvas and transmits it to through the websockets to the C++ program. The C++ program grabs the image from the libwebsocket library sockets and then creates an image using OpenCV. I have had success with it, but it is not easy and does not always give me the data I seek for some unknown reason. I am using a C++ custom program that makes use of the OpenCV and libwebsockets for the processing, then sends the updated image back to the HTML5 page. One item worthy of mention, is that I am copying the pixels from the canvas into a buffer and sending that buffer of raw pixels. I then receive the raw buffer and add an image header in the C++ program for OpenCV to use.
 
Hope this helps!
_Maxime_D at 3-Aug-12 3:22am
   
Thanks a lot for your answer!

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
0 George Jonsson 215
1 Kornfeld Eliyahu Peter 169
2 OriginalGriff 120
3 PIEBALDconsult 110
4 BillWoodruff 85
0 OriginalGriff 6,165
1 DamithSL 4,658
2 Maciej Los 4,087
3 Kornfeld Eliyahu Peter 3,649
4 Sergey Alexandrovich Kryukov 3,294


Advertise | Privacy | Mobile
Web02 | 2.8.141220.1 | Last Updated 26 May 2012
Copyright © CodeProject, 1999-2014
All Rights Reserved. Terms of Service
Layout: fixed | fluid

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100