Click here to Skip to main content
15,881,967 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
I am making a camera mouse software. Real time video is processed to track movement of the feature and translate to the mouse movement.

A search window technique is bing used. A template matching using correlation is the technique applied. Its coded in matlab.

What happens is the code works fine with a prerecorded video but not in real time.

Any idea why?
Posted

don't forget that correlation depends on the current state of the input data and a number of frames of old data (depending on correlation size), with recorded data, you have the present, past, and future data all available. in real-time you only have present data, make sure you have enough old data buffered to allow correlation process to work correctly.
 
Share this answer
 
<b><b></b>i did not quite get what u are saying.
the template is static. does not change. we need to find that template in subsequent frames.
It works very precisely with a prerecorded video.
Is it to with the correlation formula itself?
Im attaching the code here.

trial2.m (for taking the feature to be tracked. you are to click on the image which is captured and displayed. a template around the point of 20 x 20 is saved)

imaqreset;
clc;
clear;

vid = videoinput('winvideo',1,'YUY2_320x240');

% Configuring Parameter
set(vid,'FramesPerTrigger',3)
set(vid,'FrameGrabInterval',3)
%vid.TriggerRepeat = inf;
set(vid,'LoggingMode','Memory')
triggerconfig(vid,'manual')

% Set Memory
imaqmem(7000000000)
cnt = imaqmem;
mem_low = 1000000000;
mem_low1 = 2000000000;
mem_left = cnt.FrameMemoryLimit-cnt.FrameMemoryUsed;
start(vid);
%get mouse input and the trigger
preview(vid);
trigger(vid);
%end

frame = getsnapshot(vid);
frame=rgb2gray(frame);
delete(vid);

imshow(frame);
[inx iny]=ginput(1);
rectangle('Position',[inx-10,iny-10,20,20]);
temp=frame(iny-10:iny+10,inx-10:inx+10);

%figure(1);
%imshow(temp); %to disp template
r=eyeball2(temp,iny,inx);


|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
eyeball2.m (the function which captures each frame and send it for template matching to mod3.m)


function r= eyeball2(template,x1,y1)
%Resetting the image acquisition tool
imaqreset;

r=-1;
vid = videoinput('winvideo',1,'YUY2_320x240');

% Configuring Parameter
%set(vid,'FramesPerTrigger',1)
%set(vid,'FrameGrabInterval',1)
%vid.TriggerRepeat = inf;
%set(vid,'LoggingMode','Memory')

triggerconfig(vid,'manual');

% Set Memory
imaqmem(700000000);
cnt = imaqmem;
mem_low = 10000000;
mem_low1 = 20000000;
mem_left = cnt.FrameMemoryLimit-cnt.FrameMemoryUsed;
start(vid);
%preview(vid);
%trigger(vid);
I = getsnapshot(vid);
i=1;
while(mem_left > mem_low)
%while(i<=20)
%trigger(vid);
I = getsnapshot(vid);
I = rgb2gray(I);

figure(1);
%imshow(I);
imagesc(I);
%frame(:,:,i)=I;
hold on;
[a b]=mod3(I,template,x1,y1,i);
%hold off;
x1=a;
y1=b;
i=i+1;
%figure;
%imshow(template);
% if (mem_left < mem_low1)
% clear(frame);
% flushdata(vid);
% mem_left = cnt.FrameMemoryLimit-cnt.FrameMemoryUsed;
% i=1;
% end
i=i+1;
end
stop(vid);



|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
mod3.m ( the function which extracts the mask from the frame and sends to the correaltion function and sets teh mouse pointer accordingly. Psychtoolbox needs to be installed for the GetMouse() and SetMouse() functions to work.


function [x2,y2]=mod3(i1,i2,x1,y1,i)

%function
%arguments
%i1: frame from webcam
%i2: template for comparison
%x1: previous x coordinates of centre of template
%y1: previous y coordinates of centre of template

%return values
%x2: current x coordinates of centre of new template
%y2: current y coordinates of centre of new template
%x=rows (max 240)
%y=cols (max 320)

searchx=x1-50;
searchy=y1-50;
searchh=100;
searchw=100;

if(x1-50)<=0
searchx=1;
end
if(x1+50)>=240
searchx=139;
end
if(y1-50)<=0
searchy=1;
end
if(y1+50)>=320
searchy=219;
end

i1=i1(searchx:searchx+searchh,searchy:searchy+searchw); %i1 now has the search window
%figure(1);
%imshow(i1);

[r1 c1]=size(i1);
[r2 c2]=size(i2);
% corr=normxcorr2(i2,i1);
corr=zeros(r1-r2+1,c1-c2+1); % correlation matrix

for i=1:r1-r2+1
for j=1:c1-c2+1
mask=i1(i:i+r2-1,j:j+c2-1);
corr(i,j)=correlationn(mask,i2);

end
end

%finding maximum
[x y]=max(corr);
[x z]=max(x);
a1=y(z);
b1=z;
x %printing maximum value of correlation

%adding offset to get coordinates relative to entire frame
x2=(a1+r2/2+searchx);
y2=(b1+c2/2+searchy);
%converting to integer i.e. whole number
x2=int32(x2);
y2=int32(y2);
%set(gca, 'Color', [0, 1, 0]);
rectangle('Position',[y2-8,x2-5,c2,r2]);

disx=abs(x2-x1); %absolute displacement of mouse in x direction
disy=abs(y2-y1); %absolute displacement of mouse in y direction

dispunit=1;

%obtaining current location of mouse pointer
[mx my]=GetMouse();

if(disx<5)
my=my;
elseif(x2>x1)
my=my+ disx*dispunit;
elseif(x2<x1)
my=my- disx*dispunit;
else
my=my;
end

if(disy<5)
mx=mx;
elseif y2>=y1
mx=mx+disy*dispunit;
elseif(y2<y1)
mx=mx- disy*dispunit;
else
mx=mx;
end
SetMouse(mx,my);


function result=correlationn(s,t)
s=double(s);
t=double(t);
[m n]=size(s);
a=m*n;

mul=0;
sums=sum(sum(s));
sumt=sum(sum(t));
sqs=sum(sum(s.^2));
sqt=sum(sum(s.^2));
mul=sum(sum(s.*t));
if(a*sqs-sums*sums <=0)
result =0;
elseif(a*sqt-sumt*sumt<=0)
result=0;
else
sigs=sqrt(a*sqs-sums*sums);
sigt=sqrt(a*sqt-sumt*sumt);
result=(a*mul-sums*sumt)/(sigs*sigt);
end


|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

If anyone can find a solution/ the rror please let me know. Thank you.</b>
 
Share this answer
 

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900