Why Play Guitar Hero When An Android Can Do It For You?

• Download source - 37.33 KB

Introduction

This DirectShow filter is used to detect the notes inside a GuitarHero game video.

The brain of the Robot was developed by Rafael Mizrahi, a GarageGeeks made-man and research manager at Feng-GUI, the artificial vision lab.

You can look at the GuitarHeroNoid source code home page at Google code service.

Background

In the GuitarHero game, each song is presented on a set of five columns, resembling a real guitar fret board that scrolls constantly towards the player. The five columns correspond to the five fret buttons and appropriately colored notes appear in these columns.

Detecting the notes could be accomplished by using several approaches to detect moving objects; most of them might not be fast enough to implement and use. So we came up with this idea: looking at the game, you can quickly realize that the important information such as the plates is brighter than the rest of the image. Set a detection area which contains 5 trapezoids at the bottom of the picture.

f = 0; // f helps to create a trapezoid.
for (int i=0;i<m_plates_height;i++) {
  for (int j=0;j<m_plates_width-f*2;j++) {
    prgb[j+i*cxImage+(int)f].rgbtRed   = (BYTE) 0;
    prgb[j+i*cxImage+(int)f].rgbtGreen = (BYTE) 0;
    prgb[j+i*cxImage+(int)f].rgbtBlue = (BYTE) 255;
  }
  f = f + 1;
}

Representing the color of the pixels in each trapezoid from RGB as HSV (Hue, Saturation, Value), also known as HSB (Hue, Saturation, Brightness), and defining a threshold (Brightness Threshold at the properties dialog), somewhere in the middle of the Brightness value, gives you a binary representation of the pixels inside the trapezoid. Now, when the trapezoid is filled with enough white pixels, there is probably a plate over there.

// get current pixel RGB
R = (int)prgb[j+i*cxImage+(int)f].rgbtRed;
G = (int)prgb[j+i*cxImage+(int)f].rgbtGreen;
B = (int)prgb[j+i*cxImage+(int)f].rgbtBlue;

// HSV calculation
H=0;S=0;V=0;
RGBtoHSV(R,G,B,&H,&S,&V);

// YUV: doesn't seem as better as HSV
//Y = (( ( 66 * R + 129 * G + 25 * B + 128) >> 8) + 16);
//V = Y;

// threshold V to binary, and sum the white pixels
byte b = (byte)(((V >= min) && (V <= max)) ? 255 : 0);
if(b == (BYTE)255)
  {
    white_sum++;

    ...

Note: Different PlayStations (PAL NTSC) have different brightness.

We connect the game video output using a capture device into a computer. A
live video streaming filter captures the video frames as images and sends each
image into the image processing part of the program that detects the notes.

Send to Robot

The information is sent to the robot. There are three options for sending: File, Port to the parallel output, and Socket. In the socket option, you can divide the process into two computers and by that, perform a Remote surgery.

// play the 5 without a strum hit. infact, release the strum up.
OutputToRobot(((play_plate & PLATE_1) == PLATE_1),
  ((play_plate & PLATE_2) == PLATE_2),
  ((play_plate & PLATE_3) == PLATE_3),
  ((play_plate & PLATE_4) == PLATE_4),
  ((play_plate & PLATE_5) == PLATE_5),
  0 , OUTPUT_TYPE_SOCK);

Delays, Delays, Delays

To play a note, the player must hold the correct fret button and press the strum bar. After playing and watching the game, you find out that the PlayStation adds another delay to the equation. Pressing the frets buttons is recognized by the PlayStation game within some 100 milliseconds or so. Having this delay along with the delay it takes for the strum solenoids to go up and down, I realized that sending fret notes together with the strum action is not possible. I divided the protocol into two main actions:

• The first, at the area of detection, pressing one or more of the fret notes lifts up the strum. The fret notes information is added into a FIFO queue.
• The second, after about 250 milliseconds (Strum Delay at the properties dialog), pops the fret notes from the queue and sends them to the guitar with a strum down.

A nice TODO is to detect the BPM of the song during the first few seconds of the song, and adjust all those delays according to that.

Multiplayer is split-screen. In a "duelling guitars" fashion, two players tackle segments of the selected song. Unlike other modes, it is not possible to fail a song in multiplayer, but scoring dictates that one player will generally win. Just by moving the area of detection to the side of the screen, using smaller trapezoids, all parameters in a different configurations file (guitarhero.ini), and there you go, you can play with or against the GuitarHeronoid.

That's all, guys. Keep on rocking in C++.
You can find more information and videos here.