Click here to Skip to main content
Click here to Skip to main content

Why Play Guitar Hero When An Android Can Do It For You?

, 21 Mar 2007 CPOL
Rate this:
Please Sign up or sign in to vote.
An android that plays the Guitar Hero game. A DirectX DirectShow filters the video feed from the PlayStation2, detects where, when and how to play and moves the fingers accordingly.
Screenshot - gse_multipart62304.jpg

Introduction

This DirectShow filter is used to detect the notes inside a GuitarHero game video.

The brain of the Robot was developed by Rafael Mizrahi, a GarageGeeks made-man and research manager at Feng-GUI, the artificial vision lab.

You can look at the GuitarHeroNoid source code home page at Google code service.

Background

In the GuitarHero game, each song is presented on a set of five columns, resembling a real guitar fret board that scrolls constantly towards the player. The five columns correspond to the five fret buttons and appropriately colored notes appear in these columns.

Detecting the notes could be accomplished by using several approaches to detect moving objects; most of them might not be fast enough to implement and use. So we came up with this idea: looking at the game, you can quickly realize that the important information such as the plates is brighter than the rest of the image. Set a detection area which contains 5 trapezoids at the bottom of the picture.

f = 0; // f helps to create a trapezoid.
for (int i=0;i<m_plates_height;i++) {
  for (int j=0;j<m_plates_width-f*2;j++) {
    prgb[j+i*cxImage+(int)f].rgbtRed   = (BYTE) 0;
    prgb[j+i*cxImage+(int)f].rgbtGreen = (BYTE) 0;
    prgb[j+i*cxImage+(int)f].rgbtBlue = (BYTE) 255;
  }
  f = f + 1;
}

Representing the color of the pixels in each trapezoid from RGB as HSV (Hue, Saturation, Value), also known as HSB (Hue, Saturation, Brightness), and defining a threshold (Brightness Threshold at the properties dialog), somewhere in the middle of the Brightness value, gives you a binary representation of the pixels inside the trapezoid. Now, when the trapezoid is filled with enough white pixels, there is probably a plate over there.

// get current pixel RGB
R = (int)prgb[j+i*cxImage+(int)f].rgbtRed;
G = (int)prgb[j+i*cxImage+(int)f].rgbtGreen;
B = (int)prgb[j+i*cxImage+(int)f].rgbtBlue;

// HSV calculation
H=0;S=0;V=0;
RGBtoHSV(R,G,B,&H,&S,&V);

// YUV: doesn't seem as better as HSV
//Y = (( ( 66 * R + 129 * G + 25 * B + 128) >> 8) + 16);
//V = Y;

// threshold V to binary, and sum the white pixels
byte b = (byte)(((V >= min) && (V <= max)) ? 255 : 0);
if(b == (BYTE)255)
  {
    white_sum++;

    ...

Note: Different PlayStations (PAL NTSC) have different brightness.

We connect the game video output using a capture device into a computer. A
live video streaming filter captures the video frames as images and sends each
image into the image processing part of the program that detects the notes.

Send to Robot

The information is sent to the robot. There are three options for sending: File, Port to the parallel output, and Socket. In the socket option, you can divide the process into two computers and by that, perform a Remote surgery.

// play the 5 without a strum hit. infact, release the strum up.
OutputToRobot(((play_plate & PLATE_1) == PLATE_1),
  ((play_plate & PLATE_2) == PLATE_2),
  ((play_plate & PLATE_3) == PLATE_3),
  ((play_plate & PLATE_4) == PLATE_4),
  ((play_plate & PLATE_5) == PLATE_5),
  0 , OUTPUT_TYPE_SOCK);

Delays, Delays, Delays

To play a note, the player must hold the correct fret button and press the strum bar. After playing and watching the game, you find out that the PlayStation adds another delay to the equation. Pressing the frets buttons is recognized by the PlayStation game within some 100 milliseconds or so. Having this delay along with the delay it takes for the strum solenoids to go up and down, I realized that sending fret notes together with the strum action is not possible. I divided the protocol into two main actions:

  • The first, at the area of detection, pressing one or more of the fret notes lifts up the strum. The fret notes information is added into a FIFO queue.
  • The second, after about 250 milliseconds (Strum Delay at the properties dialog), pops the fret notes from the queue and sends them to the guitar with a strum down.

A nice TODO is to detect the BPM of the song during the first few seconds of the song, and adjust all those delays according to that.

Screenshot - guitar_hero_multiplayer1_small.jpg

Multiplayer is split-screen. In a "duelling guitars" fashion, two players tackle segments of the selected song. Unlike other modes, it is not possible to fail a song in multiplayer, but scoring dictates that one player will generally win. Just by moving the area of detection to the side of the screen, using smaller trapezoids, all parameters in a different configurations file (guitarhero.ini), and there you go, you can play with or against the GuitarHeronoid.

That's all, guys. Keep on rocking in C++.
You can find more information and videos here.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Rafael Mizrahi
Chief Technology Officer Feng-GUI
United States United States
Rafael Mizrahi is a CTO at Feng-GUI
 
He is also a hacker contributor at the mono project, and a GarageGeeks member.
 
Rafael Mizrahi personal blog -
http://rafaelmizrahi.blogspot.com

Comments and Discussions

 
GeneralCool project PinmemberScope23-Mar-07 6:07 
GeneralRe: Cool project PinmemberRafael Mizrahi23-Mar-07 8:24 
Hey Scope, thanks!
Let me share a secret with you. that was the only second time the robot was working. we worked on it during 2 weeks of sleepless nights.

GeneralEasily underestimated in coolness! PinmemberShawn Poulson22-Mar-07 1:47 
GeneralRe: Easily underestimated in coolness! PinmemberRafael Mizrahi22-Mar-07 23:09 
GeneralCool article PinmemberETA21-Mar-07 20:27 
GeneralRe: Cool article PinmemberRafael Mizrahi21-Mar-07 21:05 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web02 | 2.8.141015.1 | Last Updated 21 Mar 2007
Article Copyright 2007 by Rafael Mizrahi
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid