Click here to Skip to main content
Click here to Skip to main content

Celerity: Sensory Overload

, 18 Oct 2012
Rate this:
Please Sign up or sign in to vote.
Sensor-controlled XNA Tunnel Game with VR Head-Tracking
This is an old version of the currently published article.

Please note

This article is an entry in our AppInnovation Contest. Articles in this sub-section are not required to be full articles so care should be taken when voting.

Introduction 

This competition submission is a game called Celerity: Sensory Overload. It combines the classic Tunnel Game genre with sensor control and most notably a novel take on 3D.


Background  

The main inspirations for this game are: 

How could I resist?  

Time was tight as I only started after the details of the competition went up. I invited a couple of talented friends to help me out; @Dave_Panic (3D trickery guru) and @BreadPuncher (game theorist & graphic designer).

I was thinking about doing something along the lines of gyro-based steering and/or perspective and Dave suggested doing the head tracking effect with the webcam. Marrying this concept with a classic tunnel game we had the foundation of an innovative, sensor-tastic game. You know how gamers subconsciously tend to lean and move their head around when playing some games even though it makes no difference? Now it will! 

VR Head Tracking With Just A Webcam 

In Johnny Lee's video, above, he is able to detect the position of the user's head in 3D space through infra red (IR) LEDs and an IR camera. 

The effect is fantastic but requires a special IR sensor and also that the user wears an IR-emitting device. I needed to create this effect using only sensors on an Intel Ultrabook, and no other equipment. Thankfully the Ultrabook has a webcam, so we can achieve this in a round-about way.

The application processes each frame from the webcam and run it through a Computer Vision (CV) image-processing library, EMGUCV. This returns a rectangle within a box, representing the relative position of the face within the view of the camera. As the user moves their face, the rectangle will move around, giving me an X/Y offset of the user's face. The X & Y of the 3D world's view can be skewed in relation to the user's own physical position. Note that we need to flip the image horizontally as the webcam is looking in the opposite direction to us.  

This effect can be taken even further, as the rectangle representing the user's face inherently has a size. This gets larger as they move towards the webcam, so we can also determine the relative Z position, too.  

It's helpful if the user starts with their head roughly central, which we assist with as part of the intro menu UX. There the user can see if they're vaguely "calibrated" before starting. 

Designing The UI    

I wanted to make use of Metro design principles in the user interface, as although this is a desktop game and not a Metro app, I value consistency and feel the UI will be more intuitive if it shares conventions with the operating system, Windows 8. 

Here are the layout mocks for the UI. For the final thing, imagine the animated tunnel effect behind the UI, only faded out somewhat. 

 

Building The App   

I wanted to do the app as an XNA app, as it's perfectly fast enough and nice and easy to code for. I don't know C++ yet and there's not enough time to both learn it and complete the game. Unfortunately XNA is an awkward combination with Windows 8, as it's not supported in Visual Studio 2012, at least at the time of writing. My thanks goes to Ibrahim's article which helped me initially create the project. Steps 1 & 2 are essentially from there.     

Step 1: Project Pains  

Step 1 was to make a project in VS2012 running on the XNA Framework. This was achieved by initially creating the application as a class library, importing all the XNA DLLs, and then converting to an x86 windows application. Manually create the following:  

The launcher class:  

static class Program
{
    static void Main(string[] args)
    {
        using (var game = new CelerityGame())
        {
            game.Run();
        }
    }
}  

The game class:

public partial class CelerityGame : Game
{
    GraphicsDeviceManager graphics;
    SpriteBatch spriteBatch;
 
    public CelerityGame()
    {
         graphics = new GraphicsDeviceManager(this);
         Content.RootDirectory = "Content";
    }
} 

The Initialize method:

protected override void Initialize()
{ 
    base.Initialize();
} 

The LoadContent method:

protected override void LoadContent()
{
    spriteBatch = new SpriteBatch(GraphicsDevice);
} 

The Update method:

protected override void Update(GameTime gameTime)
{
    base.Update(gameTime);
}  

Step 2: Nearly Content 

Although there's no proper "Content" project you can still use the ContentManager class directly to load resources. For example: 

// "Content" is a folder in the root of my game project
Texture2D t = Texture2D.FromStream(graphics, TitleContainer.OpenStream(@"Content\example.png"))   

Step 3: SpriteFont Workaround  

Fonts are a bit more tricky. The issue is that whilst you have the class, you don't have SpriteFonts as project items in VS2012 so it requires a workaround. My solution is to simply create a Windows app in VS2010,  insert a SpriteFont there, build the project, and then copy the .XNB file from the bin directory over as a resource.  

It can then be loaded like this: 

// I have a file called "SegoeUILight56.xnb" in a subfolder Fonts within the main Content folder
// Note that unlike with Images, the code doesn't refer to "Content" or ".xnb", as these are assumed
SpriteFont f = contentManager.Load<SpriteFont>(@"Fonts\SegoeUILight56");  

Step 4: The Sound of Music 

Audio content files are similar to SpriteFonts. I used VS2010 to import an .MP3 content file. When it builds it both converts it to .WMA and produces an .XNB file. Copy both these files from the bin directory as before, and load like this:  

Song s = contentManager.Load<Song>(@"Audio\mySongName"); 

Play your song as normal:

MediaPlayer.Play(s); 

Step 5: Webcam Woes (and EMGUCV)

In this project I'm using EMGUCV version 2.4.0. I was using 2.4.2. but was able to get a much smaller DLL profile with 2.4.0. whilst still keeping everything I needed. It was a cinch to perform a basic capture from a webcam thanks to their simple tutorials, however when I attempted something more ambitious I ran into two related problems. 

  • EMGUCV gives me a System.Drawing.Bitmap rather than a Texture2D
  • The performance became disastrous 

I found a number of examples of converters online and tried several but still had the performance issues. I suspected it was an issue with running it on a single thread, but when searching the net I found many people warning against using threading in XNA. Eventually I ignored their advice and tried a combination of this method of image conversion and my first ever use of Parallel.Invoke(). All of a sudden worked a treat without me ever having to quite get to the bottom of it! Don't you just love it when that happens? 

Here's my naïve but incredibly effective solution: 

Parallel.Invoke(() => QueryCamera(elapsedMilliseconds));  

Simply by calling my existing method like this instead of directly from my Update method it was insta-fixed. Sweet!

Yes, okay I ought to read what that's actually doing but first I have a game to make.  Smile | <img src=   

One last note on the head tracking; over time the appearance of the resultant rectangle is a bit jittery z-axis wise. This is because it often catches two rectangles per face, but sometimes just either one. This makes for frequent changes in size. I fixed this by taking a recent sample history and taking a rolling average to smooth it out.    

Step 6: Raster To Go Faster 

In the hope of building a responsive UI quickly I was keen to try a library Sacha Barber mentioned, XNAML. In theory this would allow me to build the UI layer in WPF, with its masterful handling of  whilst keeping the tunnel rendering layer in XNA.

With a little fiddling I got the library working, a simple test game canvas beneath a button on top. Unfortunately XNA layer was running at a fixed 96dpi whereas the  WPF layer adjusted to my operating system setting, 120. So close but so far.

This means I'm building the UI purely in bitmaps. Given the extremities of possible screen resolutions for a Windows 8 app (1024 wide -> 2560+ wide) making it responsive and still look good is quite a lot of work. Unfortunately this involves lots of rasterising vector assets, calculating coordinates, and of course trial & error. 

If anyone knows a way of getting XNA & WPF to play nicely, full screen and for different resolutions etc. then please do let me know. 

Step 7: Alpha Mares 

To be continued. (How a default setting made all our graphics look screwy and the simple fix) 

Step 8: Dark At The End Of The Tunnel  

To be continued. (Work on the tunnel has started and our initial problems with the indices are fixed - our tunnel is now hollow!) 

Step 9: Gyromancy (Steering)    

To be continued.   

Step 10: Collision Course 

To be continued. 

Step 11: Spit & Polish 

To be continued. 

Step 12: Release 

To be continued. 

Intended Sensor-Specific Features    

I'm deliberately keeping the app as straight-forward as possible with the initial release to ensure it gets done. I'd prefer to do something focused well, than have lots of features rushed in. Here is the basic "V1" sensor feature set:   

  • Webcam-based VR Head Tracking  
  • Gyro-based Steering 

"If I have time" Potential Features  

Here are some more features I'll love to stick in either according to time once the basic app is done or possibly as a later update:  

  • Multi-touch thumb weapon controls - why dodge an obstacle when you can blast it, right? This will be two areas of the screen which independently respond to thumb taps. Maybe even different gestures for different weapons, depending how viable a thumb gesture is when twirling an Ultrabook.  
  • Subtle contrast adjustment according to the ambient light sensor. I want to keep the look of the game similar, but the greys on white can become full black on white, for example. 
  • Shake-to-escape emergency random teleport feature, a bit like "Hyperspace" in Asteroids-style games.    
  • Alternate controls, e.g. gamepad, keyboard, in case peoples' arms get tired swinging it around (this thing is light, right?)  
  • A number of additional gameplay mechanics (e.g. power-ups, alt game-modes etc.) 
  • Some proper sound design, but this might require bringing someone else in who has experience in this area. Volunteers welcome!      
  • Original music score (I'd love to do this myself, but I'm busy coding at the moment) 

Coming Soon     

There's still plenty to do, but it's taking shape very quickly and looks to be on schedule. The head tracking is working great. I will post screenshots and more juicy source in due course. 

My exam is out the way now, so it's full speed ahead. This weekend we'll be beginning proper Win8 & UB testing, as well as merging the two currently separate projects (tunnel and UI) which go to make this app. No doubt my lack of Git skills will lead to some gigantic loss of vital code... Blush | :O  

Feel free to bookmark this article or follow my Twitter feed as I'll keep updating them with the progress. 

Category 

This app is intended as a submission in the Desktop Games category. 

History  

  • V1 First Draft    
  • V2 Minor corrections & progress updates 
  • V3 Fixed typo and minor updates 

 

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

Adam David Hill
Software Developer
United Kingdom United Kingdom

Bio:

Musician turned Software Engineer (turned professional around 6 years ago). Mainly interested in games & mobility.

Sometimes I do real work, too.

 

My articles:

 

My open source software:

Follow on   Twitter   Google+   LinkedIn

Comments and Discussions


Discussions posted for the Published version of this article. Posting a message here will take you to the publicly available article in order to continue your conversation in public.
 
QuestionBravo Adam!!!! PinmemberAbhishek Nandy6-Dec-12 19:36 
AnswerRe: Bravo Adam!!!! PinmemberAdam David Hill6-Dec-12 21:40 
GeneralMy vote of 5 PinmemberFlorian Rappl6-Dec-12 7:33 
GeneralRe: My vote of 5 PinmemberAdam David Hill6-Dec-12 22:01 
QuestionThis is such a cool Ultrabook idea PinprotectorPete O'Hanlon19-Nov-12 20:58 
AnswerRe: This is such a cool Ultrabook idea PinmemberAdam David Hill29-Nov-12 12:22 
GeneralRe: This is such a cool Ultrabook idea PinprotectorPete O'Hanlon29-Nov-12 12:27 
GeneralRe: This is such a cool Ultrabook idea PinprotectorPete O'Hanlon29-Nov-12 22:31 
QuestionMy Vote of 5 PinmemberAbhishek Nandy22-Oct-12 7:56 
GeneralMy vote of 5 PinmemberAbhishek Nandy22-Oct-12 7:55 
GeneralMy vote of 5 PinmemberShintu Dhang11-Oct-12 6:43 
GeneralRe: My vote of 5 PinmemberAdam David Hill11-Oct-12 6:59 
GeneralRe: My vote of 5 PinmemberShintu Dhang11-Oct-12 7:03 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web01 | 2.8.140721.1 | Last Updated 18 Oct 2012
Article Copyright 2012 by Adam David Hill
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid