Click here to Skip to main content
Click here to Skip to main content

Celerity: Sensory Overload

By , 9 Oct 2012
Rate this:
Please Sign up or sign in to vote.
This is an old version of the currently published article.

Please note

This article is an entry in our AppInnovation Contest. Articles in this sub-section are not required to be full articles so care should be taken when voting.

Introduction 

This competition submission is a game called Celerity: Sensory Overload. It combines the classic Tunnel Game genre with sensor control and most notably a novel take on 3D.



Background  

The main inspirations for this game are: 

How could I resist?  

Time was tight as I only started after the details of the competition went up. I invited a couple of talented friends to help me out; @Dave_Panic (3D trickery guru) and @BreadPuncher (game theorist & designer).

I was thinking about doing something along the lines of gyro-based steering and/or perspective and Dave suggested doing the head tracking effect with the webcam. Marrying this concept with a classic tunnel game we had the foundation of an innovative, sensor-tastic game. You know how gamers subconsciously tend to lean and move their head around when playing some games even though it makes no difference? Now it will! 

VR Head Tracking With Just A Webcam 

In Johnny Lee's video, above, he is able to detect the position of the user's head in 3D space through infra red (IR) LEDs and an IR camera. 

The effect is fantastic but requires a special IR sensor and also that the user wears an IR-emitting device. I needed to create this effect using only sensors on an Intel Ultrabook, and no other equipment. Thankfully the Ultrabook has a webcam, so we can achieve this in a round-about way.

The application processes each frame from the webcam and run it through a Computer Vision (CV) image-processing library, EMGUCV. This returns two rectangles — one which represents the bounds of the image, and one which tells us the bounds of the user's face. The first rectangle is obviously static, and the other rectangle moves around in relation to the border, according to the X/Y offset of the user's face. The X & Y of the 3D world's view can be skewed in relation to the user's own physical position. Note that we need to flip the image horizontally as the webcam is looking in the opposite direction to us. 

This effect can be taken even further, as the rectangle representing the user's face inherently has a size. This gets larger as they move towards the webcam, so we can also determine the relative Z position, too.  

All this presumes that the user is starting with their head position "calibrated", which we handle as part of the intro menu UX. 

Designing The UI    

I wanted to make use of Metro design principles in the user interface, as although this is a desktop game and not a Metro app, I value consistency and feel the UI will be more intuitive if it shares conventions with the operating system, Windows 8.

Here are the layout mocks for the UI. For the final thing, imagine the animated tunnel effect behind the UI, only faded out somewhat. 

 

Building The App   

I wanted to do the app as an XNA app, as it's perfectly fast enough and nice and easy to code for. I don't know C++ yet and there's not enough time to both learn it and complete the game. Unfortunately XNA is an awkward combination with Windows 8, as it's not supported in Visual Studio 2012, at least at the time of writing. My thanks goes to Ibrahim's article which helped me initially create the project. Steps 1 & 2 are essentially from there.    

Step 1: Project Pains  

Step 1 was to make a project in VS2012 running on the XNA Framework. This was achieved by initially creating the application as a class library, importing all the XNA DLLs, and then converting to an x86 windows application. Manually create the following:  

The launcher class:  

static class Program
{
    static void Main(string[] args)
    {
        using (var game = new CelerityGame())
        {
            game.Run();
        }
    }
}  

The game class:

public partial class CelerityGame : Game
{
    GraphicsDeviceManager graphics;
    SpriteBatch spriteBatch;
 
    public CelerityGame()
    {
         graphics = new GraphicsDeviceManager(this);
         Content.RootDirectory = "Content";
    }
} 

The Initialize method:

protected override void Initialize()
{ 
    base.Initialize();
} 

The LoadContent method:

protected override void LoadContent()
{
    spriteBatch = new SpriteBatch(GraphicsDevice);
} 

The Update method:

protected override void Update(GameTime gameTime)
{
    base.Update(gameTime);
}  

Step 2: Nearly Content 

Although there's no proper "Content" project you can still use the ContentManager class directly to load resources. For example: 

// "Content" is a folder in the root of my game project
Texture2D t = Texture2D.FromStream(graphics, TitleContainer.OpenStream(@"Content\example.png"))   

Step 3: SpriteFont Workaround  

Fonts are a bit more tricky. The issue is that whilst you have the class, you don't have SpriteFonts as project items in VS2012 so it requires a workaround. My solution is to simply create a Windows app in VS2010,  insert a SpriteFont there, build the project, and then copy the .XNB file from the bin directory over as a resource.  

It can then be loaded like this: 

// I have a file called "SegoeUILight56.xnb" in a subfolder Fonts within the main Content folder
// Note that unlike with Images, the code doesn't refer to "Content" or ".xnb", as these are assumed
SpriteFont f = contentManager.Load<SpriteFont>(@"Fonts\SegoeUILight56");  

Step 4: The Sound of Music 

Audio content files are similar to SpriteFonts. I used VS2010 to import an .MP3 content file. When it builds it both converts it to .WMA and produces an .XNB file. Copy both these files from the bin directory as before, and load like this:  

Song s = contentManager.Load<Song>(@"Audio\mySongName"); 

Play your song as normal:

MediaPlayer.Play(s); 

Step 5: Webcam Woes (and EMGUCV)

In this project I'm using EMGUCV version 2.4.0. I was using 2.4.2. but was able to get a much smaller DLL profile with 2.4.0. whilst still keeping everything I needed. It was a cinch to perform a basic capture from a webcam thanks to their simple tutorials, however when I attempted something more ambitious I ran into two related problems. 

  • EMGUCV gives me a System.Drawing.Bitmap rather than a Texture2D
  • The performance became disastrous 

I found a number of examples of converters online and tried several but still had the performance issues. I suspected it was an issue with running it on a single thread, but when searching the net I found many people warning against using threading in XNA. Eventually I ignored their advice and tried a combination of this method of image conversion and my first ever use of Parallel.Invoke(). All of a sudden worked a treat without me ever having to quite get to the bottom of it! Don't you just love it when that happens? 

Here's my naïve but incredibly effective solution: 

Parallel.Invoke(() => QueryCamera(elapsedMilliseconds));  

Simply by calling my existing method like this instead of directly from my Update method it was insta-fixed. Sweet!

Yes, okay I ought to read what that's actually doing but first I have a game to make.  Smile | <img src=   

One last note on the head tracking; over time the appearance of the resultant rectangle is a bit jittery z-axis wise. This is because it often catches two rectangles per face, but sometimes just either one. This makes for frequent changes in size. I fixed this by taking a recent sample history and taking a rolling average to smooth it out.   

Step 6: Dark At The End Of The Tunnel 

To be continued.   

Step 7: Gyromancy (Steering)  

To be continued.   

Step 8: Collision Course 

To be continued. 

Step 9: Spit & Polish 

To be continued. 

Step 10: Release 

To be continued. 

Intended Sensor-Specific Features    

I'm deliberately keeping the app as straight-forward as possible with the initial release to ensure it gets done. I'd prefer to do something focused well, than have lots of features rushed in. Here is the basic "V1" sensor feature set:  

  • Webcam-based VR Head Tracking 
  • Gyro-based Steering 

"If I have time" Potential Features 

Here are some more features I'll love to stick in either according to time once the basic app is done or possibly as a later update: 

  • Multi-touch thumb weapon controls - why dodge an obstacle when you can blast it, right? This will be two areas of the screen which independently respond to thumb taps. Maybe even different gestures for different weapons, depending how viable a thumb gesture is when twirling an Ultrabook.  
  • Subtle contrast adjustment according to the ambient light sensor. I want to keep the look of the game similar, but the greys on white can become full black on white, for example. 
  • Shake-to-escape emergency random teleport feature, a bit like "Hyperspace" in Asteroids-style games.   
  • Alternate controls, e.g. gamepad, keyboard, in case peoples' arms get tired swinging it around (this thing is light, right?)  
  • A number of additional gameplay mechanics (e.g. power-ups, alt game-modes etc.) 
  • Some proper sound design, but this might require bringing someone else in who has experience in this area. Volunteers welcome!     
  • Original music score (I'd love to do this myself, but I'm busy coding at the moment) 

Coming Soon    

There's still plenty to do, but it's taking shape very quickly and looks to be on schedule. The head tracking is working great. I will post screenshots and more juicy source in due course. Now to do the tunnel... 

Note: I have an exam this week so the article might go quiet for 3/4 days as I need to cram! 

Feel free to bookmark this article or follow my Twitter feed as I'll keep updating them with the progress. 

Category 

This app is intended as a submission in the Desktop Games category. 

History 

First Draft   

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

Adam David Hill
Software Developer
United Kingdom United Kingdom

Bio:

Musician turned Software Engineer (turned professional around 6 years ago). Mainly interested in games & mobility.

Sometimes I do real work, too.

 

My articles:

 

My open source software:

Follow on   Twitter   Google+   LinkedIn

Comments and Discussions


Discussions posted for the Published version of this article. Posting a message here will take you to the publicly available article in order to continue your conversation in public.
 
QuestionBravo Adam!!!! PinmemberAbhishek Nandy6-Dec-12 19:36 
AnswerRe: Bravo Adam!!!! PinmemberAdam David Hill6-Dec-12 21:40 
GeneralMy vote of 5 PinmemberFlorian Rappl6-Dec-12 7:33 
GeneralRe: My vote of 5 PinmemberAdam David Hill6-Dec-12 22:01 
QuestionThis is such a cool Ultrabook idea PinprotectorPete O'Hanlon19-Nov-12 20:58 
AnswerRe: This is such a cool Ultrabook idea PinmemberAdam David Hill29-Nov-12 12:22 
GeneralRe: This is such a cool Ultrabook idea PinprotectorPete O'Hanlon29-Nov-12 12:27 
GeneralRe: This is such a cool Ultrabook idea PinprotectorPete O'Hanlon29-Nov-12 22:31 
QuestionMy Vote of 5 PinmemberAbhishek Nandy22-Oct-12 7:56 
GeneralMy vote of 5 PinmemberAbhishek Nandy22-Oct-12 7:55 
GeneralMy vote of 5 PinmemberShintu Dhang11-Oct-12 6:43 
GeneralRe: My vote of 5 PinmemberAdam David Hill11-Oct-12 6:59 
GeneralRe: My vote of 5 PinmemberShintu Dhang11-Oct-12 7:03 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web04 | 2.8.140415.2 | Last Updated 9 Oct 2012
Article Copyright 2012 by Adam David Hill
Everything else Copyright © CodeProject, 1999-2014
Terms of Use
Layout: fixed | fluid