This competition submission is a game called Celerity: Sensory Overload. It combines the classic Tunnel Game genre with sensor control and most notably a novel take on 3D.
The main inspirations for this game are:
How could I resist?
Time was tight as I only started after the details of the competition went up. I invited a couple of talented friends to help me out; @Dave_Panic (3D trickery guru) and @BreadPuncher (game theorist & graphic designer).
I was thinking about doing something along the lines of gyro-based steering and/or perspective and Dave suggested doing the head tracking effect with the webcam. Marrying this concept with a classic tunnel game we had the foundation of an innovative, sensor-tastic game. You know how gamers subconsciously tend to lean and move their head around when playing some games even though it makes no difference? Now it will!
VR Head Tracking With Just A Webcam
In Johnny Lee's video, above, he is able to detect the position of the user's head in 3D space through infra red (IR) LEDs and an IR camera.
The effect is fantastic but requires a special IR sensor and also that the user wears an IR-emitting device. I needed to create this effect using only sensors on an Intel Ultrabook, and no other equipment. Thankfully the Ultrabook has a webcam, so we can achieve this in a round-about way.
The application processes each frame from the webcam and run it through a Computer Vision (CV) image-processing library, EMGUCV. This returns a rectangle within a box, representing the relative position of the face within the view of the camera. As the user moves their face, the rectangle will move around, giving me an X/Y offset of the user's face. The X & Y of the 3D world's view can be skewed in relation to the user's own physical position. Note that we need to flip the image horizontally as the webcam is looking in the opposite direction to us.
This effect can be taken even further, as the rectangle representing the user's face inherently has a size. This gets larger as they move towards the webcam, so we can also determine the relative Z position, too.
It's helpful if the user starts with their head roughly central, which we assist with as part of the intro menu UX. There the user can see if they're vaguely "calibrated" before starting.
Designing The UI
I wanted to make use of Metro design principles in the user interface, as although this is a desktop game and not a Metro app, I value consistency and feel the UI will be more intuitive if it shares conventions with the operating system, Windows 8.
Here are the layout mocks for the UI. For the final thing, imagine the animated tunnel effect behind the UI, only faded out somewhat.
Building The App
I wanted to do the app as an XNA app, as it's perfectly fast enough and nice and easy to code for. I don't know C++ yet and there's not enough time to both learn it and complete the game. Unfortunately XNA is an awkward combination with Windows 8, as it's not supported in Visual Studio 2012, at least at the time of writing. My thanks goes to Ibrahim's article which helped me initially create the project. Steps 1 & 2 are essentially from there.
Step 1: Project Pains
Step 1 was to make a project in VS2012 running on the XNA Framework. This was achieved by initially creating the application as a class library, importing all the XNA DLLs, and then converting to an x86 windows application. Manually create the following:
The launcher class:
static class Program
static void Main(string args)
using (var game = new CelerityGame())
The game class:
public partial class CelerityGame : Game
graphics = new GraphicsDeviceManager(this);
Content.RootDirectory = "Content";
protected override void Initialize()
protected override void LoadContent()
spriteBatch = new SpriteBatch(GraphicsDevice);
protected override void Update(GameTime gameTime)
Step 2: Nearly Content
Although there's no proper "Content" project you can still use the
ContentManager class directly to load resources. For example:
Texture2D t = Texture2D.FromStream(graphics, TitleContainer.OpenStream(@"Content\example.png"))
Step 3: SpriteFont Workaround
Fonts are a bit more tricky. The issue is that whilst you have the class, you don't have
SpriteFonts as project items in VS2012 so it requires a workaround. My solution is to simply create a Windows app in VS2010, insert a
SpriteFont there, build the project, and then copy the .XNB file from the bin directory over as a resource.
It can then be loaded like this:
SpriteFont f = contentManager.Load<SpriteFont>(@"Fonts\SegoeUILight56");
Step 4: The Sound of Music
Audio content files are similar to SpriteFonts. I used VS2010 to import an .MP3 content file. When it builds it both converts it to .WMA and produces an .XNB file. Copy both these files from the bin directory as before, and load like this:
Song s = contentManager.Load<Song>(@"Audio\mySongName");
Play your song as normal:
Step 5: Webcam Woes (and EMGUCV)
In this project I'm using EMGUCV version 2.4.0. I was using 2.4.2. but was able to get a much smaller DLL profile with 2.4.0. whilst still keeping everything I needed. It was a cinch to perform a basic capture from a webcam thanks to their simple tutorials, however when I attempted something more ambitious I ran into two related problems.
- EMGUCV gives me a
System.Drawing.Bitmap rather than a
- The performance became disastrous
I found a number of examples of converters online and tried several but still had the performance issues. I suspected it was an issue with running it on a single thread, but when searching the net I found many people warning against using threading in XNA. Eventually I ignored their advice and tried a combination of this method of image conversion and my first ever use of
Parallel.Invoke(). All of a sudden worked a treat without me ever having to quite get to the bottom of it! Don't you just love it when that happens?
Here's my naïve but incredibly effective solution:
Parallel.Invoke(() => QueryCamera(elapsedMilliseconds));
Simply by calling my existing method like this instead of directly from my Update method it was insta-fixed. Sweet!
Yes, okay I ought to read what that's actually doing but first I have a game to make.
One last note on the head tracking; over time the appearance of the resultant rectangle is a bit jittery z-axis wise. This is because it often catches two rectangles per face, but sometimes just either one. This makes for frequent changes in size. I fixed this by taking a recent sample history and taking a rolling average to smooth it out.
Step 6: Raster To Go Faster
In the hope of building a responsive UI quickly I was keen to try a library Sasha Barber mentioned, XNAML. In theory this would allow me to build the UI layer in WPF, with its masterful handling of whilst keeping the tunnel rendering layer in XNA.
With a little fiddling I got the library working, a simple test game canvas beneath a button on top. Unfortunately XNA layer was running at a fixed 96dpi whereas the WPF layer adjusted to my operating system setting, 120. So close but so far.
This means I'm building the UI purely in bitmaps. Given the extremities of possible screen resolutions for a Windows 8 app (1024 wide -> 2560+ wide) making it responsive and still look good is quite a lot of work. Unfortunately this involves lots of rasterising vector assets, calculating coordinates, and of course trial & error.
If anyone knows a way of getting XNA & WPF to play nicely, full screen and for different resolutions etc. then please do let me know.
Step 7: Dark At The End Of The Tunnel
Work on the tunnel has started although currently there are a few problems with the indices. Whilst you can see the tunnel, it's not terribly hollow at the moment!
Step 8: Gyromancy (Steering)
To be continued.
Step 9: Collision Course
To be continued.
Step 10: Spit & Polish
To be continued.
Step 11: Release
To be continued.
Intended Sensor-Specific Features
I'm deliberately keeping the app as straight-forward as possible with the initial release to ensure it gets done. I'd prefer to do something focused well, than have lots of features rushed in. Here is the basic "V1" sensor feature set:
- Webcam-based VR Head Tracking
- Gyro-based Steering
"If I have time" Potential Features
Here are some more features I'll love to stick in either according to time once the basic app is done or possibly as a later update:
- Multi-touch thumb weapon controls - why dodge an obstacle when you can blast it, right? This will be two areas of the screen which independently respond to thumb taps. Maybe even different gestures for different weapons, depending how viable a thumb gesture is when twirling an Ultrabook.
- Subtle contrast adjustment according to the ambient light sensor. I want to keep the look of the game similar, but the greys on white can become full black on white, for example.
- Shake-to-escape emergency random teleport feature, a bit like "Hyperspace" in Asteroids-style games.
- Alternate controls, e.g. gamepad, keyboard, in case peoples' arms get tired swinging it around (this thing is light, right?)
- A number of additional gameplay mechanics (e.g. power-ups, alt game-modes etc.)
- Some proper sound design, but this might require bringing someone else in who has experience in this area. Volunteers welcome!
- Original music score (I'd love to do this myself, but I'm busy coding at the moment)
There's still plenty to do, but it's taking shape very quickly and looks to be on schedule. The head tracking is working great. I will post screenshots and more juicy source in due course. Now to do the tunnel...
My exam is out the way now, so it's full speed ahead.
Feel free to bookmark this article or follow my Twitter feed as I'll keep updating them with the progress.
This app is intended as a submission in the Desktop Games category.
- V1 First Draft
- V2 Minor corrections & progress updates