Click here to Skip to main content
Click here to Skip to main content

The New CodeProject Web Farm

, 31 Mar 2004
Rate this:
Please Sign up or sign in to vote.
Google runs on 10,000 Linux servers so we figured we'd go one better: 1000 clustered iPaqs! Read about the trials and tribulations of our move to a new server farm.

Introduction

With the server cluster in the office creaking alarmingly, and our fibre connection to the outside world almost warm to the touch we decided that we had to bite the bullet and move the entire mess into a true hosting centre that would provide 24/7 support, backup power, and redundant internet connections.

   
  Thorough Planning makes a difference
 

The first step in the move was to upgrade the hardware in-house and test to ensure the new servers were up to the task. We priced the main players and soon realised that only by selling our kidneys could we ever afford the required hardware. Our other option was to purchase rack mounted cases and build the machines ourselves using quality parts at affordable prices. Sure - it meant some late nights and a bit of driving around, but we're all certified geeks and can build a box blindfold in under 6 minutes. A quick call to our local supplier - the one with the unmarked entrance at the back of a block of units in a distant industrial area - and the gear was on it's way.

That's when the trouble began.

We only received a single case and enough parts to make up 2 boxes if we did without some of the fancy stuff such as monitor support and Ethernet. Not good. We decided to move ahead and build what we had to at least ensure that the rest of the shipment, should it decide to turn up, would be suitable. Leading the charge was Peter, our resident hardware / network / sysadmin / all-round legend, who had the system mostly built in no time but then popped out for an extended lunch and decided, for some extremely bizarre reason, to ask Bianca to finish up. Bianca handles all advertising accounts and marketing, plus pretty much anything else thrown her way. She's amazing. She isn't, however, experienced in building servers. Hands up who can see what's wrong with this picture:

A hint: those are wire cutters she's holding.

So it was decided that those who have the most experience in building boxes should be the ones to, well, build the boxes. There were words. There was some pushing and shoving. There was a disaster.

This can't be good "Yeah, Hi. Ummm...."

A New Beginning

We tried to get replacement hardware but in the end it was decided that the chances of us getting the servers we originally envisioned in any manageable timeline, within budget, was simply not possible. What then ensued was the biggest call-out we as a group had ever done in our four years or building CodeProject and we came up with Gold.

  
Dave in front of some of the boxed iPaqs.
They were piled up everywhere.
 

  

A friend of a friend of a... (you know how these things work) had a massive stock of superceded iPaqs. One thousand of the little tikes, to be exact. They'd been purchased by one of the larger Real Estate companies for use by their employees to provide customers with up-to-date information on homes while on the road. The problem is that none of the agents could use the devices, the application they were meant to use was no longer supported (or even worked half the time) and they'd simply given up and shipped them back. They were paid for, no one wanted them, so we ended up with the lot.

And what can an iPaq do by itself? Not a lot. But a thousand 64Mb 206MHz series 3700 iPaqs could really start to make a bit of noise.

  

CodeProject is alive!

We initially investigated whether or not it was possible for an iPaq to handle even part of the site's load and quickly realised that the units could handle everything that we required a desktop system to handle. We initially considered a reasonably radical move of installing Linux on the iPaq's and using mySQL as the backend, but instead chose to use  vxWeb as a base for a webserver, and use any one of a number of plugin VBScript interpreters to quickly put together a lean and mean CE based ASP web server. On the backend it was decided that SQL Server CE was the obvious choice since it meant an easy port.

A single iPaq was not going to be of much use. We needed to cluster! Initially we figured this was probably not going to be possible but a number of solutions ranging from traditional dual-NIC clustering to a more ambitious IR based cluster were found. We had a ton of USB adapter cables so after a bit of slicing and splicing, lots of fiddling about and copious swearing we were on our way.

  
- This is never gonna work
- Shuddup and get in the truck
- OK. Maybe you're right.

Initial development time was around 3 weeks, physical build time and testing was around 2 months and the entire move from the office to the hosting facility took around 8 hours. I was able to call on the help of Dundas Software who had tons of previous experience in CE based development, and we were also able to port the .NET parts of the site to the .NET Compact Framework with only mild swearing.

The Facility

CodeProject's new home is the Telus facility on Laird in Toronto. It's a monster. 8 foot thick walls, ram-raid protection, underground fuel tanks with enough fuel to power the entire facility, fully stocked with all lights on for 3 weeks. Biometric security, pressure sensitive floors, multiple connections to major internet backbones and some seriously grumpy security guards.


CodeProject's new home

 
Smile!



Yes Dave - they mean you.

Unfortunately we were banned from taking any pictures inside the hosting facility itself (other than the one of Dave and I in the loading dock looking dubiously at the boxes). The one pic of the rows of racks above was taken in the one place there were no security cameras, and at the expense of having Bianca sit through a 20 minute detailed explanation of the fire retardant system by our helpful, but hard to shake, escort.

The units themselves are mounted in custom sliding trays that are attached to the racks using traditional sliding rails. We use dual firewalls capable of handling up to 100MBit each, with automatic failover and restart. All switches are 100MBit for the external network, 1Gbit for the internal traffic. Each unit has it's own screen and can easily be lifted out of it's position in the tray, making a supplementary Keyboard/Video/Mouse unit unnecessary.

Performance

The increase in server capacity and network bandwidth saw an almost immediate increase in throughput. We're now able to serve more pages simultaneously with faster load times than ever before.

Scalability is a non-issue as units can either be added, or old units removed and replaced in situ without affecting the rest of the cluster. We've squeezed the thousand units into a single standard rack and will be looking to lease a second rack in the near future to expand again. After the initial teething problems it's been a great experience.

Issues

Obviously something this complex isn't all sweetness and light. Some of you will remember issues with had when we had 6 servers. Server number 2, you will remember, was always playing up and was, in the end, declared cursed and unsalvageable. Our current setup essentially multiplies these problems. Currently servers 45, 234, 294, 536 and 785 are all showing signs of early senility, and servers 239, 455 and 901 have been used in baseball batting practice. There is a limit to our patience.

Heat is another issue. We've had to install several large fans inside the rack to ensure that the batteries in the units, while charging, don't overheat. This has also allowed us to solve the other problem we had, namely dusty LCD screens. Coming in each day to clean a thousand dusty, fingerprint'd iPaq screens is simply no fun, but the constant airflow at least minimises the dust buildup.

Cables coming loose has been a surprising entrant in the 'Most Annoying Thing' competition of '04. A close runner up has been the dropping-the-stylus-into-the-lower-rack-cavity. We ended up using the old bankers trick of having a single stylus inside the rack cabinet attached to a very long string.

Conclusion

Overall it's been a wonderful experience. We have more room in the office, you get a faster, more reliable website and we get to say "Happy April Fools Day".

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

Chris Maunder
Founder CodeProject
Canada Canada
Chris is the Co-founder, Administrator, Architect, Chief Editor and Shameless Hack who wrote and runs The Code Project. He's been programming since 1988 while pretending to be, in various guises, an astrophysicist, mathematician, physicist, hydrologist, geomorphologist, defence intelligence researcher and then, when all that got a bit rough on the nerves, a web developer. He is a Microsoft Visual C++ MVP both globally and for Canada locally.
 
His programming experience includes C/C++, C#, SQL, MFC, ASP, ASP.NET, and far, far too much FORTRAN. He has worked on PocketPCs, AIX mainframes, Sun workstations, and a CRAY YMP C90 behemoth but finds notebooks take up less desk space.
 
He dodges, he weaves, and he never gets enough sleep. He is kind to small animals.
 
Chris was born and bred in Australia but splits his time between Toronto and Melbourne, depending on the weather. For relaxation he is into road cycling, snowboarding, rock climbing, and storm chasing.
Follow on   Twitter   Google+   LinkedIn

Comments and Discussions

 
Generalbut but but Pinmemberbryce7-Apr-10 12:42 
GeneralClustered CodeProject server Pinmembercharfeddine_ahmed23-Jan-08 4:46 
GeneralChris.. PinmemberRenjith Ramachandran28-May-07 18:49 
GeneralMulder PinmemberDewey15-Dec-06 22:02 
Generalgreat usage for the LCDs Pinmembercrea21-Apr-04 20:53 
QuestionWhat the... PineditorHeath Stewart14-Apr-04 10:43 
AnswerRe: What the... Pinmemberpeterchen29-Nov-04 19:58 
Generalbah (part 2) PinsussAnonymous9-Apr-04 4:56 
GeneralRe: bah (part 2) Pinsussanonumous29-Mar-05 21:54 
Generalbah PinsussAnonymous9-Apr-04 4:43 
GeneralThanks :-D PinmemberBrian Delahunty2-Apr-04 4:31 
GeneralApril Fool's got me anyway! PinmemberHarold Bamford1-Apr-04 6:26 
GeneralDave looks just like that singer on American Idol... PinmemberJohn Cardinal1-Apr-04 5:11 
GeneralRe: Dave looks just like that singer on American Idol... PinadminDavid Cunningham2-Apr-04 6:27 
GeneralRe: Dave looks just like that singer on American Idol... Pinmembermelwyn19-Apr-04 4:00 
Questiongreat work, but why all the LCDs? PinmemberENewton1-Apr-04 5:09 
AnswerRe: great work, but why all the LCDs? PinsussChris Tacle1-Apr-04 7:42 
AnswerRe: great work, but why all the LCDs? PinsussChris Tacke1-Apr-04 7:42 
AnswerRe: great work, but why all the LCDs? PinstaffBianca Wylie1-Apr-04 7:45 
GeneralRe: great work, but why all the LCDs? PinmemberENewton1-Apr-04 17:40 
GeneralRe: great work, but why all the LCDs? PinmemberSpiff Dog7-Apr-04 18:25 
GeneralYou had me at '1000 clustered iPaqs' PinmemberKant1-Apr-04 4:35 
GeneralRe: You had me at '1000 clustered iPaqs' Pinmembernetlurch1-Apr-04 8:55 
GeneralRe: You had me at '1000 clustered iPaqs' PinmemberRoger Wright1-Apr-04 16:11 
GeneralReminds me... PinmemberTim Deveaux1-Apr-04 3:26 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web01 | 2.8.140721.1 | Last Updated 1 Apr 2004
Article Copyright 2004 by Chris Maunder
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid