|
Scary, from a financial risk viewpoint.
- An island in Southeast Asia. The Local Turf Club had contracted with an Australian hardware/software company to install a new computerized betting system to replace its old, aging system so that they could support more than 200 betting terminals. The call came to me, an IT consulting manager, at their Audit firm to test and certify the new system for compliance with the rules and regulations of the Turf Club.
The stakes were quite high. The Turf Club had to make sure that the system performed flawlessly. Any hiccup would provide the excuse to the government to shut down the Turf Club and take over its extremely valuable land holdings close to the city center.
I had to understand all the rules stated in the Rulebook of the Club. For instance, only three horses could deadheat for the Win (first place), Place (second place) and Show (third place) positions. If three horses deadheated for the first position, there was no Place or Show and those betting pools will be combined with the Win betting pool and the moneys distributed among the bettors who bet on those three horses. This meant that one had to consider the possibility that there can be one horse for Win and three for place; one horse for Win, one horse for Place and up to three horses for Show; and any similar combination one can think of.
In addition, chain betting whereby the winnings of a race are bet against a second race without having to place new bets (called Perfecta) and the proceeds bet against a third race (Trifecta) was permitted. The ticketing terminals should issue tickets within seconds.
The system should lock all betting terminals before the race begins and results should be declared and winning tickets paid out within one minute of race results being announced. The same terminals that sold tickets could be used to read any ticket that is presented and pay out the winnings.
This meant that I had to understand the accumulation of moneys in various betting pools, calculating and displaying the changing odds on display terminals and calculating the payout.
The system had to have a second identical computer for backup and should automatically use whichever system was available without the need for human intervention.
When I was called in, the vendor claimed the hardware and software were ready.
With just a couple of terminals, I placed several tens of bets in various combinations on imaginary races and hand calculated the results. They agreed with the numbers the computer was spitting out. So the software could be certified. By turning off a disk drive or two, or one of the two computers, we could determine that fault tolerance was perfect. But the betting terminals themselves were very erratic.
We called the vendor in Australia and impressed upon them the need for 100% reliability in the terminals. They sent a hardware engineer to test and fix the terminals already received by the customer and tightened up the quality control at their factory.
A few weeks later, all the betting windows were used and bets placed on imaginary races by a crowd in a dress rehearsal. It was successful and I had the confidence I could leave on my annual vacation to the US.
When I came back, the Club authorities expressed their satisfaction to me at the successful transition to the new system. There was no possibility of fallback to the old system as those terminals had been ripped out to make space for the new ones.
A couple of years later, I was scanning some computer related news. One item I found interesting was that the transition to a computer system at the Santa Monica Race Course had resulted in failure!
|
|
|
|
|
|
Not me, but a relative of mine: He did his (compulsory) service in the military at a time when computing technology consisted of a slide rule[^] and a book of tables. Having an engineering education, he was given responsibility for calculating the launch angle of the mortars.
Gradually, he realized that if he made a miscalculation, that could lead to several young men returning to their wives, their kids and parents alive. If he did his task properly, they might return home, but in a coffin. Parents would loose their sons, wives their husbands, kids their parents. Even soldiers are humans, with a life. Even enemy soldiers.
He couldn't take that responsibility: Making a mistake saves lives. Doing the job correctly kills humans. Sitting back home, or in a nuclear proof shelter, it is easy to say: But this is war. It is different when you are personally responsible. You could do a deliberate miscalculation, to let people live.
He went to the military administration telling that he refused to take that on himself. He refused further military service. In those days, that was like openly branding yourself as An Enemy of The People. Your relatives and friends might turn their back on you. Yet, he chose that, rather than knowing that his correct calculations would cause people to loose their loved ones. Making himself a murderer, disguised in a military uniform.
Of course we all know that the commandment "Thou shalt not kill" really is a short form of "Thou shalt not kill any of us". Many of us live by the extended version. Maybe you could strangle your neighbor with your bare hands, or exterminate him with a mortar, because of his religion, his ideas about economy, or some moral ideas he might have. That's what usually identify the 'enemy' that must be fought. And killed, if he does not submit to your religious, economic and moral ideas. He couldn't.
Nor could I. But in my days of youth, requesting for a non-military 16 months service was far more accepted. I never had to flee from a personal responsibility comparable to that of this relative of mine.
|
|
|
|
|
I'm just now catching up with a small backlog of posts I wanted to go back to, and stumbled upon this.
trønderen wrote: he realized that if he made a miscalculation, that could lead to several young men returning to their wives, their kids and parents alive. If he did his task properly, they might return home, but in a coffin.
Interesting conundrum, and I understand his position...
But OTOH, it's not like sparing the enemy means lives are saved and that's the end of the story...rather, if you don't kill your enemy, he gets a chance to kill you, your family, and your follow countrymen. Is that a better outcome?
If it's kill or be killed, it's a pretty simple decision in my mind. But ultimately, when you've reached this point, when wars are waged, invariably it's because some "leader" has failed miserably at his job.
I didn't really mean to revive this old thread, but somehow I felt compelled to contribute my 2 cents. 
|
|
|
|
|
� Forogar � wrote: Has anyone else been in a similar situation?
Driving?
Fascinating read - thank you for posting that.
|
|
|
|
|
Financial risk not safety...
What seems like a million years ago I was a newbie mechanical designer / draftsman. Within a few months of starting at a company, I drew up a C-frame for a 170 ton hydraulic press. The C-frames would cost ~$40,000 each and they intended to build 6 in the first run. I had to urgently request that they have a senior engineer check my work before release...
Fast forward 35 years...
I'm still with the same company and my team and I design the control systems for million dollar machines without blinking an eye.
Times change... 
|
|
|
|
|
Developed heat treatment control systems used in the field for pre and post welding. Lots of amps and high heat. Not "dangerous" per se but could ruin expensive projects if the system miscalculated (ramping up / heating; holding; ramping down / cooling).
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
Well I can surely speak for the "scary responsibility" of hanging my wet bath towel up on a hook located behind my current computer then leaving the room in which all this is located, while running a strenuous calculation which mounts all my RAM, ... and being out shopping and suddenly recalling that I have a drying towel on the hook and ... perhaps I better get home before all my RAM is used, the box sets itself on fire, my dry towel now begins to go up in flames and my apartment is set ablaze.
Not really a job, thinking suddenly, but certainly something to panic about while dredging things up from my subconscious during a period of unemployment.
|
|
|
|
|
RAM usage == fire hazard? 
"the debugger doesn't tell me anything because this code compiles just fine" - random QA comment
"Facebook is where you tell lies to your friends. Twitter is where you tell the truth to strangers." - chriselst
"I don't drink any more... then again, I don't drink any less." - Mike Mullikins uncle
|
|
|
|
|
jeron1 wrote: ==
In your stocking feet go stand on a lamp cord, you ... you ... programmer you!
|
|
|
|
|
RedDk wrote: you ... you ... programmer you!
Overly dramatic voice:
How dare you!
"the debugger doesn't tell me anything because this code compiles just fine" - random QA comment
"Facebook is where you tell lies to your friends. Twitter is where you tell the truth to strangers." - chriselst
"I don't drink any more... then again, I don't drink any less." - Mike Mullikins uncle
|
|
|
|
|
Not really scary, I worked on the Smart Motorway System (that has now been shelved thank ![Green Alien | [Alien]](https://codeproject.global.ssl.fastly.net/script/Forums/Images/alien.gif) ) I was not popular for saying infront of the Highways England people who were observing the test "I think the recovery crews will need a power washer, not an amblulance". Went down like a lead balloon, but did get an ammount of support from the Highways England people.
|
|
|
|
|
I work in automation systems so it's a fairly common-place thing for me to have a lot of scary responsibility of many varieties. Probably the most scary had to do a financial responsibility. I was working on a semiconductor processing system that was dealing with the chemical processing of wafers. These systems were very, very time-sensitive. We had several cassettes, each full of fifty wafers in process at a time in the system. One type of system dealt with acids that would etch off oxide that had been deposited on the wafers. Most customers would dilute the acid so the etching took longer but it was rather tolerant to immersion time variances. Other customers insisted on very high acid concentration which etched very fast and were very sensitive to the immersion time. It had to be accurate to the second. If the time was off by more than a second then the wafers over or under etched and the entire cassette was scrapped which resulted in the loss of millions of dollars of product. We spent an immense amount of time making sure that the entire process was exactly accurate.
Imagine my frustration at how poorly the customer trained their operators. I had already been instructed to assume a sixth grade reading level from the operators and at times that seemed optimistic. I remember vividly watching an operator and guiding them during the first shift of production. We had to use a flaky wafer transfer unit that would load the cassettes into the system, as required by the customer, and these things would give errors often. When that happened the operator was supposed to cancel the transfer in progress and try again. Instead, I watched the operator press abort three times in the first shift, causing approximately ten million dollars in scrapped product. I was then instructed to put a double level of "are you sure" confirmations instead of just the one. I then had to tell them not to even use abort except for very special circumstances like power loss when the UPS couldn't keep the robots alive because the batteries were too weak. That was very, very frustrating.
One amusing thing happened when we were installing a system in Erfurt, Thuringia, Germany. We were undergoing our final confirmation testing and the process engineer told us the etch times needed to be accurate to within one second. He emphasized this by saying, "and I mean German seconds, not American seconds." I got a good chuckle from that. When they approved the system he gave us a stopwatch that had been made in the USSR since Thuringia was formerly a part of East Germany.
"They have a consciousness, they have a life, they have a soul! Damn you! Let the rabbits wear glasses! Save our brothers! Can I get an amen?"
|
|
|
|
|
How does a German Second compare to a New York Minute? *raises eyebrow*
Real programmers use butterflies
|
|
|
|
|
Rick York wrote: magine my frustration at how poorly the customer trained their operators. I had already been instructed to assume a sixth grade reading level
I, I kid you not, had to change all text in a version of the software to add recognizable pictograms to any prompt or error because a large part of the working force, QA in a pharmaceutical company, was totally illiterate.
GCS d--(d-) s-/++ a C++++ U+++ P- L+@ E-- W++ N+ o+ K- w+++ O? M-- V? PS+ PE- Y+ PGP t+ 5? X R+++ tv-- b+(+++) DI+++ D++ G e++ h--- r+++ y+++* Weapons extension: ma- k++ F+2 X
|
|
|
|
|
There are no doubts from me. 
"They have a consciousness, they have a life, they have a soul! Damn you! Let the rabbits wear glasses! Save our brothers! Can I get an amen?"
|
|
|
|
|
I added "who did what when logging"; which everyone is aware of and can access (read). It does reduce the times they reach for the kill switch. And voice with the text: WARNING! WARNING! ... (and cross and skull bones here and there).
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
I did that too with logged events but they couldn't be bothered to have separate accounts for each user so everyone used a generic "operator" account.
"They have a consciousness, they have a life, they have a soul! Damn you! Let the rabbits wear glasses! Save our brothers! Can I get an amen?"
|
|
|
|
|
Not on the same scale at all, but at one time (1980s, when I was in my mid-20s) I worked on a credit-card processing system. It was all batch COBOL stuff and ran for many hours each night. Myself and a colleague took turns on overnight support, which involved using a portable dial-up teletype to get a core dump, from which we diagnosed any problem and either corrected faulty input or coded up a workaround. One day my colleague dealt with a problem but inadvertently used a comma, not a full stop, in one correction. I don't know the exact problem, but suffice to say that during the review next day we realised he'd "lost" a little over £12,000. He was mortified and for the next few weeks I was petrified when making code changes at 3am on my own... (Sitting on the floor in my hallway because the modem lead wasn't long enough).
|
|
|
|
|
For this reason alone (just kidding, I love games) I went to videogame programming. No interaction with clients and nobody dies.
But coming of age I've become more stoic about it. Civil and aeronautic engineers have a hell of a lot more lives on their shoulders...and they seem to shrug off the what ifs pretty well.
|
|
|
|
|
� Forogar � wrote: Has anyone else been in a similar situation?
My first bug in the wild wasn't scary, until things went wrong.
Caused some machinery to be out of sync. 14k in damages, I was 19 at the time.
The boss just remarked I won't make the same mistake twice
Always been cautious about the jobs I accept. Never anything medical again either.
Bastard Programmer from Hell 
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
Not I. The closest I've been is to work on a commercial desktop program that keeps companies' escrow accounts balanced. If their escrow accounts can't balance, someone is going to jail. The risk is not even close to the same level as your project, but some people act as if it was.
I tell all the new devs that work on the escrow account code, "If the money isn't right, then you don't have a job - no excuses."
Bond
Keep all things as simple as possible, but no simpler. -said someone, somewhere
|
|
|
|
|
Some years ago I created the software to monitor the jacks under an aircraft as it was being serviced. Basically you could put load x on jacks y to z and unzip the plane around the middle without it falling apart, that kind of stuff. The actual, aircraft is no longer flying but it was that one with the long pointy nose that flew at Mach 2 (no names, no pack drill!).
The whole project was being run by an engineering company so we were basically a sub contractor, the software was just down to me and it communicated with a "black box" that monitored the load cells on the jacks. We used an IBM PS2 computer which had been recently released and had a decent hi res graphics screen. The program had a configuration part where you entered the calibration information for each of the 15 or so load cells. When running it showed the load from each cell dynamically overlaid on an outline of the aircraft. Should a load exceed a preconfigured limit then a siren attached to the console would sound. I also supplied a detailed manual that covered all aspects of configuration and use.
So, the main contractor picked up the whole caboodle and took it off the the airport for delivery to the maintenance hangars, my business partner and I decided to visit too but when we got to the hangar they had not arrived. We went on into town for another appointment and called in again on the way back.
We were told that the main contractor had been but there were problems so thay had taken everything away with them.
Later that day, early evening I had a call from the manufacturers of the load cells. The whole system had been dropped off with them and they were testing it using their massive presses. They found the system was working 100% with significantly higher accuracy than the minimum requested.
Apparently an engineer had been busy jacking up the aircraft during the test. The jacks were manual and he began to have problems, so he fitted a 2ft extension to the jack handle and carried on before realising that something was not right and the actual load must be way more than what was displayed.
The main contractor had taken my manual, ripped off the covers and added their own but did not bother to read it. Each load cell had characteristics that were up to 100% apart so they had the wrong load cells aligned to the wrong calibration parameters thus the cells were reading way off.
So, now you know why said aircraft had a bent droopy nose
|
|
|
|
|
Back in the days when I worked for Orange Mobile Communications UK as an engineer, the department handled a lot of the call routing for the Tetra Network in the north of England and Scotland.
On top of that we where also responsible for the B2B SMS routing systems used by a number of the large Metropolitan Scottish Fire services covering places like Fife, Glasgow and even Aberdeen.
Aberdeenshire was of particular concern, because should an incident happen on one of the North Sea oil/gas rigs, Aberdeen would be the central operations area co-ordinating emergency services and evacuations.
Tetra for those that don't know is the 2 way GSM based digital radio system that the UK police use, or did, I don't know if it's still used today
Basically though, our engineering team where responsible for making sure all those communications and SMS messages got sent to the correct message displays in the cab's of Fire Engines and Ambulances, and that police officers requests for back up and assistance where routed to the correct control rooms.
One small slip up in the routing software or the systems controlling any of the GSM infrastructure could result in a Fire Engine going to the Wrong place, or some one needing help not getting it in time, needless to say we where very quick to investigate even the smallest most innocuous report for any of those systems.
|
|
|
|
|
While I have developed "scary" systems, the scariest was being a victim of faulty programming in my Ford F150.
It seems that Ford had gone to what they call EPAS--Electronically Powered Assisted Steering in my 2012 F150. Essentially, EPAS is drive by wire and has nifty software features such trailer sway control and crown load detection.
In one circumstance, I was driving a country road, hit a significant deflection in the road surface and the truck took over steering and rolled. As far as I can guess, it seems that there the software had an algorithm for "roll-over protection" and it "thought" the truck was rolling. But instead of correcting by simply holding straight, it elected to counter-steer.
I ended up hanging upside down from my seat belt while the truck dialed 911 and got an emergency dispatcher on the line.
Truck was considered totaled because the airbags had deployed and it costs more to "clean"the truck and replace the airbags than the truck was worth.
The irony is that about three weeks after I settled with my insurance company, I got a urgent recall notice from Ford to bring my truck in and have the software updated.
I will not trust drive-by-wire in difficult situations---No software developer or engineer can encode ALL the circumstances that might be encountered.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
