|
Another up vote for Runbox here. 8 years with them. Multiple domains. Can only recall 2 times they were down due to DDOS attacks. A very green server farm!
|
|
|
|
|
#Worldle #615 5/6 (100%)
🟩🟩🟩🟩⬜⬇️
🟩🟩🟩🟩⬜⬆️
🟩🟩🟩🟩⬜↘️
🟩🟩🟩🟩🟨↘️
🟩🟩🟩🟩🟩🎉
https://worldle.teuteuf.fr
"A little time, a little trouble, your better day"
Badfinger
|
|
|
|
|
At around 2AM this morning things started to go a little pear-shaped with our servers. Our tech guys were looking into it trying to work out what was happening, but it became a all-hands exercise in trying to work out what broke what.
We had updated CodeProject's code. So we redeployed, cleaned and deployed, rolled back and cleaned and deployed. It wasn't our code.
Our requests per second were a little crazy. As in 1000x what we normally would get, but nothing that set off the DoS alarms. Things were adjusted to ease the load but the load remained uneased. Finally, with zero load we still had the site pinned. It turns out the firewall needs replacing. Firewall fixed, load reduced, site back up. Mostly.
There were also a series of Windows patches that were installed as part of routine maintenance. These were to do with HTTP security, so they naturally got some attention. Uninstalling Windows patches can be painful, but once one of the patches was removed everything popped back up. However, that was only on 1 of the servers, so that patch, since it's a security update, will be reinstalled, and if it causes issues the entire server will be binned and a new one rolled in.
So: fun and games at CodeProject central and I apologise for being down for so long. This was a bit of a trifecta, but we're now into mopping up and analysis stage so ![Beer | [beer]](https://www.codeproject.com/script/Forums/Images/beer.gif) all round.
cheers
Chris Maunder
|
|
|
|
|
Well done Chris
In a closed society where everybody's guilty, the only crime is getting caught. In a world of thieves, the only final sin is stupidity. - Hunter S Thompson - RIP
|
|
|
|
|
It was very educational and I see how addicted I (we) are to CP.
Anyway, great you are back and I (we) don't need to look for another dealer 
|
|
|
|
|
Must... not... tell... war... story...
It involved a customer replacing the 1Gb hub we have in our equipment with a 100Mb hub they had laying around. They had the audacity to (1) not tell us what they had done, (2) lie when we asked them point blank what they had changed, and (3) tried to hide the evidence when a [very sleepy after a 10 hour drive] field service dude showed up.
Irrelevant side note: the outage must be Jeremy's fault, since he raised the subject of uptime a few threads down.
Software Zen: delete this;
|
|
|
|
|
Been there done that 
A customer called and told me things didn't work.
I asked them if anything at all had happened that could've caused this?
Nope, nothing had happend.
Really, nothing you can think of?
Nothing.
This customer never called so I wasn't too familiar with their software and setup.
After a few hours of looking into it I found nothing.
When I called them back the IT manager "suddenly remembered" they'd restarted the server "but that can't be it, right?".
We fixed the problem five minutes later 
|
|
|
|
|
Gary R. Wheeler wrote: Irrelevant side note: the outage must be Jeremy's fault, since he raised the subject of uptime a few threads down.
I thought the same 
M.D.V. 
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
|
10% of capacity? But what could do wrong?
cheers
Chris Maunder
|
|
|
|
|
Especially on a machine printing both sides of the paper at 17 feet per second, full color, front and back.
Software Zen: delete this;
|
|
|
|
|
Chris Maunder wrote: if it causes issues the entire server will be binned and a new one rolled in
So you know Chris, ever think about hosting CP on Debian? 
Jeremy Falcon
|
|
|
|
|
We have this awful albatross of a webforms project that ruins the fun. .NET Core, Linux, PostgreSQL or MariaDB and I'd be so happy and our costs would be dramatically lower. Sigh.
cheers
Chris Maunder
|
|
|
|
|
|
The hamsters probably needed a breather anyway.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
Sounds like my morning execpt on a much much mush smaller scale. And it was the cable company coming and going.
|
|
|
|
|
Still running on AWS? This happens to us sometimes too.
|
|
|
|
|
Things go sideways sometimes. A bit of adrenaline to get the blood pumping.
Sounds like you’ve got it figured out.
Cheers, and thanks! ![Java | [Coffee]](https://www.codeproject.com/script/Forums/Images/coffee.gif)
Time is the differentiation of eternity devised by man to measure the passage of human events.
- Manly P. Hall
Mark
Just another cog in the wheel
|
|
|
|
|
Maybe some day you will upgrade your servers to Linux...
|
|
|
|
|
Trading known problems with brand new unknown ones. Hmmmm, tough choice...
|
|
|
|
|
Obviously it was tongue in cheek, but seriously, Windows updates too often break stuff, and Murphy's law ensures that they break said stuff at the most inconvenient times.
|
|
|
|
|
obeobe wrote: Obviously it was tongue in cheek, but seriously,
The problem is that some people make that sort of suggestion while being absolutely serious.
obeobe wrote: Windows updates too often break stuff
Can't argue there...
obeobe wrote: they break said stuff at the most inconvenient times.
Is there ever a convenient time during which updates should be okay to break stuff...?
|
|
|
|
|
dandy72 wrote: Is there ever a convenient time during which updates should be okay to break stuff...?
When I am on vacation and unreachable?
|
|
|
|
|
...so less qualified people rush their own fix that you then inherit when you come back...?
|
|
|
|
|
well yes...but that of course leads to job security...since of course by then the higher ups learn that only I can fix things!
|
|
|
|
|
so there was some windows patch and the firewall config raising the cpu weird.......
Caveat Emptor.
"Progress doesn't come from early risers – progress is made by lazy men looking for easier ways to do things." Lazarus Long
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
