Some time back there was a major disaster at The State’s Technology Services Division. Details were sketchy but for two full days, employees of The State were unable to logon to their computers or access email, and that this caused business within The State to grind to a halt. This started when employees of The State came in to work following a three day weekend, they found their workstations overloaded with “cannot logon” and “Exchange communication” error messages. The server room was blistering at 109 degrees Fahrenheit and was lying with dead and dying servers. The initial guesses for the disaster was the failure of primary, secondary and tertiary air conditioners all at once. But after cycling the power the A/C’s were working fine. The employees worked day day and night to order new equipment, build new servers and get everything on track. After hundreds of overtime hours and 200 thousand dollars worth equipment they managed to bring everything back online. When the Exchange servers were finally restored, the following email finally made its way to everyone’s inbox, conveniently answering the “why”
As for the employee who sent it, he decided to take an early retirement.
Source
45 responses to “What happens when you turn the AC off in the server room?”
atleast his intentions were good
In this age and time, in a critical state IT facility, someone could just “get a card from a desk” for entry to the servers room, something is really really wrong … someone other than this stupid “early retireer” should have their heads rolling …
Totally bogus story…but still humorous. 😉
This is so totally BOGUS. What state?
not even bogus. i’ve witnessed a couple big companies with keycards to the server room easily accessible from a desk and AC plugged right into the wall.
I wish people would making up these “omg” stories and posting them in hopes of seeing it on slashdot or something. Go write a novel or something that doesn’t make me feel like I wasted 30 seconds of my life reading your invent-a-drama.
All eco-maniacs should be shoot on sight !
Everywhere I’ve worked has stickers over the switch/control panel of critical a/c saying DO NOT SWITCH OFF or DO NOT TOUCH. Every sys admin knows the risk of a/c going down, so I suspect the story is bogus too…. sorry! fun though 🙂
I just want to say that there is an old lesson to this story: if it’s not yours don’t touch it. This guy learned it. You should too.
Thats toll fake, u r tellin that there were no beckup A/C ??? what will happen if A/C breaks down by itself?
This hit live journal a month or so ago.. Don’t know about the “state” bit, but the author claims the person who did the deed was female for a start, that they had retrieved keys from the facilities managers desk in order to enter the room in the first place, and that they were actually in the middle of major upgrades, and so the monitoring system that normally would have alerted them to the failure of the airconditioning units had been switched off due to the fact that people were going in and out of the room so frequently.
When is “Sometime back?”
Even “sometime back” large installations had set up a warning system (ie paging system) to indicate temperature problems…..
“sometime back” meant mainframes with dumb terminals and NOT personal computers.
I give this a stamp of BOGUS.
I worked for a company in Adelaide Australia
the air con died
The CEO didnt want to spend money on the room he hates IT
His answer was to remove the windows and door of the server room
took 3 weeks to get air con replaced
I quit not long after
Cute (but fake) story. Most modern servers would shut themselves down if they got too hot. You might have to deal with scrambled data on your RAID if the servers shut down expectantly, but I doubt you’d have to spend $200,000 on new equipment.
lol it might be bogus but still funny, Cameron that sounds like the place i work at (it’s not as i’m in the uk) but if they had there way they would have us in there puffing at the servers to cool them down!!
Paul
http://www.keepblogging.net
Just because there are signs that say DO NOT TOUCH, doesn’t mean that there is some dummy out there who wouldn’t do this. I’ve had a user come to me and ask me if it was okay that he spray WD40 on his hard drive fan…yeah…they’re out there. Stupid people walk among us.
For everyone who thinks this story is bogus, think really hard – have you ever encountered anyone stupid or ignorant enough to do this? Don’t give people too much credit. Morons walk among us.
I have access to all computer nodes ,servers and A/C units in my building and i do not need any kind of pass card or whatever. After im in the building thats it. Im in. Not that i would do anything to lose my job but it is very possible for this to happen. Also, our systems have alarms that notify the boss if anything is arye in the CPU room. I would figure this is common practice.
This is so fake. Why do people fall for this crap?
I can see this being totally legit. At my office we have several employees who have keys to get into my office which houses the keys to the server room. They could easily come in after hours, walk in my office, grab the key and go turn the a/c off in the server room.
The one bit that I’m finding hard to believe however is the servers dying in a 109* room. The a/c in my server room failed and temps got up to 120* which pissed off all the computers but they still worked fine.
Some people are completely stupid. Someone thinking “hmm these computers don’t need to be cold all the time” is totally possible.
I wouldn’t usually weigh in on something like this but I would agree with the fake nature some people cited. You do not have a server room with critical equipment without a monitoring system monitoring the temperature among other things. The second the temperature hit a critical level peoples phones would have been going off and there may have been minutes-hours of “hot time” depending on their response scenario, but not 200’000 dollars worth of downed servers.
If someone turned off the AC in my server room there would be a blood trail. No e-mail, just blood.
I can believe this perfectly. I worked at a place that put a E2900, along with a bunch of other sun servers in a room that did not have sufficient cooling. The E2900 then committed suicide after it over heated. It also had further hardware issues for weeks to come after it had been installed into the real data center.
I’m guessing 9 of 10 of the those who answer here doesn’t know shit about IT companies. Yes, key card can be found on IT people’s desks and, no, very few server rooms with 3 A/C has any alarm going off if the heat gets up.
But the story is probably a fairytale.
Nobody is asking the important question of this bogus story…
WTF were they doing with MSFT Exchange Server in the first place? Must be a government entity to be that stupid. Thank you for wasting our tax dollars…
I’ve seen this happen, while I was in the office, in fact. I had just started as a NOC tech in a fairly large colo facility. The HVAC software was buggy, so I was told to ignore _ALL_ HVAC compresser failures (4 chillers w/ 2 compressors each).
All 8 compressors failed. I was told to ignore it, only to walk into the COLO about 45 min later to 90+ degree heatwaves. Turns out that that only AC1 was at issue. My boss was promptly fired, and AC was back up 5 min later.
Some people’s kids…
Silly little Americans… what adventures will they get up to next?!
I don’t doubt that this is totally possible. I’ve seen some seriously stupid people out there. But I don’t think this story is real. The email image is too convient to be real. The users name neatly replaced with dashes. The to field simply “IT Department” with no extra tags.
I give it a bogus/funny/possible.
Our comms room overheated last year – two of the servers were permanently knackered as a result. This sort of stuff does happen. AC failed as a result of running at maximum for weeks (they’d refused to upgrade the AC despite being warned it was needed due to an increase in the number of servers). Chronic underinvestment in IT and the supporting infrastructure is all too common even. This is particularly troublesome when the CEO and/or senior management are more interested in short term financial savings to take into account the total cost of systems failure.
F A K E
therefore, not funny.
I can somewhat believe this story. This scenario is why ONLY the IT department has keys to our server room at my campus. Because the average person has no idea what they are doing if they screw with something in our server room.
It’s a believable story – if it was 1985. If it was real, it would give a real location and not “The State”.
That is why my air handler in my server farm has a numerical password on it. If you don’t provide the correct password it won’t do squat for you
did they get fired?
Technology is great…. when it works! If it wasn’t setup correctly, it could fail.
I too have worked in a company where the server room got too hot and fried some hardware. Our bells and whistles didn’t go off as they were supposed to alert us that the server room was too hot. As I said, technology is great…. when it works…
When somebody says, “working for the state”, that’s a generic term, which means “working for the government”. To all of the early posters, this was probably fake, but you guys make yourselves sound really dumb by saying stuff like “what state?!”
Servers dying at 109 degrees? My PC keeps running when it’s 170 degrees hot inside :p Well OK, the power supply burned yesterday, but it worked well for over 2 years.
Craig’s List just came back up, after a Looong weekend(and this is just the initial gossip!) some kook got in and started slaughtering power equipment with an axe!
Two or three days of outages for some really big sites!
if the room was 109 degrees, then the temperature inside the equipment was far higher… Just because my computer has a load temp of 116 degrees, doesn’t mean that the room it’s contained in is that temperature. This story is probably fake, but a nice diversion.
yeah it’s bogus but that hardly takes the edge off because this kind of stuff does happen.
+1 for creative and
cross post to rec.humor.funny
Why would the AC controls be in a server room? How would this guy have any idea how to turn off AC systems in a commercial building? I work in IT and I wouldn’t know where to begin trying to shut down the AC to our computer room.
Not bogus or impossible at all. The biggest cock-ups are usually caused by intelligent and well educated people. The school district where I work ordered that all faculty refrigerators were to be unplugged before the summer break. Great suggestion for saving energy. But they forgot to tell the faculty to REMOVE THE FOOD before unplugging. (Teachers are not encouraged to “color outside the lines”.)
I have had our Director of Physical Planning tell me “Your server has a fan, why does it need an AC?”
This is when you make a librarian head of a Technical Department.
Story may be bogus but there is more truth in it than untruth. 🙂
I love listening to all the people claim how this is impossible. It clearly is not.
Ive worked in all kinds of offices and seen this kind of neglect happen before. People dont take securing servers seriously. Ive seen rack mounted servers sitting on the floor on top of telephone books. Ive seen AC systems busted and replaced with fans from 1970. Ive seen a manager yank the plug on a server because he was having a conversation with a contracter and it was “too noisy”. Depratments simply dont want to spend money on IT. They think HVAC in a server room is some kind of luxury.
This crap DOES happen.
You know, I’ve seen this happen 20 times in my office. We have a small server (one server with 5 raid drives)…no, not a big operation. It is in a closet with a fan which sucks air in from the hallway which is always 60-62 degrees (bad duct work in the building causing unequal cooling). If someone turns off the closet fan, which is about twice a month, the server closet/cage get up to 100 degrees (estimating). Would hope that a server room would have temp controls and better security for a State office…but…it sure sounds familiar…if not simply possible.
True story or not, I liked it.
Besides, who cares if it is real or not?