An annual electrical maintenance resulted in a total shutdown of power for about eight hours on a Sunday. Unknown to IT, one of the departments actually had a staffer coming back every Sunday in order to do an online electronic filing of certain shipping documents.
Never mind that nobody in that department saw the notices of the impending power shutdown on every elevator door over the entire week, or noticed the company-wide e-mail blast. The lone staffer arrived as usual that fateful afternoon to a darkened office. Obviously, he failed to do the requisite filing, resulting in a compound fine being imposed on the company.
The company’s General Manager was upset and asked why the uninterrupted power supply (UPS) investment still resulted in non-functional servers. When it was pointed out that the power in the current UPS could only last the one dozen servers for between 15 to 20 minutes, the order was given (over my objections) to purchase sufficient UPS capacity to last through a “full day” of power outages.
Preliminary estimates with engineers from APC indicated that we needed two 42-U racks packed with UPS and extender batteries to be able to meet the desired runtime. The cost? $20,000.
The idea was given up only when I realized that a fully running server room without a powered air-con or ventilation is not a very good idea. To spare myself an urgent visit to Toni’s View from the Cubicle blog for tips on getting a new job, I doubled the estimate to accommodate the air-conditioning - at which point we also ran out of space in the server room. Thankfully, the directive was scrapped after that.
If you have to ask: yes, a diesel generator was totally out of the question since the server room was located squat in the middle of an office complex.
Refusal to buy server racks (Penny wise, pound foolish)
Unless you’ve been working in MNCs all your life, you’ve probably encountered this one before: Management refusing to purchase proper server racks.
Now, a certain reluctance to splurge a few multiples of grand on a high-end kit is understandable. But the situation becomes a little intolerable when we’re talking about just a couple of simple bare-bones 42-U racks costing less than a grand each - to house no less than a dozen and a half servers currently scattered all over.
Have you encountered a situation like this before?
In my case, I finally got my way in this scenario. But I wanted to hear from more of you who serve at the “front line,” however. How would you justify the value of proper server racking?
Splurging on the wrong things (Reacting from fear)
Just before I joined this particular company, one of the database servers suffered a serious hard disk error, resulting in a corrupted database. The near line backup was no good because its mediocre hard disk had long ran out of sufficient capacity for even one full backup.
We recovered the data for the most part. Due to the resulting anxiety, management wanted to replace two of the database servers with brand spanking new ones.
I had just joined the company, and the exact instruction given by my boss was: “Just go for the best. I’ll pay.” Now, you must understand that these database servers, though critical, were used by no more than five users each; there are so many cheaper methods to prevent a recurrence of the problem. I confess that I didn’t follow the instructions in the end. I quietly told the vendor to just give me something mid-end, and we ended up spending about $16,000 on two HP servers.
Still, the money spent could have been better used elsewhere- like replacing a couple of production servers that were more than eight years old and for which there is no functional equivalent in terms of hardware.