| |
|
Vertical relocation case study:
High
technology equipment movers proves Occam's
razor...simple solutions are best.
On a chilly
spring afternoon IBM's western regional manager invited
us to bid on a most challenging project, "Due to new
earthquake guidelines we need to upgrade our most
critical data center floor, panels and struts...a
shutdown will cost millions".
I was
nonplussed, figuring that a dozen or so racks was no
real challenge. We have often engineered complex
logistical solutions, with secure mirrors and rapid
deployments. But when Scott told me that the center
was comprised of over 400 fully stocked IBM, EMC, Dell,
and HP enclosures he had my full attention. He
explained that these enclosures managed both the
active electrical grid and customer service for a few
million customers in multiple western states. A failure
would most likely result in rolling blackouts, lawsuits,
and very angry customers.
He was
putting the project out for bid, and he had come to us
on the recommendation of a large ISP whose operations we
had relocated a few months ago. We won the bid after
some major internecine battles, which culminated with a
76 person conference call between 75 sharp stakeholders
and yours truly.
We considered
many options. Traditional relocation and virtual
accommodation methods was impractical due to both the
politics and key operational concerns. Jointly we
decided on an innovative vertical solution.
Vertical methods, wherein the
equipment would be lifted off the raised floor panels by
a minimum of 2 feet, and held for 2 hours, was needless
to say controversial. However, we quickly discovered
that the lifting of the equipment was only a small
portion of our challenge. I recalled the sage advice
that "all tasks are fundamentally simple...its the
quality of solutions to the inevitable sharp problems
that occur, that is the truest test of success"!
|
 |
The primary
challenges are in both the logistics of minutia; cable
strain, interface integrity management, and the larger
issues of stabilizing an active enclosure, with drives
spinning, and rack cooling fans whirling.
Once the project got started, it
took 2 months just to figure out the logistics, and to get
the equipment that we needed designed and manufactured.
After a few fits and starts, some of the appliances that we
had developed, proved untenable due to weight distribution
and center of gravity issues, discovered during exhaustive
pre-test simulations.
There were of course no shortage
of ideas from our engineers. Everything from overhead
cranes, to hydraulic platforms were proposed and tested.
We finally decided on a series of procedures that provided
maximum stability, and extraordinary safety. A crew of 6
headed by a senior structural engineer, required from 4-8
hours per enclosure to accomplish the tasks safely.
Each phase of the 6 phase
operation informed an extensive checklist. These checklist
embodied situational and procedural guidance for the crew,
while simultaneously generating documentation for OSHA, the
insurance companies, and the ever nervous stakeholders.
We will skip the minute details
for now, but we can report complete success. We
successfully performed over 400 vertical relocations, over a
6 month period...without a single outage. The utility's
customers never lost power, their bills remained accurate
and timely, and none of the stakeholders jumped, as
threatened, from high office windows.
A carefully executed, ego-less,
yet simple plan is the stuff of success.
|
|
