Back in the 1980s, when I worked for Digital Equipment Corporation (DEC), the second-largest computer company in the world, a call came in from Delco-Remy Electronics, a division of General Motors at that time. It seemed the hard drive on a refrigerator-sized system tucked in a room on the third floor of an 1800’s building in Anderson, Indiana had crashed, and this took down the system to order raw material and track shipments.
Just-in-time manufacturing had just become popular, and all the details were inside this system. Without it, nobody knew how much to order, how much to build, or to whom to ship. Delco-Remy sent a work shift home and shut down.
Hardware failures happen all the time. Why not just repair the hardware, restore from backups, and get back up and running again? The answer to that question is what makes this story good.
General Motors had recently bought EDS and outsourced operations to them. The EDS people thought the Delco-Remy programmers did backups. The programmers thought EDS did backups. When the two groups finally talked to each other, they both realized nobody did backups. Ever. There were no backups. Not even any paper records, sitting in a vault somewhere. This system had the single, one and only copy of all the data and software Delco-Remy Electronics needed to operate, and that hard drive failure destroyed it.
With Delco-Remy down, General Motors manufacturing plants around the world would also stop, because cars need ignition systems and other electronics. No electronics, no cars, and a significant chunk of the global economy goes silent. And with just-in-time manufacturing, nobody had any cushion for onsite raw material.
The General Motors CEO called DEC’s CEO, and hierarchies of vice-presidents told somebody below them to make this all better. The buck rolled down to me.
I flew onsite without even a change of underwear. I had a plan to recover, but by the time I arrived, their IT team was already on it. Nobody on that team slept for about three days, while I twiddled my thumbs, tried to stay out their way, and reported progress to managers.
I learned two lessons from that experience. First, the world depends on armies of overworked and under appreciated IT people to keep the systems we take for granted running. Second, if you need more life out of your underwear, turn it inside-out. And if you still need more life after that, wash it in the sink and set it on the hotel AC unit to fast-dry.
Do me a favor. If you enjoyed this story, share this blog post everywhere and tell everyone you know about my novels. Virus Bomb and Bullseye Breach are fiction. This story really happened. I have plenty more.
Who says IT is boring?