Stunts, Idiocy, and Hero Hacks 208
snydeq writes "InfoWorld's Paul Venezia serves up six real-world tales of IT stunts and solutions that required a touch of inspired insanity to pull off, proving once again that knowing when to throw out the manual and do something borderline irresponsible is essential to day-to-day IT work. 'It could be server on the brink of shutting down all operations, a hard drive that won't power up vital data, or a disgruntled ex-employee who's hidden vital system passwords on the network. Just when all seems lost, it's time to get creative and don your IT daredevil cap, then fire up the oven, shove the end of a pencil into the motherboard, or route the whole city network through your laptop to get the job done,' Venezia writes."
Re:Rubber Band (Score:4, Interesting)
Floppy disk in the wash (Score:5, Interesting)
I got "lucky" to solve a problem for someone back in college: she had written her thesis on a 3.5 floppy, had no backup (this is when you had to go to the "computing center" to work, as practically no one had a machine of their own, so you had to take all your stuff with you), and had run the disk through the washing machine.
She came in, crying hysterically (it actually took a few tries just to figure out what was wrong), and realized what had happened. I had one of the few "eureka!" moments of my life, and grabbed another floppy, carefully cut it open, did the same with her disk, then air-dried it. I put the platter in the "new" disk, with its dry fabric covering (whatever that stuff was...), taped it shut, and put it in the Mac (SE...no hd) and yep, the disk was readable and I was able to get her thesis off and onto a network drive, then we copied it back onto a new disk and assured her I'd hold onto the thesis on the network drive until the end of the semester.
Funny thing, she kept the disk I had used, taped around the edges, and the next year I saw her again and asked how things were, and she was still using it. Go figure.
Computer Tech (Score:5, Interesting)
The dude couldn't afford anything new, so I offered my most MacGyver-ish attempt. I went over to the car-audio shop, grabbed some speaker wire, spliced out some copper about the same size as a pin, and voila!
After bending some of the pins back with a mechanical-pencil tip, and inserting the new 'pin' into the socket below the missing pin on the CPU (cut to semi-correct length), it booted right up! He took it home and all was well. I don't work for said company any more, but how long that 'fix' lasted is questionable.
Never told the boss about that one.
Depends on the point of view (Score:5, Interesting)
The first "stunt" depends on your point of view. If you have nicely brainwashed and duped by marketing material that "Vendor gear good, PC bad" that may sound as a stunt. If you actually know what you are doing you can run networks for years on this.
Nearly any laptop today has the forwarding grunt of an upper end of a 3800, there are plenty of servers that are on par with a 7200 or low end 7600 and most supervisor modules. You can run a network on this on a daily basis and do a _LOT_ of things a Cisco cannot do or cannot do at sufficient performance.
To put the so called "stunt" into a perspective, I used to run a production installation with 20+ 802.1q trunks via 800MHz Via EPIAs with 600+ entry ACL lists including content filtering with VRRP failover, load balancing to multiple upstream uplinks, OSPF, hardware accel-ed openvpn and ipsec, 16+ class hierarchy CBQ QoS and a few more bells and whistles. For years. Not for 48 hours.
Nothing wrong with it if you can do it. If you cannot - well, not everything in life is learned on CCXX and RedRat certification courses. C'est la vie.
Speaker Wire (Score:2, Interesting)
I had non-booting server board back in the late 90s and managed to track the problem to a scratch through one of the traces on the bottom of the board. Something had fallen between the board and the offsets and had worn through the circuit.
Having nothing to lose, I fired up the soldering gun and pulled out the only wire I had from a pair of speakers and sure enough, once the circuit was made the board booted and remained stable long enough for us to order and install a replacement.
Car Battery (Score:5, Interesting)
I once repaired a critical UPS that was attached to a critical database server actively recording data in the middle of a test shot with jumper cables and the battery from my truck. All that just to replace a fan that kept sending the UPS in to panic mode for an overheating battery and trying to start a shutdown sequence on the database server.It was a 12v power source for the UPS (old, old equipment) coming out of the AC to DC power supply. The UPS was part of a suite of equipment that included the database server, the array, a backup device, a network switch and the UPS hardwired to each of them in it's own rack. Don't ask me who made it. All I know is it was an Informix based DB and the maker was some esoteric, specific solution company I never heard of and before my time anyway. All I knew was the replacement parts had a 2 week lead time and I have no idea why this company chose to hold up such critical data with such arcane and unsupportable equipment. But, I had to shutdown the UPS to do the work but the battery didn't have enough juice to support the 30 minutes it was going to take to do the work. The battery power would have been killed once the unit was off anyway.
So I attached my jumper cables and the 600 amp battery from my truck to the output rails on the UPS, after the control switches. From there it was just juice to the rails and then to the server and it's data array. The car battery had about 45-55 minutes of juice for the suite to run on full-tilt. So I shut the UPS down and the servers, thankfully, stayed up! Had a box fan blowing on the battery and jumper cables. I disassembled the UPS case, cut the bad fan out and spliced the old connector on to the new fan I got at a local surplus store for $3. Plugged it all in, reassembled and turned the UPS on. It went through diagnostics and everything went green. Then the overload light started blinking and the warning chime came on. I pulled the jumper cables off and the overload warning went away and things stayed stable. The fan stayed on and nothing went down.
I probably should have gotten an award for it because it was a test shot for a multi-billion dollar contract but I was more afraid of disciplinary action over the risk than getting any praise for it. As far as I know, to this day, only two other people at that company know what happened
Unreadable CD/DVD (Score:4, Interesting)
Jackass #2 related (Score:4, Interesting)
In the Dim Times, my company had a couple of hard drives (those newfangled 3.5" Scuzzy drives) that wouldn't spin up and had critical data on them. My solution:
2. Hold the drive in my fingertips (so the platters were parallel with my palm)
3. Power on the computer, then "snap" the drive with a twist parallel to the platters, relying upon inertia to break the stiction.
4. Recover data from now-spun-up drives.
5. Power down, then physically destroy the interface pins on the drive to ensure nobody tried to use it again.
Since then, I've used that trick several times on dead/dying hard drives. As long as the heads are trying to move (indicating electrical life), it's worked every time.
Re:Depends on the point of view (Score:5, Interesting)
Or, depending on where you source your notebook computers from, the whole thing could fall over in a few hours.
A company I worked for did a similar stunt a few years ago by repurposing some old Latitude D600s as a development cluster when they ran out of money for real servers. On the surface it looked like a smart idea -- The hardware was already paid for, had a small form factor and every single one had its own built-in UPS. What could possibly go wrong?
The answer to that is that every few days at least one of them would die and need to be rebooted, reimaged or simply beaten with a club. Some things are designed to sit in racks and run non stop for years at a time, others are designed to sit on a table at Starbuck's and run for a few hours before shutting down. The trick is in knowing which ones are which.
Re:Car Battery (Score:5, Interesting)
:)
After a hurricane had wiped out power to Miami, I had to drive into a facility I maintained to get their email servers back online. It was critical for their remote employees to send in orders and time sheets. This was back before outsourced email services such as Google or Yahoo were available.
When I got there the power was still off. I had to rely on a 300W inverter plugged into the owner's truck battery. We ran a high gauge extension cord about 50' to the truck parked outside. Next we added a power strip to the end of the extension and plugged in the modem, server and monitor. On powering up the fuse for the cigarette adapter blew. We clamped up directly to the battery then. Powered up the monitor, modem, then PC. Everything worked for about 3 seconds until the BIOS splash screen turned on. Then it all went dead. The 300W inverter was not enough to power on both the server and the old CRT. We had the bright idea to charge a UPS for 30 minutes. With the monitor plugged into the UPS, we had just enough juice to see that the server has hanging on a bad filesystem. Then it died.
This is where it got fun.
I unplugged the monitor. As the system booted, I replayed in my head the steps I needed to bring the filesystem back. I knew that needed to login to maintenance mode first. I knew this by entering the root password then typing (blindly) "touch /tmp/foo; find /tmp -name foo". When I saw the hard drive light flicker when I pressed enter I knew I was at the shell.
I had to check the filesystems... I didn't remember what partition it was on, so on a piece of paper I wrote out an awk script that would peek through /etc/fstab, grab the relevant filesystems and the appropriate /dev entry, then pass that to stdout. I piped that output to a file then used that file to run fsck. All of this was done without seeing my commands or the output from those commands.
When the remote user was able to connect via mail then I knew it was working..
It wasn't particularly ingenious, but the circumstances made it memorable. Missing pieces of the room, navigating around downed trees to get to the site, complete darkness except for a door propped open on the other side of the room (server room was the farthest room in the office and had no windows or doors to the outside), hot hot hot hot hot (Florida weather), and users calling every five minutes trying to connect... Power came up later that day, but what an experience.
Minuteman Missile System (Score:5, Interesting)
Back in the early 60's I was on a three man combat targeting team and we had two minuteman missiles to startup and target one day. So we went to the first site and the maintenance team had just finished installing a new guidance and computer package and the nuclear warhead. They closed the 80 ton door that protects the missile and so it was out turn to perform.
We started up the on-board computer and ran some checks and then began loading in the targeting data such as whether it was a air burst or ground burst and all of the war-plans associated with it as well as the launch codes and targets.
After this is accomplished then the guidance package goes through some testing and self calibration and finally becomes "ready"
Ready is actually called "Strategic Alert" and lights a green light on our console.
The missile system sat in strategic alert for a few minutes and so we figured we had completed our job and would button up the site and head to our second site.
Suddenly the "Launch Commanded" light lit on the console and a fraction of a second later the "Launch in Progress" light also lit.
I quickly popped out a bunch of the circuit breakers on adjoining panels causing the support equipment to stop functioning.
At this stage we did not know if we had a bad console (portable between sites) or a computer failure on-board. Anyway the missile did not blow the umbilical nor launch so we believe we stopped it just in time. If we tried to check our technical data then we would have been dead most likely.
We contacted job control and they agreed not to attempt a restart and rather have maintenance replace the guidance/computer package yet again and return it to Autonetics for repair.
The next site we went to for startup went perfect and the console worked flawlessly...
That has been nearly 50 years ago now and i still occasionally wonder if the missile had actually entered "launch" or if the on-board computer was giving erroneous launch status.
Re:Rubber Band (Score:5, Interesting)
Re:Rubber Band (Score:4, Interesting)
If you would have had the dry ice and the drive in a cooler you dont have to worry about condensation. CO2 will displace all the air in the cooler, and itwill certianly be too cold in there to have more than 0% humidity.