Forgot your password?
typodupeerror
Image

Stunts, Idiocy, and Hero Hacks 208

Posted by samzenpus
from the when-it-just-has-to-get-done dept.
snydeq writes "InfoWorld's Paul Venezia serves up six real-world tales of IT stunts and solutions that required a touch of inspired insanity to pull off, proving once again that knowing when to throw out the manual and do something borderline irresponsible is essential to day-to-day IT work. 'It could be server on the brink of shutting down all operations, a hard drive that won't power up vital data, or a disgruntled ex-employee who's hidden vital system passwords on the network. Just when all seems lost, it's time to get creative and don your IT daredevil cap, then fire up the oven, shove the end of a pencil into the motherboard, or route the whole city network through your laptop to get the job done,' Venezia writes."

*

This discussion has been archived. No new comments can be posted.

Stunts, Idiocy, and Hero Hacks

Comments Filter:
  • Rubber Band (Score:4, Funny)

    by Gotung (571984) on Monday December 13, 2010 @01:44PM (#34535984)
    I once fixed an issue that was holding up the operations of a $50 million dollar a year company with one well placed rubber band.
    • Re:Rubber Band (Score:4, Interesting)

      by ColdWetDog (752185) on Monday December 13, 2010 @01:48PM (#34536018) Homepage
      I once took out most of the internal network of a major hospital by innocently tugging on some duct tape while waiting for a Novel server to reboot. But I think we're not supposed to talk about those sorts of 'solutions'.
      • by Yvanhoe (564877)
        I once saw people hand-starting a critical hard-drive whose motor had failed. It was opened so they knew it would die quickly but they did it to quickly copy a few critical MB of acounting data.
        • Re:Rubber Band (Score:4, Insightful)

          by FatdogHaiku (978357) on Monday December 13, 2010 @03:25PM (#34537182)
          I have packed drives in dry ice to get one last read out of them. Once the drive is very cold it's a race to get the data before condensation builds up on the circuit board. Side note, bare metal "screams" when pressed against dry ice... gas hammering against the metal.
          My normal MO for this is stick cables on the drive, stick the drive in a heavy freezer baggie with the cables sticking out the open end of the bag. Rubber band or tape it shut as best you can with the cables sticking out. Put a block of dry ice on the top and bottom of the drive (outside the bag) and wrap it with something like a thin towel or paper to keep the dry ice in contact with dirve. Stick the whole thing in the freeze for at least half an hour before trying to spin up the drive. Leave the dry ice in place and hook the drive up (I have done both USB and IDE connections). If you are lucky you can HURRY and copy your data... If not you're out $5 for dye ice.
      • by Cytotoxic (245301)

        I did something similar to a major fiber backbone for several carriers. It seems that the stack of old, abandoned telecommunications equipment in our new building wasn't as old and abandoned as I was led to believe. It seems that by pulling those tiny knobs labeled "Sprint", "MCI", "AT&T" etc. to find out what they were for, I was disconnecting their fiber uplinks. Ooops. The tech who came out about an hour later was really puzzled when he couldn't find anything wrong. Well at least until I came ba

    • by pilgrim23 (716938)
      WAY back in the day when Mainframes ruled the land, data was written to 9 track tape. Occasionally a data check would occur when data was written to a bad spot (crystal int eh ferrite mix coating the tape, wrinkle, etc) on the tape. Old Operator trick: Pop open the Vacuum door on the tape drive, pull out the tape, Lay the spot on that tape over the top of the tape drive door, briskly rub with a big square rubber eraser kept for this purpose, put tape back and hit START on the drive. Usually the t
      • What did that actually do?

        • Re:Rubber Band (Score:5, Interesting)

          by pilgrim23 (716938) on Monday December 13, 2010 @03:56PM (#34537872)
          it would smooth out crystals in the ferrite matrix of the tape. Seriously. Data at that time was measured as 1600BPI or 1600 bits per inch of tape recorded in 8 discreet tracks or "not very much". If a spot on the ferrite coating bridged the "tracks" this caused a data check (and bridging was the usual cause of issues). Running the eraser over the tape smoothed and broke this connection, resorting in the tape drive being able to read this bad spot. We are not here talking about the femto-micron gap between bits on a modern hard drive
      • by baegucb (18706)

        Also, DITTO could fix alot of those problems on IBM mainframes with a tape to tape copy. Oddly enough, from a hacker point of view, DITTO bypassed any security I ever found on an IBM mainframe. Great for looking at SYS1 datasets...wtf? RACF passwords in plaintext? ;)

      • by PPH (736903)

        You just erased the decimal point in my paycheck.

        Thank you!

    • by AaronD12 (709859)
      I once ran our company's support web site on my iBook G3 while the server was undergoing maintenance. This maintenance ended up lasting a week -- and people were amazed at how fast the support web site was running!
  • by wandazulu (265281) on Monday December 13, 2010 @01:53PM (#34536050)

    I got "lucky" to solve a problem for someone back in college: she had written her thesis on a 3.5 floppy, had no backup (this is when you had to go to the "computing center" to work, as practically no one had a machine of their own, so you had to take all your stuff with you), and had run the disk through the washing machine.

    She came in, crying hysterically (it actually took a few tries just to figure out what was wrong), and realized what had happened. I had one of the few "eureka!" moments of my life, and grabbed another floppy, carefully cut it open, did the same with her disk, then air-dried it. I put the platter in the "new" disk, with its dry fabric covering (whatever that stuff was...), taped it shut, and put it in the Mac (SE...no hd) and yep, the disk was readable and I was able to get her thesis off and onto a network drive, then we copied it back onto a new disk and assured her I'd hold onto the thesis on the network drive until the end of the semester.

    Funny thing, she kept the disk I had used, taped around the edges, and the next year I saw her again and asked how things were, and she was still using it. Go figure.

    • by LMacG (118321) on Monday December 13, 2010 @01:58PM (#34536104) Journal

      I don't think you understand the term "got lucky." Oh right, I'm reading Slashdot ...

      • +1 for the laugh!

      • by Ihmhi (1206036) <i_have_mental_health_issues@yahoo.com> on Monday December 13, 2010 @03:29PM (#34537268)

        Hell, with that kind of IT heroics, who's to say he didn't? He might just be doing the gentlemanly thing and not talking about it.

        I mean, really, "I saved her thesis from being lost forever and then banged her brains out with her leaning against a blade server" is a tad bit uncouth, wouldn't you say?

        • Re: (Score:2, Funny)

          by Anonymous Coward

          Uncouth? Yes. But it's *really* hot. Tell us more about the blade server.

        • by Jawnn (445279)

          Hell, with that kind of IT heroics, who's to say he didn't? He might just be doing the gentlemanly thing and not talking about it.

          I mean, really, "I saved her thesis from being lost forever and then banged her brains out with her leaning against a blade server" is a tad bit uncouth, wouldn't you say?

          Uh..., wait a minute. I'm trying to get a mental picture... Oh yeah. Definitely!

    • by Skater (41976)

      My girlfriend (now wife) managed to snap the USB connector off of her thumb drive...of course, without backups. I was able to solder it back on and it worked. (Before I started, I wasn't sure how hard it would be, but the soldering went pretty well, so I wasn't that surprised when it did work.) I copied the data off of it, burned the data to CD, and promptly threw the USB drive away to prevent any temptation in using it again!

      In hindsight, I probably should've at least taken a picture of it.

      • Yeah, I did the same thing with this USB mini disk my girlfriend has (it's actually a mini hard disk).

        It was really hard to do because the 4-wire lead between the connector and the PCB was only 1" long, and also it was exposed stranded wire that had only been casually coated - there was no actual insulation (cheapass Chinese construction), so I had to collect several rouge strands to prevent a short.

        I finally got it soldered and got the data off, and told her she can't use it anymore. I guess I could take

        • by tehcyder (746570)

          I guess I could take a picture if you want to see it :D

          You could even have her holding the USB mini disk if you want.

    • by CAIMLAS (41445) on Monday December 13, 2010 @03:07PM (#34536866) Homepage

      Did you get laid for that one? I had several such opportunities resulting from obscene levels of gratitude.

      I came to a school as a sophmore, and ended up staying in a co-ed mixed year dorm. The school didn't have a heavy (any) IT/CS focus, and this was right around when the quality of floppy disks and drives was "questionable" - on a good day, with very careful handling (1999-2001), you might get a disk to work in a drive that didn't write it. All it took was one successful dd recovery of a floppy disk and the word got around.

      I loved that old Toshiba floppy drive: it was so much more reliable than the drives of the era, ran quickly, and could read pretty much any 'corrupt' data. Very rarely was anything unreadable.

      As someone else said, being poor, on a time crunch, or limited by other people's failure to plan does seem to result in some pretty good 'hacks'. I didn't think the list they picked was all that spectacular: many of the "unconventional" ones have been done before by many others, I'm sure.

      * Riser card creep? Hot glue (I always keep a gun handy now)
      * Routing traffic through a Linux laptop? That might be a "jackass hack", but many people do it on a planned, regular basis, and have for the better part of a decade. Move along...
      * Cook your drive? I've never heard of that trick, though I have frozen drives to recover data (bucket of ice, water, and a little water purifier salt, with a triple-bagged hard drive).
      * enable password on the network? Anyone using rancid and no encryption does this; I'm sure there are others.
      * heartbeat - I had to do this temporarily (or something like it). I used netcat.
      * The timezone settings? Pretty sure that there's nothing 'jackass hackish' about that - that's just a common part of remote system deployment. I've worked at several places which have done things in a similar fashion.

      Other "jackass hacks" I've done (that I don't think are all that incredible):

      * Expensive network MFD printer's built-in ethernet died - but it had USB. Hooked a laptop up and shared the printer, with scans getting automatically dumped to a shared path until a replacement could be acquired (small office).
      * Could not get a back plate adapter for a supermicro tower chassis from them on time to use a standard, quality ATX PSU, as I'd already had multiple (shit) PSUs from SM. Spent an hour that night at home cutting one from an old Dell Optiplex case via a cardboard stencil so I could get the system back up.
      * Plastic CPU mounting bracket used for the HSF on a first-generation Opteron cracked due to the OEM tension bracket being too tense. -Carefully- drilled 4 small holes in the board and attached the HSF via a cat5 insulated strand garrote, tensioned on the back of the board.
      * Modem bank had modems that were hanging with regularity. Better airflow helped, but no cigar. Determined the wall wort PSUs were getting warm and causing the modems to crash. Wired up an old(er) tower PSU to provide the power to the modems directly, and threw in a couple 12v fans for good measure. Problem solved.
      * Customer complains about server noise. Five minutes and a drop of machine oil on the CPU fan and the problem is 'fixed' - no charge, and the customer is happy.
      * Virtual server had a controller + disks blowout... put the system's VMs on a remote network share from backup and booted it from USB. Had it back up in slightly more time than it took to copy the images over.

      These are just the tricks of the trade: we make hackish decisions like this every day to "just get the job done". Hopefully we can go back and fix them properly at a later time.

      • by skids (119237)

        Well, personally I've got plenty of such war stories, but you never seem to have an opportunity to get laid for looping back unused OC-3 ports to boost signal when a fiber degrades, putting spare optical amps into a line to graph signal level and verify the cause of an intermittent outage occurs exactly when outdoor temperature crosses 0C, or taping a pencil and a mouse to a cd tray in order to develop input drivers for a machine located miles away (think "eject -c").

        Occasionally I wish I'd stayed in user-h

        • I've had to make my own baluns to replace existing ones that broke in the middle of the night (or on a weekend) several times. (Back when E1s where more popular, now everybody is getting SIP trunks).

          After the hosting company I worked for got an old an outdated server cracked, with hundreds of websites defaced, and we found out that the backup tapes were fried, I wrote a perl script to recover most files from the google cache :). They had only defaced all index.html files ... some sites got PHP files replace

    • by JamesP (688957)

      This reminds me you could bore a hole in a 720k disk to "turn it into" a 1.44Mb disk. WOW HEY!

      Not very reliable though...

      • by i.r.id10t (595143)

        Very reliable - just flip it over, use a second disk as a guide for where to punch the hole, and punch the hole using a hole puncher.

        Of course, when the 45 minute audio tapes came out I was all happy about having 50% more data storage, but the new tape stretched too much for data use...

  • by NiceGeek (126629) on Monday December 13, 2010 @01:54PM (#34536062)

    I don't have a story on the the scale of any of these but I remember fixing my Commodore 1541 drive by adding an extra screw. The drive belt had gotten stretched somehow and I was getting all kinds of read errors, and being poor, decided to attempt a repair. Turns out there was an empty screw hole near the drive belt, I put one in, stretched the belt around the screw to take up the extra slack and that drive was still working when I finally got rid of my C64 a couple years later.

    I know, cool story bro.

  • by digitalsushi (137809) <slashdot@digitalsushi.com> on Monday December 13, 2010 @01:56PM (#34536088) Journal

    I worked with Paul a long time ago at a mom&pop in NH. And I know that he personally did the drive trick and it worked. It was a 9 gig scsi drive with an smtp mqueue on it. He was extremely elated that it had worked, and his portrayal of the story to a wide-eyed netadmin noob (me) was one of those late-night, sipping coffee at the Red Arrow while the raid rebuilds sorta memories that you'll take to your grave.

  • Computer Tech (Score:5, Interesting)

    by cosm (1072588) <thecosm3@gmail . c om> on Monday December 13, 2010 @01:59PM (#34536112)
    Back when I was a computer tech for one of the big retailers, I had a customer bring in a machine that wouldn't boot. After interrogating the customer a little more, it turned out he had tried 'upgrading' his CPU, and in the process had broken off one of the Athlon XP's (shows age) pins by inserting the CPU in the wrong orientation.

    The dude couldn't afford anything new, so I offered my most MacGyver-ish attempt. I went over to the car-audio shop, grabbed some speaker wire, spliced out some copper about the same size as a pin, and voila!

    After bending some of the pins back with a mechanical-pencil tip, and inserting the new 'pin' into the socket below the missing pin on the CPU (cut to semi-correct length), it booted right up! He took it home and all was well. I don't work for said company any more, but how long that 'fix' lasted is questionable.

    Never told the boss about that one.
    • HA! that reminds me, i just did that a few months ago. I was straightening some pins on a CPU i had stupidly dropped and suddenly one of the pins broke!

      I gasped, but thought about it and cut a pin off of a pentium pro cpu i had on my desk. Put it in the socket where the regular pin should be, slap the cpu on top and lo and behold the machine booted and ran fine!

      The machine is still working in my company right now. I wonder how common it is of a hack. I was quite surprised that it actually worked, and stayed

      • by Jeng (926980)

        I wouldn't be surprised if hacks such as that were the grounds for Intel developing their LGA design.

      • by gfody (514448)
        I broke a pin off an old 486 chip once. Didn't do anything about it and the computer worked fine.
  • by arivanov (12034) on Monday December 13, 2010 @02:02PM (#34536142) Homepage

    The first "stunt" depends on your point of view. If you have nicely brainwashed and duped by marketing material that "Vendor gear good, PC bad" that may sound as a stunt. If you actually know what you are doing you can run networks for years on this.

    Nearly any laptop today has the forwarding grunt of an upper end of a 3800, there are plenty of servers that are on par with a 7200 or low end 7600 and most supervisor modules. You can run a network on this on a daily basis and do a _LOT_ of things a Cisco cannot do or cannot do at sufficient performance.

    To put the so called "stunt" into a perspective, I used to run a production installation with 20+ 802.1q trunks via 800MHz Via EPIAs with 600+ entry ACL lists including content filtering with VRRP failover, load balancing to multiple upstream uplinks, OSPF, hardware accel-ed openvpn and ipsec, 16+ class hierarchy CBQ QoS and a few more bells and whistles. For years. Not for 48 hours.

    Nothing wrong with it if you can do it. If you cannot - well, not everything in life is learned on CCXX and RedRat certification courses. C'est la vie.

    • by Minwee (522556) <dcr@neverwhen.org> on Monday December 13, 2010 @02:53PM (#34536714) Homepage

      If you actually know what you are doing you can run networks for years on this.

      Or, depending on where you source your notebook computers from, the whole thing could fall over in a few hours.

      A company I worked for did a similar stunt a few years ago by repurposing some old Latitude D600s as a development cluster when they ran out of money for real servers. On the surface it looked like a smart idea -- The hardware was already paid for, had a small form factor and every single one had its own built-in UPS. What could possibly go wrong?

      The answer to that is that every few days at least one of them would die and need to be rebooted, reimaged or simply beaten with a club. Some things are designed to sit in racks and run non stop for years at a time, others are designed to sit on a table at Starbuck's and run for a few hours before shutting down. The trick is in knowing which ones are which.

      • by arivanov (12034)

        There is a world of difference here.

        Forwarding 1GBit of traffic coming over 802.1q trunk is under 5% load on an _UNDERCLOCKED_ laptop if you have an Intel NIC. Probably around 10% on Broadcom and other NICs which have checksum offload and do 802.1q brute force. Realtek is a ritual suicide so unless you want to do precise QoS and need the interrupts it is better not to mention it. You can have a laptop work for months on this doing this and survive. I would never consider using it anyway. A miniITX (MB in a

    • by skids (119237)

      Really it's a matter of whether your employer has the ability to recruit Linux networking talent. If you're a rare hire, and they cannot usually get that talent, best to stick with solutions a Cisco Certified Network Alpaca is familiar with.

      • by Lumpy (12016)

        But hiring a competent IT tech that knows Linux is far FAR cheaper than hiring a CCNE.

  • Speaker Wire (Score:2, Interesting)

    by Anonymous Coward

    I had non-booting server board back in the late 90s and managed to track the problem to a scratch through one of the traces on the bottom of the board. Something had fallen between the board and the offsets and had worn through the circuit.

    Having nothing to lose, I fired up the soldering gun and pulled out the only wire I had from a pair of speakers and sure enough, once the circuit was made the board booted and remained stable long enough for us to order and install a replacement.

  • by digitaldc (879047) * on Monday December 13, 2010 @02:12PM (#34536234)
    Trust me, they will come in handy in the lab at some point. Even for sudden headcrab infestations.
    • by skids (119237)

      End cutters. Don't forget the end-cutters. The flat claw kind, not the slanted ones. Great for stripped bolts.

  • by AndGodSed (968378) on Monday December 13, 2010 @02:12PM (#34536240) Homepage Journal

    I once took my laptop and used it to set up an Apache + DNS server while replacing a webserver that died. All I did was to post a "Emergency Maintenance" page while we swopped out the server.

    Every IT guy who has been in the trenches for 10+ years has "I once" stories. Oftentimes they salvaged hundreds of thousands of rands of damages for the company, or helped mitigate a bad management decision.

    The thing is, one of several scenarios invariably happen:

    1 - You get no recognition because no one understands what you did. ("Oh, you had another web server running on your laptop, that's dandy!")
    2 - You get an accusing look. ("How was it possible that this happened? Sure you fixed it but this should not have happened, make sure it doesn't happen again.") - I saw something like this happen to a senior network admin once, something totally out of IT's control that occurred due to a bad management decision not to buy a spare router. We used an old PC with IPtables to route traffic on a network over a weekend while our suppliers tried to source one.
    3 - The dark suit analogy: Doing a good job is like spilling coffee on a dark suit, you feel warm all over, but nobody notices.

    Being in IT is a bitch, and management doesn't help - IT is honouring the impossible promises of management to unthankful clients.

    • by jjohnson (62583) on Monday December 13, 2010 @02:25PM (#34536404) Homepage

      "How did this happen?"

      "We were running without a spare router because you turned down the request I made to purchase one. I can forward you and your boss all the documentation on my request, including the cost analysis for suffering an outage like this because we didn't buy the router, if you like."

      • by ArsonSmith (13997) on Monday December 13, 2010 @02:45PM (#34536630) Journal

        That'd be a nice story to tell to the other people in line at the unemployment office. You can even show them the printouts of the email you made before you were walked out.

        • by operagost (62405)
          Well, it will help you in case they try to deny your benefits. And it will also be nice to have in that "wrongful termination" lawsuit.
      • by AndGodSed (968378) on Monday December 13, 2010 @03:09PM (#34536900) Homepage Journal

        "Yes but you should have made sure we made the correct decision. You are an expert in your field why did you not push your position harder?

        Should I get another IT guy willing to take responsibility for his department or are you going to make sure we make the correct decisions in the future?

        You are an adult, you should be aware that people can make the wrong decisions and be prepared for any eventuality.

        Now go and get me a quote on replacement hardware so that I can make a decision and take it up with the MD."

        Verbatim.

    • by Ihmhi (1206036)

      Speaking from experience, IT is one of those jobs where nobody notices or cares about you until something goes wrong.

      • by AndGodSed (968378)

        And the question is invariably along the lines of "Hey why can't we access our email?" or "Why is the internet down." as opposed to "Hey were you aware that..."

        IT guys are not bloody clairvoyant.

        My response is usually either "I wasn't aware that [x or y]" OR "Working on it, talk later, cheers."

    • by AK Marc (707885)
      Worse, when you use the $300 spare laptop to replace a $10,000 supervisor module, they ask why they can't just deploy them everywhere to save money. I've had quick fixes become corporate standards more than once, and it's never pretty. Better to never let anyone know what happened and not take any credit either.
  • Car Battery (Score:5, Interesting)

    by Ynsats (922697) on Monday December 13, 2010 @02:12PM (#34536242)

    I once repaired a critical UPS that was attached to a critical database server actively recording data in the middle of a test shot with jumper cables and the battery from my truck. All that just to replace a fan that kept sending the UPS in to panic mode for an overheating battery and trying to start a shutdown sequence on the database server.It was a 12v power source for the UPS (old, old equipment) coming out of the AC to DC power supply. The UPS was part of a suite of equipment that included the database server, the array, a backup device, a network switch and the UPS hardwired to each of them in it's own rack. Don't ask me who made it. All I know is it was an Informix based DB and the maker was some esoteric, specific solution company I never heard of and before my time anyway. All I knew was the replacement parts had a 2 week lead time and I have no idea why this company chose to hold up such critical data with such arcane and unsupportable equipment. But, I had to shutdown the UPS to do the work but the battery didn't have enough juice to support the 30 minutes it was going to take to do the work. The battery power would have been killed once the unit was off anyway.

    So I attached my jumper cables and the 600 amp battery from my truck to the output rails on the UPS, after the control switches. From there it was just juice to the rails and then to the server and it's data array. The car battery had about 45-55 minutes of juice for the suite to run on full-tilt. So I shut the UPS down and the servers, thankfully, stayed up! Had a box fan blowing on the battery and jumper cables. I disassembled the UPS case, cut the bad fan out and spliced the old connector on to the new fan I got at a local surplus store for $3. Plugged it all in, reassembled and turned the UPS on. It went through diagnostics and everything went green. Then the overload light started blinking and the warning chime came on. I pulled the jumper cables off and the overload warning went away and things stayed stable. The fan stayed on and nothing went down.

    I probably should have gotten an award for it because it was a test shot for a multi-billion dollar contract but I was more afraid of disciplinary action over the risk than getting any praise for it. As far as I know, to this day, only two other people at that company know what happened

    • Re:Car Battery (Score:5, Interesting)

      by digitalhermit (113459) on Monday December 13, 2010 @02:56PM (#34536734) Homepage

      :)

      After a hurricane had wiped out power to Miami, I had to drive into a facility I maintained to get their email servers back online. It was critical for their remote employees to send in orders and time sheets. This was back before outsourced email services such as Google or Yahoo were available.

      When I got there the power was still off. I had to rely on a 300W inverter plugged into the owner's truck battery. We ran a high gauge extension cord about 50' to the truck parked outside. Next we added a power strip to the end of the extension and plugged in the modem, server and monitor. On powering up the fuse for the cigarette adapter blew. We clamped up directly to the battery then. Powered up the monitor, modem, then PC. Everything worked for about 3 seconds until the BIOS splash screen turned on. Then it all went dead. The 300W inverter was not enough to power on both the server and the old CRT. We had the bright idea to charge a UPS for 30 minutes. With the monitor plugged into the UPS, we had just enough juice to see that the server has hanging on a bad filesystem. Then it died.

      This is where it got fun.

      I unplugged the monitor. As the system booted, I replayed in my head the steps I needed to bring the filesystem back. I knew that needed to login to maintenance mode first. I knew this by entering the root password then typing (blindly) "touch /tmp/foo; find /tmp -name foo". When I saw the hard drive light flicker when I pressed enter I knew I was at the shell.

      I had to check the filesystems... I didn't remember what partition it was on, so on a piece of paper I wrote out an awk script that would peek through /etc/fstab, grab the relevant filesystems and the appropriate /dev entry, then pass that to stdout. I piped that output to a file then used that file to run fsck. All of this was done without seeing my commands or the output from those commands.

      When the remote user was able to connect via mail then I knew it was working..

      It wasn't particularly ingenious, but the circumstances made it memorable. Missing pieces of the room, navigating around downed trees to get to the site, complete darkness except for a door propped open on the other side of the room (server room was the farthest room in the office and had no windows or doors to the outside), hot hot hot hot hot (Florida weather), and users calling every five minutes trying to connect... Power came up later that day, but what an experience.

      • by WhiteDragon (4556)

        When I was in college, my monitor died on my desktop (that I also used as a server). I spent about two weeks running the computer blind (with tricks like you mentioned of using the 'find' command to make the hd light blink so I knew I was at the command prompt). For tasks that I absolutely needed to see the output of, I'd pipe them through lpr (to my old dot matrix printer.) I actually set up the printer as a tty at one point too. Since it was a dot matrix printer, it wasn't page based, so you could act

      • You should have used the scroll lock light to display the output text as Morse code.

  • Unreadable CD/DVD (Score:4, Interesting)

    by xded (1046894) on Monday December 13, 2010 @02:13PM (#34536256)
    Look for scratches on the bottom side, brush with toothpaste (the plain one, no additional abrasive ingredients), rinse, read.
    • by RogL (608926)

      Look for scratches on the bottom side, brush with toothpaste (the plain one, no additional abrasive ingredients), rinse, read.

      Or as happened repeatedly with a former boss:

      Rush out a data CD for him to test, he tests it during lunch & it's unreadable...

      Look for scratches on the bottom side, take it ito the men's room to wash off the peanut butter & jelly he'd gotten on it while eating, gently wipe dry & have him try it again. Deliver a stern lecture on the proper handling of CDs containing the master copy of the company's chemistry databases.

      • Even if this was years ago, why would you keep something so important on a CD? Don't you have a network?

  • Stunts, Idiocy, and Hero Hacks

    With a title like that, I was sure it was going to be another Wikileaks story.

    Phillip.

  • Late at night doing the stock prices, if one card of the COBOL pack was wrong, you'd find a punched-out confetti on the floor, and stick it back in the errant punch-hole, using a tube of polystyrene cement. Quick dry, no snagging, no delay. Just don't run those packs if you find them crisp in some archive.
    • by baegucb (18706)

      "the COBOL pack" must have been some shop name, COBOL is a language. Card readers did not time out. Why not just re-punch the card. The data should have been printed at the top of the card.

      • We handpunched each line of COBOL into each card. If the run went wrong, we debugged, changed the wrong card, and recompiled (we had tapes, that took time). Our handpunches didn't print at the top. Timeout? you always re-fed the whole program pack. Mending holes meant you didn't risk introducing new errors by repunching (alpha). And it was quick. And we had glue. And coffee kept hot on the 'central processor'. Happy days
  • From stunt #2 (bake the hard-drive):

    ...the disk presented to the SCSI controller just fine -- but it also didn't seem to spin up at all...

    Ya, I had that happen once, but I simply rapped my SCSI drive with the handle of my screwdriver - hard - right on the spindle head while the controller was trying to spin it up - "WHACK". Sucker spun up on the second hit. Still works fine.

    • Sounds familiar. I had an external hard drive with a lousy USB cord that wasn’t delivering the full juice needed to spin up the drive. When it was plugged in, it just made a sad clicking noise. Hold it in the palm of your hand, though, and a gentle twitch of the wrist would start it right up.

  • Simplest was splicing in a 4th drive power cord for a machine that needed another drive.

    Also wired an ISA card directly to the motherboard after the socket snapped off.

    Wired a laptops power supply to the motherboard, again after the socket ended up broken.

    Had an old sparc station that had a pin broken off of the keyboard/mouse cable and had to wire that together as well.

  • Jackass #2 related (Score:4, Interesting)

    by Caerdwyn (829058) on Monday December 13, 2010 @02:49PM (#34536678) Journal

    In the Dim Times, my company had a couple of hard drives (those newfangled 3.5" Scuzzy drives) that wouldn't spin up and had critical data on them. My solution:

    • 1. Find a long internal-type SCSI cable (about 30").
      2. Hold the drive in my fingertips (so the platters were parallel with my palm)
      3. Power on the computer, then "snap" the drive with a twist parallel to the platters, relying upon inertia to break the stiction.
      4. Recover data from now-spun-up drives.
      5. Power down, then physically destroy the interface pins on the drive to ensure nobody tried to use it again.

    Since then, I've used that trick several times on dead/dying hard drives. As long as the heads are trying to move (indicating electrical life), it's worked every time.

    • by mcrbids (148650)

      I've used the 'snap the drive in your hand' trick a few times, and add one more - the 'swap the motherboard with a new drive' trick!

      1) dead drive not seen by BIOS but spins up.

      2) RMA drive with advance shipping option, where they send new drive and u use the box to ship back dead drive. (With cred card)

      3) swap HD mainboard with RMA drive, copy data off now-functioning drive.

      4) swap mainboards back, ship dead drive to Mfg.

      This has worked several times over the years.

  • by OmniGeek (72743) on Monday December 13, 2010 @02:53PM (#34536712)

    I once was a sysop for a small company's Data General system, where large datasets were stored as TAR archives on nine-track tapes; some poor soul had copied TO the tape instead of FROM the tape, and desperately needed to recover a file that was still there on the part of the tape beyond the end of the inadvertent write. You could read up to the added end-of-tape marker, but the tape just wouldn't read any further. Screwed, yes? Well, not quite. I set the system to rereading the damaged tape, waited 'till just before it reached the offending end-of-tape marker, and briefly put my thumb on the roller that measured tape travel, causing the drive to jump the tape ahead ('cause the sensor said "the tape is not moving!") and right past the EOT marker. Voila! The system read out the rest of the files on the tape, fortunately including the one they really needed, and I was briefly a hero. Hero never lasts, of course, but it was fun.

  • by beadfulthings (975812) on Monday December 13, 2010 @02:53PM (#34536720) Journal

    I'm surprised they didn't mention the technique for unsticking recalcitrant half-height RLL and MFM hard disk drives by slamming them gently, but firmly, down onto a smooth horizontal surface (like your desktop). They would occasionally stick when the heads became goo-ed to the platters due to breakdown (or solidification, I was never sure which) of the lubricating material. When all other hope was abandoned, and you knew the drive was headed for the graveyard, a good, solid (but gentle) whack would often get it spinning again. The idea was to keep the drive as parallel as humanly possible to the horizontal surface. It was one of the few hardware tricks I had to summon male assistance to handle--my hand was not large enough to get the necessary firm one-handed grasp on the drive. Boy, do I feel old. Probably because I am old.

  • In a previous life as a network admin, I went to check out a warehouse that my department was taking over.. Everything was normal, as I made my list of comptuers, and started writing a to-do list for myself.. on about the third day I noticed the picking line had stopped.. I went over to the pickers area and saw the local IT guy opening up a computer.. so I asked what was up.. He said "oh this hard drive sticks now and then"... it was an non-network 486 running Windows 95 that had the entire carousel datab
  • Fire Axe (Score:5, Funny)

    by PPH (736903) on Monday December 13, 2010 @03:09PM (#34536896)

    Back in my days as an engineer at Boeing, I supported some automated test equipment on the factory floor. One day, one of the ATE failed to download the required s//w update, so I was called out to investigate. It turned out that the network drop adjacent to the equipment had been disconnected in the nearby network closet. (locked, of course). So I, with the factory manager in two, called the IT department to get it plugged back in.

    Me: "I'm in the Renton plant, at column XYZ and we need this network drop reconnected. Production has been halted."

    IT Operator: "OK. We'll start a ticket on that. But standard turn-around is 24 hours".

    Me: "We can't wait 24 hours. We need to get this equipment updated to get the line up and running. Is there any way to escalate this?"

    IT Operator: "Sorry. That drop is was identified as being inactive and was unplugged."

    Of course it was inactive. The ATE is only powered up when needed. At other times, the little light on the switch in the closet would be off.

    At this point, the factory manager asked for the phone. Very calmly, he spoke to the IT operator.

    Manager: "You can cancel that ticket. My engineer assures me that he can reconnect the drop once he gains access to the network closet. The plant fire department is just downstairs and we'll have them bring up a fire axe to open the door."

    The IT department dispatched a tech who arrived within 15 minutes.

  • I don't think my parents were ever as impressed with me as on the day when I rescued their data, including many years of precious photos, from their crashed hard drive. After diagnosing the click of doom on their drive, I wrapped it in a towel, then two bags of that blue freezer gel, another towel and a plastic bag, and in this state left it to freeze overnight. It had a SATA cable and a SATA power adapter cable sticking out, and I did my best to seal the plastic bag with tape to avoid condensation once I t
  • by Sanat (702) on Monday December 13, 2010 @03:35PM (#34537410)

    Back in the early 60's I was on a three man combat targeting team and we had two minuteman missiles to startup and target one day. So we went to the first site and the maintenance team had just finished installing a new guidance and computer package and the nuclear warhead. They closed the 80 ton door that protects the missile and so it was out turn to perform.

    We started up the on-board computer and ran some checks and then began loading in the targeting data such as whether it was a air burst or ground burst and all of the war-plans associated with it as well as the launch codes and targets.

    After this is accomplished then the guidance package goes through some testing and self calibration and finally becomes "ready"

    Ready is actually called "Strategic Alert" and lights a green light on our console.

    The missile system sat in strategic alert for a few minutes and so we figured we had completed our job and would button up the site and head to our second site.

    Suddenly the "Launch Commanded" light lit on the console and a fraction of a second later the "Launch in Progress" light also lit.

    I quickly popped out a bunch of the circuit breakers on adjoining panels causing the support equipment to stop functioning.

    At this stage we did not know if we had a bad console (portable between sites) or a computer failure on-board. Anyway the missile did not blow the umbilical nor launch so we believe we stopped it just in time. If we tried to check our technical data then we would have been dead most likely.

    We contacted job control and they agreed not to attempt a restart and rather have maintenance replace the guidance/computer package yet again and return it to Autonetics for repair.

    The next site we went to for startup went perfect and the console worked flawlessly...

    That has been nearly 50 years ago now and i still occasionally wonder if the missile had actually entered "launch" or if the on-board computer was giving erroneous launch status.

  • A small addition was made to the autoexec.bat on the client, simply to run curl to access the Perl CGI script, then feed the output to the settz utility, thereby properly setting the time zone of each client every time it booted

    Being able to modify the autoexec.bat file, they could have written a solution that required no third-party software. I used to change all manner of systems settings via .bat files, even modifying registry settings by creating .reg files on the fly and calling regedit to load them.

  • by Lumpy (12016) on Monday December 13, 2010 @04:23PM (#34538354) Homepage

    "bubblegum and duct-tape fix"

    Yeah.. as if a Watch-guard, NOKIA, or CISCO firewall are any better than a Laptop running Debian.

    Sounds like the writer knows nothing about firewalls at all.

  • by eepok (545733) on Monday December 13, 2010 @04:38PM (#34538610) Homepage

    It was the 3rd year of undergrad and my roomie and come to me with a problem: his brother's computer goes black on booting up to windows. Safe Mode wouldn't work. Finals was the following week and his brother had 2 papers on the computer, unfinished, which had to be turned in on Monday. I told him to bring the computer down and I'd do what I could.

    I hooked his computer up to my KVM (best low-space hobby troubleshooter investment I ever made). I booted up and the diagnosis was definitely correct. The second Windows tried booting, the screen would go black and Safe Mode crashed on each attempt. He didn't have a recovery disc for his factory installation so I just had to wing it... without seeing my actions on screen.

    I booted up to XP Pro on my computer (which he was using on his) and wrote down all the keystrokes, tabs, enters, etc. in order to get down a method of setting the display settings to minimum settings (Windows key, up x times, right once, etc.). That didn't change anything. I then set out to uninstall the drivers, again writing down the operation as performed on my computer and then repeating the process blind on his. That still didn't fix it. "Oh!," I thought, "Maybe I'll uninstall the device and reboot... duh!" I did that blind, rebooted, and the desktop was viewable. I spent the next 3 hours removing viruses and malware.

    Lesson to the brother: Don't install ATI drivers for their built-in software overclocking when you have an NVidia card.

    Lesson to me: Fix computer, get beer.

    • by lordlod (458156) on Tuesday December 14, 2010 @01:06AM (#34543106)

      When you are spending so long doing something awkward it's normally worth sitting back for a few minutes and reconsidering the goal and approach.

      Goal: Recover documents off computer.

      Solution 1: Spend hours writing down key strokes and working blind.

      Solution 2: Plug harddrive into another computer and retrieve files.

      Solution 3: Use VGA mode or any Windows install disk to recover drivers.

      Most of the time when you are working hard it's because you are doing it wrong.

  • A friend on an IT coop asked for my help when a student repoted that the paper they had been writing all semester long was corrupt (stored on a floppy disk and no backup copy, of course). Needless to say, the student was very upset.

    I wrote a quick little program (in Pascal I believe) to read as much of the document file as possible and output just the printable characters. Told the student they'd have to reformat the text and double-check it but essentially it was all there. It felt good to help.

    That was

As the trials of life continue to take their toll, remember that there is always a future in Computer Maintenance. -- National Lampoon, "Deteriorata"

Working...