National Warranty Expiration Week

Last week must have been National Warranty Expiration Week. My Roomba, Treo, and my Mac, all vital to me, had issues. The problem with my Mac was the strangest.

On Wednesday morning I got to work and suddenly I couldn’t get to my email at home. My machine was up, but it was rejecting my password. I couldn’t ssh in either and I couldn’t sudo. Uh oh, I think I’ve been hacked. I could login as Anne so I did that and looked around. No strange processes or connections from the outside world. I remembered I had created a passwordless SSH key at work, so I dug that up and got into my account. Everything looked normal, but I still couldn’t sudo, so I didn’t know for sure because I couldn’t look at all the system logs.

I have backups, but mostly only to a second drive in the machine so if someone rm -rf’d both of those droves, I’d have a problem. I considered rushing home to pull the ethernet cable out of the wall before any more damage was done. I decided to wait until the evening.

When I got home, the first thing I noticed was that the machine was really slow. Just trying to make it restart took a very long time and I ended up just hitting the reset button. Starting up again was also slow. It loaded all the services, but didn’t show the login window. Not good. I could SSH in though, so there was something more complicated going on. I suspected my hard disk was going bad.

I restarted the Mac with the Jaguar installer disk and changed the passwords on my account and root. I restarted, but still couldn’t login as me. I ran the disk utility too. It reported some minor errors (something about crosslinked files), but it said it repaired them. Then again, they kept reappearing whenever I rebooted. It wasn’t looking good for my hard drive.

Anne’s account was useful to log into the box, but since she doesn’t have an administrator account, it didn’t do me an account. Then I remember I gave Dave an account on my Mac once so he could try to fix some stuff AND I gave him admin rights. Starting up from the installer CD again, I changed the password on his account and I reboot later I was in as Dave. Thanks, Dave!

Now to fix my account. Even as root, passwd just kind of there and then returned nothing. Editing the /etc/passwd file did nothing of course since Apple uses NIS. passwd -i did return immediately. It looks like something was wrong with NIS was wrong. Great. I know absolutely nothing about NIS.

I guessed it was time for an NIS primer. I found a page on Apple’s site that described the nisinfo command. I compared my entry with Anne’s and found that her authentication method was ‘;basic;’ while mine was ‘;shadow;’. I updated my entry, copied Anne’s password hash to mine, and did the same for root. That did it. I was able to login as me again and as root. But this didn’t solve the mail problem. It still didn’t start up.

Between reboots, I checked out the Apple site to see how much Mac minis were. Still only 1.42 Ghz? Geez. It would certainly be faster than my current 466Mhz G4 though.

The disk fixes didn’t fix the startup problem though. I searched for “overlapped extent allocation” on Apple’s site using my laptop (how could anyone live without at least two computers?) and followed the instructions. The directions said this shouldn’t happen on journaled file systems (which mine was), but I did it anway.

reboot into single user mode
fsck
reboot into single user mode
fsck
find / -inum (inode num)

Aha! The file it revealed was this one

/Library/WebServer/Documents/squirrelmail-1.4.2/functions/i18n.php

Sure enough, although my webserver was up, squirrelmail definitely didn’t work. It spewed a whole bunch of random garbage to the login page. I made a copy of the bad file to my home directory and removed it. I rebooted one more time and now fsck didn’t report the crosslinked file error again. That was the good news. The bad news was that the machine still didn’t start up all the way after a reboot. Then again, why should it? This file had nothing to do with booting up properly.

I dug around some more in the logs.

One entry struck me:

/var/log/system.log
Oct 4 08:34:19 Bowtah mach_init[2]: Server 0 in bootstrap d03 uid 0: “/usr/sbin/DirectoryService”: exited as a result of signal 4 [pid 1393]
Oct 4 08:34:19 Bowtah crashdump: Started writing crash report to: /Library/Logs/CrashReporter/DirectoryService.crash.log

I opened up the crash log and found this:

Link (dyld) error:

dyld: /usr/sbin/DirectoryService malformed library: /System/Library/PrivateFrameworks/DirectoryServiceCore.framework/Versions
/A/DirectoryServiceCore (not a Mach-O file, bad magic number)

/System/Library/PrivateFrameworks/DirectoryServiceCore.framework/Versions/A/DirectoryServiceCore

Hmm, what was wrong with this file? I viewed it. Aha #2! The first part of the file was the i18n.php file from squirrelmail! This was the file that it was crosslinked with and it happened to be a pretty important file. Now I just need to figure out where to get a backup copy. Well, I could reinstall the OS, but that wouldn’t be much fun and I’d likely lose some settings. My backups were there, but I needed a bootable computer to run Retrospect. I know – my parents’ Mac! I scp’ed their copy of the DirectoryServiceCore over, replaced mine and rebooted. Finally, it booted normally! Wow, I didn’t think I would was going to pull this off.

I guess the only question left now is why the heck I would spend the time to type out all this boring junk. A few ideas come to mind.

  • If I don’t, I’ll never remember it
  • I don’t have anything better to do on my flight back to the west coast
  • I guess I find this sort of thing ‘fun’ (assuming it ends well). It’s about solving a mystery and coming to a logical conclusion, learning somet things along the way.

Now I just need to figure out why my Roomba won’t hold a charge anymore. Well, it doesn’t have hard drive, so at least I know it’s not a problem with crosslinked files.

3 Comments

  1. Wow, sunspots? Jeff, October has been a wierd month.

    Our new wireless access point went up this week. No radio/antenna activity on it. No big deal, right?

    Verizon has it out for this maher.org server. Nothing but packetloss over the past week, over a fiber connection that shouldn’t go down. This is over 40 miles from our house.

    Our well pump wasn’t working on Sunday night, and turning it on and off made various lights in the house bright, off, then dim. Randomly. Our freezer was lukewarm and some outlets in the house were producing 58 volts. Our yard is now a dirt ditch playground for the electric company.

    Oh yeah, the power company cut our phone lines too.

    I will build a shrine out of defective electronic equipment. I recommend you do the same.

    My Treo just rebooted.

  2. Your roomba’s battery has maxed out.

    Rechargables have a limited number of times that they can recharge, and memory or no, they’ve got a limit. It’s why one of the things I look for in any electric doo-dad is how easy it is to replace the battery and how much they cost.

Leave a Reply

Your email address will not be published. Required fields are marked *