The Great Reconstruction


The webserver that hosted this site was also recently hosting (albeit temporarily) the Lupper worm. That has been rectified, but it wasn’t simple. There was an installation of awstats on the old server setup that I had mostly forgotten about. While I believe the firewall I had setup on the box prevented any of the backdoors the worm opened from being useful, I couldn’t be certain. I decided to rebuild the box. I chose to use a new gentoo install. After some initial pains with booting the LiveCD (because my CD-ROM doesn’t support DMA), I was off to the races. Slowly. I started on Wednesday of last week, and just managed to turn on the blog again last night.

During the process, I managed to lose all the content to the blog. Luck was on my side, and I managed to recovery it. WordPress uses MySQL as its backend database. Before I wiped the main drive of the server, I had exported the contents of the database and stored them on a separate drive. After getting gentoo up and running, I “emerge”d apache, MySQL, WordPress and other stuff I needed. (The virtual hosting in gentoo is well done, btw.)

I was in the process of restoring my database and made a one character error. Instead of using “< " to redirect the text file to the mysql command I used ">“ and redirected the output of the mysql command over my text file. One slip of the finger and two years worth of postings were gone. Ouch. I wasn’t pleased.

I managed to figure out that I could dump the unused blocks on the drive using debugfs. I let that go for a while and ended up with an 800MB+ file. A few greps later, and I confirmed my hope: the data was still on the drive. A little more investigation and I found the data was relatively contiguous in the dump file, AND because I hadn’t compressed the mysql dump, I could actually piece it back together. I threw together a couple of C programs. The first chopped out the stuff before my data and the second chopped out the stuff after my data. I ended up with a few megabytes of mostly text data that needed to be cleaned up, and as you can see, it worked.

What have I learned?:

  1. Understand your system
  2. Keep up to date on releases if you are going to have a public website
  3. Make sure backup files are read-only
  4. Don’t make major changes to your setup while tired, or if you do, triple check before hitting “return”

I’m confident managing this system will be much easier now that I am using gentoo. The old server had been running RedHat 8 with lots of stuff hand upgraded by me. Now it is running a 2.6.x kernel, and a sane packaging management solution. Kudos to gentoo. Hooray for being able to go to bed at a sane time again.

Be Sociable, Share!
  1. No comments yet.
(will not be published)