My home Linux server that I run this site on had it’s hard drive die last week. I found out that my weekly db backups had not been copying the MySQL dumps to the zip drive in the server, and the last one that was on the zip disk was ONE YEAR AGO.
Ok, lesson learnt here – CHECK YOUR BACKUPS! It’s one thing have a backup strategy in place, but it’s another thing to actually check your back ups to make sure they are good! The annoying thing in this case is I know the weekly MySQL dumps were running on a cron job, but they were all stored on the same drive that failed… not much good there.
Next lesson – Acronis True Image is awesome. I had a complete image of the drive that I was able to restore to a new disk. I took that image after the last substantial changes to the server, so didn’t lose any server setup.
Next lesson learnt – restoring a drive image to a completely different hard disk setup…. NIGHTMARE getting GRUB to boot Linux. Probably should have tested this before. Using VirtualBox though I was able to restore the image to a virtual drive and run a virtual copy of my server from the last backup. Completely awesome and a life saver.
Next lesson: Cheap RAID cards are most likely cheap because they are ‘fake RAID’ and rely on OS drivers to provide RAID support. I bought 2 new Hitachi Deskstar 250GB drives and set them up in a RAID 1 config (mirroring) for some degree of fault tolerance. Found out that the mirroring was not doing anything. Abandoned the H/W config on the card and set it up in Ubuntu software RAID config instead – excellent instructions here on how to do that.
So my server’s now back up. Time to look into FTP’ing weekly backups to some online hosted backup service instead….total time spent restoring backups, reconfiguring the server, reinstalling software – about 12 hours, but lost over 1 years worth of blog posts and technical articles on my site…