System Recovery using Knoppix

Written by Kevin Korb as a presentation for GOLUG Presented on 2006-04-06

• What is Knoppix?
Knoppix is a Live Linux distribution. That means that Knoppix is a distribution of Linux (based on Debian) that boots and runs from a CDROM or DVD disc and does not need to use the hard drive. There are many other live Linux distributions out there other than Knoppix and many of them will work for this just as well. I have personally used Knoppix-STD and Insert (from the UBCD) to do this same procedure. Just make sure you pick a disc that has all the hardware support and tools that you need.
• Why use it for recovery purposes?
Since Knoppix boots into a fully functional system without the use of the hard drive it is the perfect recovery platform. You will have all the recovery tools you would expect to find on a system that is functioning and doesn't need any recovery. Best of all, you can even use the computer while recovering it.
• When is this useful?
• Disk failure or corruption. If your boot disk fails or becomes hopelessly corrupted you can use Knoppix to easily recover it from a backup.
• System cloning. If you have a new computer and want to make it a clone of another computer you can use Knoppix to setup the new system. The procedure is the same as restoring from backup except that you then have to change the hostname and IP address to avoid conflicts.
• Bad kernel. If you have a kernel that does not boot (and don't have an old kernel that does) you can use Knoppix to boot the system, chroot into your OS, then setup a proper kernel.
• Bad bootup config. If you have something wrong with one of your config files that prevents the system from booting up at all you can boot Knoppix and edit the files from there.
• Windows problems. Since Linux can mount a Windows filesystem it is also possible to use Knoppix to fix some problems with Windows systems. Note however that NTFS support in Linux is safe as read-only but mounting an NTFS filesystem as read-write can cause problems. Vfat (fat16 and fat32) work fine either way.
• When shouldn't you use Knoppix for recovery?
Knoppix would only be needed to recover a system that can't boot from its hard drive. If you need to recover a data partition (like /home) but can still bootup then you can just do the restore from your normal OS install instead of booting up Knoppix which will be slower.

• Bootup Knoppix: Since the entire procedure is command line I normally bootup Knoppix without its GUI. The command for that on the boot prompt is:

  knoppix vga=normal 2

• Setup the network interface:

ifconfig eth0 192.168.100.12 netmask 255.255.255.0 broadcast 192.168.100.255
ifconfig eth0 up
route add default gw 192.168.100.1

• Setup system for remote access (optional):

edit /etc/resolv.conf
passwd root
/etc/init.d/ssh start

Note: Unless you are able to copy the host ssh keys from somewhere (/etc/ssh/*key*) the system will generate new keys. Therefore it may not be wise to bring the system up on its normal IP address if you plan to ssh into it from remote.


• Use scp to get the files you need:
Use the scp command to copy over any files that you need such as the sfdisk backup file used in the next step.


• Partition new disk:

sfdisk < sfdisk_backup_file

Note: If you don't have an sfdisk backup file of your partition table or you don't want to use the same layout then run fdisk and partition the disk manually.


• Setup any disk mappings (if needed):
If you use any software RAID, LVM, or Device Mapper mappings this is when you would set them up. RAID volumes would be setup with the mdadm command and dmcrypt partitions would be setup with cryptsetup or cryptsetup-luks. The details on this though are beyond the scope of this document.


• Format the partitions:
Use mkswap, mkreiserfs, mkfs.xfs, mke2fs (-j), and jfs_mkfs to format the new partitions with the filesystem of your choice.


• Mouth the partitions:
Mount the new filesystems in a tree under /g and activate the swap partition. Examples:

mkdir /g
mount -vt reiserfs -o noatime,notail /dev/sda3 /g
mkdir /g/boot
mount -vt reiserfs -o noatime,notail /dev/sda1 /g/boot
swapon /dev/sda2

• Use rsync to restore the files from the backup server:

  rsync -va -essh --progress --numeric-ids 192.168.100.11:/backup/rsync/[host]/_.current/ /g/
  rsync -va -essh --progress --numeric-ids 192.168.100.11:/backup/rsync/[host]/_boot.current/ /g/boot/

  Note: If you are cloning a live system you will want to add a -x to the rsync to prevent rsync from recursing into other filesystems. You may also want to add some --exclude= parameters to exclude some data directories that don't need to be cloned.
  
  
• Recover the /dev directory from tarball:

  cd /g
  tar -psxvzf dev.tar.gz

  Note: If you do not have this you can copy the /dev directory from the Knoppix CD and it might work. If you are running udev and don't have this then you can copy the /dev from udev after you reboot.
  
  
• Enter a chroot (optional):

  mount -vo bind /dev /g/dev
  mount -vo bind /proc /g/proc
  chroot /g /bin/bash
  source /etc/profile

  Note: You shouldn't need to do this for a simple restore but it is useful for other types of recoveries (like if you need to run your package manager to replace a core package needed to bootup). This is also useful if you need to build a different/better/correct kernel.
  
  
• Install grub:

grub-install --root-directory=/g /dev/sda

• Verify bootup files (if needed):
Verify the contents of /g/etc/fstab and /g/boot/grub/menu.1st and make sure that they are still accurate. This is especially true if you have modified the partition layout.


• Verify network config files (if needed):
Verify the contents of /g/etc/hosts, /g/etc/conf.d/hostname, and /g/etc/conf.d/net to make sure that they are still accurate. This is especially true if you are cloning a system and want the new system to have a different name and IP from the old one.
Note: The files in the conf.d directory are the ones that Gentoo uses. Your distro is probably different.


• Boot from the hard drive like normal
  
  
• Recover /dev from udev (if needed):
Login as root

mount -vo bind / /mnt
cp -aRv /dev /mnt/
umount -v /mnt

Note: this is only needed if you are running udev and did not have a /dev tarball to restore earlier.


• Recover any data partitions (if needed):
Use the same format, mount, rsync procedures to restore any data partitions that were also lost. This could be done under Knoppix but it is usually faster to do it from the normal OS once that is possible and you might want to have the normal OS up and running as quickly as possible.