Fixing Problems with the Root Filesystem (RedHat and CentOS)


Here is a simple example demonstrating a problem with the root filesystem. The cause is a typo in /etc/fstab. The fix is easy if we can figure out how to boot Linux. Attempting to boot Linux results in the following:
Initializing USB keyboard:                              [  OK  ]
Initializing USB mouse:                                 [  OK  ]
Checking root filesystem
LBEL=/:
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate
superblock:
      e2fsck -b 8193 <device>

fsck.ext3: No such file or directory while trying to open LBEL=/
                                                        [FAILED]

*** An error occurred during the file system check.
*** Dropping you to a shell; the system will reboot
*** when you leave the shell.
Give root password for maintenance
(or type Control-D to continue):

Booting from the mkbootdisk floppy gives the same result. The problem isn't with /bootit is with /. We need a way to get the system up to fix or restore /.
There are three common ways to boot when the Linux boot fails:
  • Boot from another hard disk
  • Boot from a rescue CD
  • Boot from a floppy rescue disk
Booting from a Second Hard Disk

The box we have been using in the examples is a dual-boot system with Red Hat on one disk and SUSE on the other. Repairing a damaged Red Hat root filesystem is easy in this configuration. We can just boot SUSE, mount the Red Hat root filesystem to some temporary mount point, and fix the problem.

Booting from a Rescue CD

A convenient method of repairing the root filesystem is to use a bootable Linux CD. We will use the Knoppix distribution (http://www.knoppix.net/), but there are many choices, and the most popular are listed at the end of this section. A rescue disk might have been provided with your Linux distribution CDs that can serve the same purpose.
You will probably need to modify the BIOS settings to boot from the CD-ROM or DVD drive before the hard disks. After that is done, restart the computer with the Knoppix CD in the drive. After the Knoppix CD boots, log into KDE. There is no password. Open a terminal window and run df to see what is mounted. The following listing shows that no disk filesystems are mounted.

knoppix@ttyp0[knoppix]$ su - root
root@ttyp0[~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 3471 1113 2358 33% /
/dev/scd0 716308 716308 0 100% /cdrom
/dev/cloop 1943588 1943588 0 100% /KNOPPIX
/ramdisk 95948 2020 93928 3% /ramdisk
root@ttyp0[~]#

The hard disks are not mounted, but we can see that they are in /etc/fstab. It is easy enough to mount them, as you can see here:

root@ttyp0[~]# cat /etc/fstab
/proc /proc proc defaults 0 0
/sys /sys sysfs noauto 0 0
/dev/pts /dev/pts devpts mode=0622 0 0
/dev/fd0 /mnt/auto/floppy auto user,noauto,exec,umask=000 0 /dev/cdrom
/mnt/auto/cdrom auto user,noauto,exec,ro 0 0
# Added by KNOPPIX
/dev/sda1 /mnt/sda1 ext3 noauto,users,exec 0 0
# Added by KNOPPIX
/dev/sda2 /mnt/sda2 ext3 noauto,users,exec 0 0
# Added by KNOPPIX
/dev/sda3 none swap defaults 0 0
# Added by KNOPPIX
/dev/sdb1 /mnt/sdb1 ext2 noauto,users,exec 0 0
# Added by KNOPPIX
/dev/sdb2 none swap defaults 0 0
# Added by KNOPPIX
/dev/sdb3 /mnt/sdb3 reiserfs noauto,users,exec 0 0
root@ttyp0[~]# mkdir /mnt/sda2

Mount /dev/sda2 filesystem with /mnt/sda2 (In my system /dev/sda2 partition is mounted with / filesystem)
root@ttyp0[~]# mount /dev/sda2 /mnt/sda2
root@ttyp0[~]#

We can see in the following listing that a typo is keeping our Linux box from booting. The fstab has a typo: LBEL=/should be LABEL=/ instead.

root@ttyp0[~]# cd /mnt/sda2
root@ttyp0[sda2]# ls
bin etc lib opt quota.group suse_root_sdb usr
boot fd lost+found original quota.user swapfile var
cdrom home misc proc root tftpboot web
dev initrd mnt prod_serv sbin tmp
root@ttyp0[sda2]# cat etc/fstab
LBEL=/ / ext3 defaults LABEL=/boot /boot ext3 defaults none /dev/pts
devpts gid=5,mode=none /proc proc defaults none /dev/shm tmpfs defaults
/dev/sda3 swap swap defaults /dev/cdrom /mnt/cdrom udf,iso9660
noauto,ro 0 0
/dev/fd0 /mnt/floppy auto noauto,owner,/swapfile none swap pri=2
/dev/sdb3 /suse_root_sdb reiserfs defaults /dev/sdb1
/suse_root_sdb/boot ext2 defaults root@ttyp0[sda2]#

Now just fix fstab, unmount the repaired root filesystem. The following shows the corrected fstab.

root@ttyp0[sda2]# cat etc/fstab
LABEL=/ / ext3 defaults LABEL=/boot /boot ext3 defaults none /dev/pts
devpts gid=5,mode=none /proc proc defaults none /dev/shm tmpfs defaults
/dev/sda3 swap swap defaults /dev/cdrom /mnt/cdrom udf,iso9660
noauto,ro 0 0
/dev/fd0 /mnt/floppy auto noauto,owner,/swapfile none swap pri=2
/dev/sdb3 /suse_root_sdb reiserfs defaults /dev/sdb1
/suse_root_sdb/boot ext2 defaults root@ttyp0[sda2]# cd
root@ttyp0[~]# umount /mnt/sda2
root@ttyp0[~]#

Now reboot from Harddisk.