For a project I’m working on I was recently provided with a CentOS 6.3 VM that kept giving me a kernel panic every time I tried to bring up the VM. I have never worked with CentOS and have not used Fedora or RedHad in over seven years, but I do have a strong Unix and Ubuntu background so I took up the challenge to get this VM up and running.
I wasted a lot of time trying to bring up the VM first in VBox and VMware just to get the kernel panic notice every time but the I focused on the kernel grub boot parameters. I took out the quiet and changed audit from 1 to 3 and was finally provided with the answer of what the problem was. The boot param was set to /dev/vda2 and the notice came back that partition was not found.
The rescue information in the RedHat and CentOS documentation was informative in helping me to modify the grub boot parameters and identify the issues but because this was a VM, mounting to get access to it too a little bit of ingenuity. I probably could have booted from a rescue CD on booting the VM but the chroot path I took did teach me some new methods to access a system.
I changed the grub boot param from /dev/vda2 to /dev/sda2 and the VM booted into maintenance mode. Finally after a day of effort I was getting somewhere but I did not know the root password to do the maintenance. That’s where chroot came in.
For those of you who are unfamiliar chroot changes the apparent disk root directory (and the current running process and its children) to another root directory. So basically I can mount this VM’s OS using another OS. The OS has to be the same architecture, i16, 64 bit, etc, as the one you want to chroot into. I used CentOS 6.5 because that made sense since it was available and just a minor release ahead. In theory you should be able to use Ubuntu or something else and maybe if I have a free night and some beer I’ll test that out.
I downloaded CentOS 6.4 64-bit and built a new VM with that. Once that was accomplished I mounted the VM I needed to fix onto the new OS and started the new VM. After it booted I noticed the 6.3 /boot device was being loaded and not the 6.5 /boot device. I do not know why it does that so to fix that I edited the /etc/fstab and removed all devices associated with the OS I was trying to fix. I needed them to be recognized by the OS in order to fix them but did not need them to mount at boot.
After rebooting I was now in the proper environment to be able to mount the OS and do a chroot. I followed the following the instructions provided here:
Following the guidance there I mounted the / and /boot devices onto new mountpoints, cd’d to the new root mount point, ran all of the mounts listed and entered the chroot /mount/root command and that was it. The root was changed from the the CentOS 6.5 to CentOS 6.3 root. I was in.
To be honest, from a security perspective I was a little surprised how easy it was for me to do this. If I was to do this with a secure system I would definitely use encryption.
First, I need to change the root password so that I could log into the system when booting as a separate VM. Unfortunately I kept getting “passwd: cannot identify you”. Either I needed to make another config or mount setting, possibly the root account I used to mount in was not configured the same as on this device, but rather then go and figure that out, I went into the /etc/passwd file and deleted the X bit from the root account so that root no longer required a password. This way I could log in as root and set a new password.
Now that I had that taken care of I went into the /etc/fstab and set the devices to the proper devices. Using blkid and df I could see what those were set accordingly for /, /boot and swap.
Next I edited the /boot/grub/menus.lst file and changed the device listed in there to the proper device. This is what was causing the kernel panic in the first place.
After all this was done I shutdown the VM and built a new VM using the CentOS 6.3 VM I had just modified. The VM still had some errors while booting but I was finally at the login.
I was in. I could do my maintenance. First thing I did was change the root password.
Next I fixed the swap because it was out of sync with the system. Ran the following commands to fix that:
Swap was now working. Next I decided to replace the boot images. Since the architecture was different from the original I wanted to replace the img files in /boot.
to do that I ran:
mkinitrd /boot/initramfs-$(uname -r).img $(uname -r) –force
to create a new initrd for the current kernel
If you have a specific kernel version build you may need to rebuild that too. To find out what modules are on the system look in /lib/modules. I had an specific kernel version build as well so I ran the mkinitrd command for that as well. To find out builds are on the system look in /lib/modules.
The “–force” is required to overwrite the files already in /boot.
Rebuilding the images fixed some of the boot errors, not all, but now I had a more stable system that would properly boot. I could go in, finish the rest of the maintenance and use the VM for it’s intended purpose.