grub2 – Robin on Linux

A successful rescue for a remote server

After installed CUDA-9.2 on a remote server, I found that the system can’t load nvidia.ko (kernel module) with dmesg:

Unknown symbol __stack_chk_fail (err 0)

The reason is the current kernel running on my system has turned on the CONFIG_CC_STACKPROTECTOR compiler option. Therefore I change the default entry of grub2 and reboot the server, for entering a new kernel without this option.
But unfortunately, the server never start up again. All my code and data (includes my colleague’s code and data) are on this server, so we get a little nervous then.
Since the server is in a remote datacenter, we can’t just plugin in a keyboard and a screen to debug. Thus I use the out-of-bound system to reboot this server to diskless-mode. After entering this mode, I mount the disk for ‘/boot’ directory:

mount /dev/sda1 /mnt/

and manually change the ‘/boot/grub2/grubenv’ like this (the ‘save_entry’ is 2 before):

# GRUB Environment Block
saved_entry=1
#########################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

Then reboot the server again. This time, the server started up smoothly now. All our code and data is untainted.

Upgrade to kernel-4.4.1 on CentOS 7

After I compiled and installed kernel-4.4.1 (from kernel.org) on my CentOS 7, I reboot the machine. But it can’t boot up correctly.
Using

/usr/lib/dracut/skipcpio /boot/initramfs-4.4.1-XXX.img |zcat| cpio -id

to extract the content in initramfs and check them, I found out the ‘mpt2sas’ kernel driver had not been added into initramfs so /boot partition could not be loaded.
Seems this problem is common. Because changing dracut source code or configure file on all servers is not viable, I chose to add command in my kernel rpm spec file:

dracut --add-drivers "raid_class megaraid_sas dm-mod nvme mpt3sas scsi_transport_sas xfs" --kver %{version}-%{release}.%{_target_cpu} --force

This will add drivers to the corresponding initramfs file.
But the kernel could not boot up either. This time, I found that the command line in GRUB2 is like:

root=LABEL=XXX

Looks we should change it to UUID. Add another command in kernel rpm spec file:

new-kernel-pkg --package %{name} --kernel-args="root=`grep -o -P "(?<=root=)\S+" /proc/cmdline`" --update %{version}-%{release}.%{_target_cpu}

This will get UUID of boot disk from /proc/cmdline and give it to GRUB2 configure file.
Now, the kernel-4.4.1 boot up correctly on CentOS 7.