Today I have a tale of mystery, of horror, and of hope. The allure of a newer kernel and packages was too much to resist, so I found myself upgrading to Fedora 30. All the packages had downloaded, all that was left was to let DNF reboot the machine and install all the new packages. I started the process and meandered off to find a cup of coffee: black, and darker than the stain this line of work leaves on the soul. After enough time had elapsed, I returned, expecting the warming light of a newly upgraded desktop. Instead, all that greeted me was the harsh darkness of a grub command line. Something was amiss, and it was bad.
(An aside to the reader, I had this experience on two different machines, stemming from two different root problems. One was a wayward setting, and the other an unusual permissions problem.)
How does the fledgling Linux sysadmin recover from such a problem? The grub command line is an inscrutable mystery to the uninitiated, but once you understand the basics, it’s not terribly difficult to boot your system and try to restore the normal boot process. This depends on what has broken, of course. If the disk containing your root partition has crashed, then sorry, this article won’t help.
In order to get a system booting, what exactly needs to happen? How does booting Linux work, even? Two components need to be loaded into memory: the kernel, and the initramfs. Once these two elements are loaded into memory, grub performs a jump into the kernel code, which takes over and finishes the machine’s boot. There is one more important detail that we care about — the kernel needs to know where to find the root partition. This is typically part of the kernel parameters, specified on the kernel boot line.
When working with an unfamiliar shell, the help command is a good starting point. grub runs in a very limited environment, and running the help command scrolls most of the text off the screen. There is an environment variable that helps out here, enabling output paging:set pager=1.
Finding What You’re Looking For
ls is your friend. Don’t know which drive is which? ls to the rescue. grub uses a unique nomenclature for accessing partitions. You might see entries like (HD0,0) or (hd0,msdos1). A modern grub will even let you list the files and folders contained in that partition using a command like ls (hd0,msdos1)/.
We want to start by figuring out which partition stores the kernel and initrd files. Those files might be in a boot folder, or just in one of the partitions. The kernel is generally named vmlinuz-kernel_version.architecture so for example: vmlinuz-5.3.7-200.fc30.x86_64. The initrd we need will match the kernel’s version. Something like initramfs-5.3.7-200.fc30.x86_64.img.
The last needed bit of information, the root filesystem location, can be a bit trickier to find. While searching through partitions, you may find one with a root filesystem layout, containing boot, bin, etc, home, etc. You can likely figure out what the kernel will call the partition based on the name in grub.
hd0 is probably sda, hd1 is probably sdb. The second half of grub's name tells you which partition it is, so (hd0,msdos1)is likely sda1.
Putting It Together
To actually boot, we issue three commands in grub. The first command sets the kernel image and any kernel boot options. The one required option is setting the root location:linux (hd0,msdos1)/boot/vmlinuz-4.19.0-6-amd64 root=/dev/sda1
Next we set the initrd option:initrd (hd0,msdos1)/boot/initrd.img-4.19.0-6-amd64
Once those options are set, we can tell grub to try to boot the kernel. It a simple command:boot
Assuming we set the right options, and the system isn’t otherwise terribly broken, that should boot your machine back into normalcy. Time to troubleshoot what caused grub to go off the rails to begin with. That however, is for another time.
Since we’re here, there are a few other tricks worth knowing about grub and booting. The most useful is probably single user mode, which is enabled by adding a “1” to the boot options.
linux (hd0,msdos1)/boot/vmlinuz-4.19.0-6-amd64 root=/dev/sda1 1
On some distributions, this even bypasses the need to know a root password, which is useful if you find yourself locked out of a system. Many modern systems still require logging in as root to proceed. Still, single-user mode is helpful for troubleshooting other boot and system problems.
One more trick to have up your sleeve is the ability to blacklist a driver. Adding blacklist amdgpu, for example, would prevent the amdgpu driver from loading at all, regardless of the hardware present. If a buggy or misconfigured driver is causing a crash during boot, blacklisting it will likely let you successfully boot.
Hopefully this is enough to give you the edge next time you’re debugging a Linux boot problem, and adds a couple tools to your repertoire. Happy hacking.