In some environments, virtual machines (VMs) may boot using an older or incorrect kernel
version instead of the intended one. This results in network interfaces not initializing properly,
which leads to all SRM VMs becoming unreachable on the network. This article describes the
symptoms, root cause, and resolution, including the required supplementary steps.
Symptoms
After upgrading from SRM 6.x.x.x to higher SRM version, the following symptoms are observed
upon powering on the vApp:
• No network interfaces available — all VM network adapters are down except for the loopback
interface.
• Excecuting ip link shows no interface other than lo.
• Excecuting uname -a reveals the system is running the old kernel instead of the new kernel.
• Excecuting
modprobe vmxnet3 fails because the kernel module path /lib/modules/<old kernel version> does not exist for the old kernel.
• Excecuting grub2-once --list and ls -l /boot show a mismatch: GRUB still points to the old
/dev/sda boot location.
• All SRM VMs (FE, PBE, ABE, and Collectors) are unreachable on the network after snapshot
restore and power-on.
Cause
During the upgrade from 6.x.x.x to higher SRM version, the latest kernel was updated. These
kernel changes were applied to the /boot partition, but recent changes in the kernel packages did
not automatically run grub-install and grub-mkconfig. Due to recent kernel changes, if /boot is
on a secondary disk partition, we must manually run grub-install and grub-mkconfig. This is why
the VM failed to boot with the latest kernel.
The following scenarios can trigger this issue:
- With the boot migration script followed by a reboot.
- With the boot migration script followed by upgrade to the latest SRM version.
- Standalone SRM upgrade process followed by a reboot.
Resolution
To fix or overcome this issue, follow the below Steps to resolve the old kernel boot issue:
Step 1: Log in to the affected VM via the vSphere console and run the following three
commands in sequence:
Step 2: Reboot the VM:
reboot
Step 3: After rebooting, verify the expected SRM kernel version:
uname -r
Expected output: the new kernel version.
Step 4: Confirm that network interfaces are restored:
ip link
Once the latest kernel version is confirmed and network interfaces are restored, please
follow the KB below. (if the /boot is in secondary disk eg: sdb, sdc ..)
Affected Products:
SRM