20230331 release not work with ESXi 7.0U3 VM on Dell R730

[Tokisaki]

after my test the 5.15.98 working good.but the 20230331 told me VFS:unable to mount root fs

[Sebastian-Roth]

@rluzuriaga @George1421 Can anyone confirm the latest inits are causing this issue?

@Tokisaki Sorry for the very long delay. Summer is a pretty busy time. Allow me a question. The version number 5.15.98 is a Linux kernel version but 20230331 is a FOS init version. Can you make checks with old and new kernels and inits to see which one is causing the problem?

[villalvilla]

Hi there! I can confirm that lastest's init.xz release is also giving the same problem.
I have tested with ESX 7.0.3 VMWare's x64 Virtual Machine and giving exactly the same issue.
Bad news continue: We are testing FOS in Alder Lake's 5150 mini pcs, that come with 12Gigas of RAM and NVME hard disk and also giving the same issue.
I am compiling in this moment (it is taking ages) init.xz and bzImage for x64 from master branch, so will let you know how it goes.

I have added verbose 7 in the FOG Client Boot Settings in the FOG Server frontend, so I was able to see the log that I attach to this comment as "FOS Kernel Panic.jpg"

Thanks for your help

[villalvilla]

Update: kernel 5.15.98 isn't working (also). As the Virtual Machine's screen is bigger, I could compile this valuable information:

As you can see, it sais: RAMDISK: Couldn't find valid RAM disk image starting at 0 so the problem in both of the systems is that it is not recognizing the RAMDISK partitions... Might it be because of AHCI drivers? It doesn't make any sense, because the driver used for SDA is SD. If I run an ubuntu live image, it designates SD and AHCI drivers for hard disk.

Any help will be welcomed, because I am really stuck here...

[rluzuriaga]

@villalvilla can you try with the new kernel and init from this release and let me know if this works.

[dpieczynski]

I'm experiencing the same issue with all kernel versions that are > 4.x. The last kernel/initrd pair that works fine for me is the one released with FOG 1.5.9.

I am trying this in a lab with 18 identical Lenovo computers. The only difference that I know of is that some of them were equipped with NVMe drives and some with SATA-connected SSDs. Interestingly, the ones with NVMes work perfectly fine with all kernel versions.

I'm happy to debug this further if more information is needed.

[dpieczynski]

I did a git bisect and the first bad commit is 5c0b983.

[rluzuriaga]

@dpieczynski is your setup exactly the same as the original post?

[dpieczynski]

@rluzuriaga if you mean the "ESXi 7.0U3 VM on Dell R730" part of this issue, then no. I'm imaging physical Lenovo computers. I didn't open another issue because the error I'm getting seems to be exactly the same.

[rluzuriaga]

@dpieczynski Ahh okay, so this is a bit different than the original issue but I'll keep the issue here for consistency.
A few questions...
Have you tried the experimental kernel release 6.6.69?
If you have and you are still getting the same issue, can you share what model/specs of the Lenovo computers? Are you able to image other computers with that kernel?

[dpieczynski]

@rluzuriaga I have tried the experimental kernel and it panics the same way as all the releases since the switch from 4.19.145 to 5.10.12.

The computers I mentioned before are Lenovo V520S-08IKL. The kernel does not panic on the ones that have NVMe drives. Those can be imaged successfully. The only difference I can pinpoint is the drive: NVMe vs SATA SSD and slightly different "machine model" in BIOS (10NM0020PB for the NVMe ones and 10NM006CPB for the SATA ones). I don't know if the machine model signifies any important difference. Maybe it's different because some of the computers come equipped with NVMe? They all have the same processor (Core i5-7400) and RAM amount (8 gigs).

[rluzuriaga]

@dpieczynski Sorry for the long delay... Are you able to load and/or install some Linux distro like Ubuntu or Fedora on one of the computers that doesn't work.
If it loads, can you send us the output of lspci, please?

[dpieczynski]

@rluzuriaga I'm attaching the lspci, lscpu, and dmidecode outputs for both the "good" and "bad" computer.

dmidecode_bad.txt
dmidecode_good.txt
lscpu_bad.txt
lscpu_good.txt
lspci_bad.txt
lspci_good.txt