Boot Debian from NVMe

What is the output of

run usenvme

In uboot console after booting from nand?

Nand is the most complex setup…why not using mmc as boot device? There you can use uenv.txt to configure uboot (i try to do the same in ubifs,but setup is more difficult than only flashing an image).

The default bootmenu entry is always mmc and needs to be changed (via mmc uenv.txt) to use nvme bootentry.

The output is:

BPI-R4> run usenvme
** Bad device specification nvme 0 **
Couldn't find partition nvme 0:1
nvme partition not found

Ao far I haven’t found any description how to get u-boot onto eMMC.

Thats really strange…have not changed anything for nvme except adding the menuentry by default

For emmc it is basicly same as r3

https://wiki.fw-web.de/doku.php?id=en:bpi-r3:uboot#writing_emmc

Easiest way is to start from nand (after flashed from sdcard like you did) and booting a kernel+initrd from usb. Then flash emmc bl2 to boot0 and sdmmc image to userpart.

Good morning (o;

Where do I find this rootfs.cpio.zst? The google drive link there isn’t valid anymore.

Or can I just use the initrd.gz image from official Debian?

Debians initrd is the installer.

I hope this link works

https://drive.google.com/drive/mobile/folders/1WLWAR1FC-rF4n2SgFecBlU1ym_XKqAR_/15Y5Y3NAOwg_IMmN3k6hdb7pAQj9oTVTl/1_UtkNl0S44_Xy46NAEn1KoFDSOkQwAsb

Okay…booting up went fine (o;

Do I have to do the sgdisk stuff as mentioned on your page? And if…what is ${LDEV}?

I see following partitions on mmcblk0:

brw-rw----    1 root     disk      179,   0 Jul 12 06:46 /dev/mmcblk0
brw-rw----    1 root     disk      179,   8 Jul 12 06:46 /dev/mmcblk0boot0
brw-rw----    1 root     disk      179,  16 Jul 12 06:46 /dev/mmcblk0boot1
crw-------    1 root     root      244,   0 Jul 12 06:46 /dev/mmcblk0rpmb

So just do a:

dd if=bpi-r4_emmc_bl2.img of=/dev/mmcblk0boot0
dd if=bpi-r4_sdmmc.img of=/dev/mmcblk0rpmb

the sgdisk fix is because image can only contain main GPT at beginning the disk but not the backup GPT at the end of disk (because image creation process cannot know target disk size and 16GB+ image is not a good idea).

the uboot sdmmc.img has no kernel and rootfs, here you have to use the full image ceated by my bpi-router-images repo (these images have kernelversion and userspace name like bookworm in filename). the uboot-file is only a base image with the bootchain which is taken by images repo to add kernel and userspace.

second the image has to be flashed to /dev/mmcblk0 which is the emmc user-part (not sure what the rpmb partition is - not used till now).

Hmm…but actually I want to load kernel and rootfs from NVMe via MMC…from NAND SPI it didn’t work and always took /boot from SD…

The 8GB MMC is just way too small and slow for my needs.

you can define a uEnv.txt on the emmc BPI-Boot partition to point to nvme (see above):

first 2 lines should not be needed only setting the bootmenu default or maybe setting the nvme-vars

Hmm…somehow the

gunzip -c bpi-r4_bookworm_6.12.23-main.img.gz | dd of=/dev/mmcblk0

didn’t work…can’t boot from MMC…just gives me a system halt… I can boot though via NAND SPI and then choose MMC which in turn loads the rootfs from NVMe and /boot from MMC:

root@bpi-r4:~# lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mtdblock0     31:0    0     2M  1 disk 
mtdblock1     31:1    0   126M  0 disk 
mmcblk0      179:0    0   7.3G  0 disk 
├─mmcblk0p1  179:1    0     4M  0 part 
├─mmcblk0p2  179:2    0   512K  0 part 
├─mmcblk0p3  179:3    0     2M  0 part 
├─mmcblk0p4  179:4    0     2M  0 part 
├─mmcblk0p5  179:5    0   100M  0 part /boot
└─mmcblk0p6  179:6    0   6.4G  0 part 
mmcblk0boot0 179:8    0     4M  1 disk 
mmcblk0boot1 179:16   0     4M  1 disk 
nvme0n1      259:0    0 238.5G  0 disk 
├─nvme0n1p1  259:1    0     4M  0 part 
├─nvme0n1p2  259:2    0   512K  0 part 
├─nvme0n1p3  259:3    0     2M  0 part 
├─nvme0n1p4  259:4    0     2M  0 part 
├─nvme0n1p5  259:5    0   100M  0 part 
└─nvme0n1p6  259:6    0 238.4G  0 part /

drives me nuts though I could live with that setup if I remember all steps I’ve done (o;

Anyway…gotta prepare the pizza dough now :wink:

Where do you get this? Do you see uboot?

If not i guess partconf is not set right,so boot nand till uboot and do this in uboot console:

mmc partconf 0 1 1 0

As i said,if /boot is from mmc you have to adjust /etc/fstab on nvme if you sure rootfs is loaded from nvme (look in bootlog for bootargs and then mount command).

There are of course many ways to achieve booting Debian from NVME.

It’s really what you’d prefer, Frank’s method also has advantages compared to the below.

Personally I would like to use extlinux.conf booting up linux from NAND. I’ve described a different bootchain here that does that:

https://forum.banana-pi.org/t/bpi-r3-bpi-r3-mini-bpi-r4-nand-distro-boot-linux-recovery/23758?u=ericwoud

After booting up from nand there are several ways to install Debian, but also, the tool bpir-build has an Ubuntu option to get Ubuntu on NVME.

The advantage here is that there is no need to work with U-Boot commands, there is always a linux (initrd) available. From there it is quite easy to install any distro.

I’ve tested this on R3, in my R4 I do not have an nvme installed yet. U-Boot should be able to use it on R4 also.

Edit:

Confirmed, boots on R4 also.

Modifying uboot environment variables is a solid approach

Hi all,

I was able to build my SPI FLASH Firmware U-Boot using EFI and then I’ve install Debian 13 Trixie on SSD NVMe. U-Boot EFI detects the EFI partition on NVMe (of course, U-Boot needs to be last version or with PCIe patches) and then I’m booting GRUB EFI. In GRUB I’ve added devicetree /boot/… to load the DTB.

Is working very nice.

I’m working o a BASH script which builds the firmware and the rootfs for NVMe, but is not finish yet.

Which dtb? There are some overlays which are probed in the fit approach. Thats why i think extlinux and efi is not the best way…you are fixed to 1 config and if e.g. on R3 the other spi/mmc device is active it does not work.

Which dtb? There are some overlays which are probed in the fit approach. Thats why i think extlinux and efi is not the best way…you are fixed to 1 config and if e.g. on R3 the other spi/mmc device is active it does not work.

Btw. Just a note for others: Debian trixie is not yet released (date 2025-08-09), it is freezed and can be tested,but it is not yet defined as stable.

https://wiki.debian.org/DebianTrixie

Even is not relaaed yet, is in testing, I’ve created a rootfs.

For DTB I’m using the one based on eMMC.

Now I’m testing dtoverlay tool from RaspberryPi to build a new DTB from base DTB + DTBO. The apply to GRUB “devicetree” and reboot de device.

Example of DTB + DTBO merged by dtoverlay:

dtmerge
dtoverlay
dtparam
libdtovl.a
mt7988a-bananapi-bpi-r4-emmc.dtb
mt7988a-bananapi-bpi-r4-emmc.dtbo
mt7988a-bananapi-bpi-r4-sd.dtb
mt7988a-bananapi-bpi-r4-sd.dtbo
mt7988a-bananapi-bpi-r4.dtb

I used this command:

# ./dtmerge ./mt7988a-bananapi-bpi-r4.dtb mt7988a-bananapi-bpi-r4-emmc.dtb ./mt7988a-bananapi-bpi-r4-emmc.dtbo

# ./dtmerge ./mt7988a-bananapi-bpi-r4.dtb mt7988a-bananapi-bpi-r4-sd.dtb ./mt7988a-bananapi-bpi-r4-sd.dtbo

# ls -1 /boot/dtb-6.16.0-rc1-EasyOS-bpi-r4/mediatek/
mt7986a-bananapi-bpi-r3-emmc.dtbo
mt7986a-bananapi-bpi-r3-mini.dtb
mt7986a-bananapi-bpi-r3-nand.dtbo
mt7986a-bananapi-bpi-r3-nor.dtbo
mt7986a-bananapi-bpi-r3-sata.dtbo
mt7986a-bananapi-bpi-r3-sd.dtbo
mt7986a-bananapi-bpi-r3.dtb
mt7988a-bananapi-bpi-r4-2g5.dtb
mt7988a-bananapi-bpi-r4-emmc.dtb
mt7988a-bananapi-bpi-r4-emmc.dtbo
mt7988a-bananapi-bpi-r4-sd.dtb
mt7988a-bananapi-bpi-r4-sd.dtbo
mt7988a-bananapi-bpi-r4.dtb

And here is the GRUB config:

menuentry 'Debian GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-05329a7f-410a-419b-840b-2fde100897cc' {
        load_video
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod ext2
        search --no-floppy --fs-uuid --set=root 05329a7f-410a-419b-840b-2fde100897cc
        echo       'Loading Linux 6.16.0-rc1-EasyOS-bpi-r4 ...'
        linux      /boot/vmlinux-6.16.0-rc1-EasyOS-bpi-r4 root=UUID=05329a7f-410a-419b-840b-2fde100897cc ro console=ttyS0,115200n1 earlycon=uart8250,mmio32,0x11000000
        echo       'Loading initial ramdisk ...'
        initrd     /boot/initrd.img-6.16.0-rc1-EasyOS-bpi-r4
        echo       'Loading devicetree ...'
        devicetree /boot/dtb-6.16.0-rc1-EasyOS-bpi-r4/mediatek/mt7988a-bananapi-bpi-r4-emmc.dtb
}

And this is the OS:

# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
NAME="Debian GNU/Linux"
VERSION_ID="13"
VERSION="13 (trixie)"
VERSION_CODENAME=trixie
DEBIAN_VERSION_FULL=13.0
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

And here is the OS booted with the merged DTBO to DTB with eMMC:

# ls -1 /dev/mmcblk0*
/dev/mmcblk0
/dev/mmcblk0boot0
/dev/mmcblk0boot1
/dev/mmcblk0p1
/dev/mmcblk0p2
/dev/mmcblk0p3
/dev/mmcblk0p4
/dev/mmcblk0p5
/dev/mmcblk0rpmb

I think is enough flexible also the solution U-Boot EFI + GRUB EFI.

Source for dtoverlay can be found here:

I am using the standard Linux commands

fdtoverlay
fdtput
1 Like

Good tip! Thanks. I will give it a try.

//LE: is working perfect with fdtoverlay. Thanks for this.