Is it possible to make NVME as boot instead of sdcard or NAND or EMMC?

Banana Pi Router design BPI-R4/BPI-R4 Pro(MT7988)
1

Hello Is it possible to burn the image on nvme and make it the default boot instead using sdcard or EMMC ? If yes how to do that ?

Thank you

2

No you cannot directly boot from nvme…it is only possible booting from api or mmc and acessing nvme from uboot stage to load kernel and userspace from nvme

3

The problem is I have file .img I did following commands from NAND to EMMC

echo 0 > /sys/block/mmcblk0boot0/force_ro

dd if=file.img of=/dev/mmcblk0
 mmc bootpart enable 1 1 /dev/mmcblk0
sync

But after reboot and change to EMMC bootstrap, it not wrote and system halt

4

I do not see that you write bl2 to boot0 partition

5

Because debian11, Does bl2 for openwrt only or even for debian11 ?

6

how to change uboot TFTP server IP address to locate the debian11 .img to install it directly to EMMC ?

7

For my images (debian bookworm,ubuntu jammy/noble) from gdrive (automatic images) you can get uboot/atf files from my uboot github repo in releases

Not sure if they are compatible to bpi images,but you can try. Openwrt uses fip in ubi,i still use fixed offset

My uboot supports usb and nvme :slight_smile: and you could override serverip environment variable (and if net differs ipaddr/netmask).but full image could be bit large for using tftp.

8

What I’ve done is baked a u-boot image with the following default command into my NAND:

bootcmd=pci enum; nvme scan; nvme info; if fatload nvme 0:1 $loadaddr boot.scr; then source $loadaddr; fi

That way I can just have a u-boot script on a small fat (could even be ext4 or something, u-boot can read that) partition, and can update the kernel without needing to ever touch the NAND.

What I’d still like to add is a way to detect if this is a cold boot or a reboot. To load a different kernel on each, to have a fallback if a new kernel does not come up after a reboot.

9

There was another thread for this topic. I’d really like to find some people to work on this together.

I’d like to take U-Boot source and make some modifications so we have a build that creates a NAND image that we can put on the R4 and it will just work.

Would you be interested?

Could even make it so that we have a failsafe in the NAND with double firmware. Like a micro loader and then either load ubootA or ubootB.

And inside U-Boot we have USB boot, NVME boot and the usual SD and MMC boot. Or network. And the web failsafe from the mediatek u-boot fork. That would be absolutely awesome. Because with the web failsafe, you don’t even need a serial console.

10

It is very good idea I would help you but I don’t have experience with uboot and hardware upgrades, I am new with this side

But please let me know if any progress happens because I am interested such like this idea

11

Hi,

Good news — NVMe boot is fully working on BPI-R4, including sysupgrade from both CLI and LuCI.

Everything is documented and ready to use — no Linux machine needed, installation runs entirely via GitHub Actions:

Happy to answer any questions!

2 Likes
12

That’s very interesting — thanks for sharing this. I’m currently working through the same topic (NVMe boot / NVMe root) and just recently recovered from a soft-brick after manually flashing the wrong memory region — so I’m trying to be a bit more cautious now. At the moment I’m running: SD boot → Debian Trixie (thanks @frank-w for the image). What I’m consistently running into is that any attempt to switch the root filesystem to NVMe doesn’t persist. From what I can tell, the FIT image (bpi-r4.itb) hardcodes the kernel command line (specifically root=/dev/mmcblk0p6), which effectively overrides uEnv.txt and even manual U-Boot bootargs. So even though the NVMe root is fully prepared and functional, the system always falls back to SD root. That’s why your statement that “NVMe boot is fully working” caught my attention. Before I go further down the rabbit hole (patching FIT images or rebuilding boot chains), I’d like to understand: Does your approach bypass the FIT image entirely or modify it? Is this still using the standard U-Boot from Banana Pi / frank-w images, or a custom build? How is early NVMe initialization handled (since stock U-Boot on my side doesn’t properly support nvme scan)? Does your method preserve a fallback (e.g. NAND / UBI boot), or does it fully replace the boot path? Also — just to be transparent — I’m still relatively new to this platform, so if I’m misunderstanding something fundamental about the boot flow, I’d appreciate being corrected. My goal is fairly simple: a stable setup with SD (or NAND) as fallback and NVMe as primary root (or ideally full NVMe boot later), running a few Docker services alongside networking. Thanks again for sharing — I’ll definitely take a closer look at your repo.

13

After digging a bit deeper into your repo, I think I’m starting to understand what you’re doing. It looks like you’re not working around the existing boot flow, but actually introducing a proper NVMe boot path at the U-Boot level (env + dedicated boot target). That’s a completely different approach than what I’ve been trying so far. That raises a couple of questions for me:

Would you say the key part is mainly the U-Boot env logic (boot_nvme) or the underlying patches enabling early NVMe init?

How tightly coupled is your setup to OpenWrt — do you think the same boot approach would work with a different rootfs (e.g. Debian), assuming the kernel supports it?

Does your NVMe boot path rely on a specific FIT/DT setup, or is it mostly handled before that stage?

Also — being honest here — I’m at the point where I’m comfortable experimenting, but I’d rather not randomly poke at bootloader-level stuff without understanding the implications :sweat_smile:

Also, If you think parts of your approach are reusable outside OpenWrt, I’d be very interested in trying that out and reporting back.

And if that’s completely off-track — feel free to say so, I’m still learning how this board is meant to be used. Really appreciate you sharing this, it definitely helped me reframe the problem.

14

I have an ArchLinuxARM/Ubuntu installscript, that comes with a toolbox command that helps setup such boot chain: nvme rootfs and U-Boot with backup initrd on nand

Read the readme on my GitHub page first.

I have a bit different approach in U-Boot: using extlinux.conf.

You can use my image only to setup Nand with U-Boot and initrd. You can also choose to use this Ubuntu/Debian/ArchLinuxARM on your nvme, or use your own.

1 Like
15

i have r3 bootable from nvme without issues…the dts root= is only a fallback when bootloader (here uboot) does not define its own. i guess you do not overwrite nvme-root.

16

Hi Murat,

Thanks for the detailed questions — happy to clarify.

Our approach is OpenWrt-specific and works as follows:

The key is a custom U-Boot env variable nvme_boot=1 which triggers a dedicated NVMe boot path in U-Boot. We don’t bypass or modify the FIT image — we use the standard OpenWrt sysupgrade FIT, but with a patched fit.sh that handles NVMe correctly (writes kernel to p1/ext4 and rootfs to p2/raw).

To answer your questions directly:

  • We use a custom U-Boot build (based on frank-w’s work) with NVMe/PCIe init enabled
  • Early NVMe init is handled at U-Boot stage via pci enum; nvme scan
  • NAND remains fully intact as fallback — nvme_boot=1 is just an env variable, deleting it returns to NAND boot
  • The approach is OpenWrt-specific — porting to Debian would require adapting the boot scripts and partition layout

We’re currently finishing p3 ext4 partition support (remaining disk space for Docker/data), after which the full setup will be documented.

Feel free to ask if anything is unclear!

1 Like
17

Hi Eric,

Thanks for mentioning your approach — extlinux.conf is an elegant solution and your toolbox looks really useful for people who want Debian/Ubuntu/Arch on NVMe.

Our setup is OpenWrt-specific so the approaches complement each other nicely — different tools for different use cases.

18

HI Wozi,

1st: :rofl: :rofl: :rofl: 2nd: I was just writing my answer to @frank-w when I saw your reply and my jaw dropped and reality hit → hard. So, in essence “me noob|me have not working uboot” and I was very deep in diagnosing your answer ;)… thus I want to, at least, post my answer still. Even though you literally rugpulled it from under me … :wink: This would have been my answer…

Hi @frank-w , Quick update after digging deeper into this.

I tried your suggestion and set everything directly in U-Boot:

setenv device nvme
setenv partition 0:1
setenv root /dev/nvme0n1p2
run newboot

But it fails already at the storage layer:

Bad device specification nvme 0
Couldn't find partition nvme 0:1

So it seems the issue is not bootargs or DTS fallback, but that U-Boot cannot actually access NVMe in my current setup.

For reference:

  • NVMe works perfectly in Linux
  • /dev/nvme0n1p1 contains bpi-r4.itb , dtb, extlinux, etc.
  • both SSDs are connected via PCIe (not USB)
ls -l /dev/disk/by-path/
...
platform-11280000.pcie-pci-0003:01:00.0-nvme-1 -> ../../nvme0n1
platform-11290000.pcie-pci-0002:01:00.0-nvme-1 -> ../../nvme1n1

However in U-Boot:

pci enum
nvme scan
nvme info

→ no output / no devices found

So it looks like NVMe is simply not initialized or accessible at bootloader stage.

Does this suggest I might be missing the correct ATF/U-Boot combination, or that NVMe support in this build is incomplete on R4 Pro?

Just want to make sure I’m debugging the right layer now.

19

Hi Murat :smile:

Yes, that’s exactly the root cause — stock U-Boot on BPI-R4 doesn’t initialize PCIe/NVMe properly. Our build includes the necessary patches for that.

Two key things make it work:

  1. PCIe Gen3 — our U-Boot/ATF build includes PCIe Gen3 support which is required for NVMe to be visible at bootloader stage. Without it, pci enum; nvme scan returns nothing.
  2. Custom DTS — we use a dedicated NVMe DTS (mt7988a-bananapi-bpi-r4-nvme.dtso) that properly maps the PCIe/NVMe path.

Note: our setup targets BPI-R4 (MT7988A), not BPI-R4 Pro — PCIe mapping may differ on Pro, worth checking with @frank-w.