Trying to run Ubuntu 24.04 from NVME on BPI-R4. I think I might be in over my head, but I’m determined to figure this out

Banana Pi Router design BPI-R4(MT7988)
21

should be the right file.

22

I finally have the time to mess with this again tonight. I read over your comment atleast 10 times so far. It’s amazing how everything you said sounds super easy and makes complete sense… yet I somehow still have no idea how to actually do it.

I think my biggest issue at this point is that I simply don’t understand the terminology, and the actual process taking place. Ive never worked at this low of a level before.

Even Arch Linux sets up the core components like this during install.

But here, I don’t understand what all the different images and files are for, when they are needed, and where they go. I thought I did, but it not booting tells me otherwise lol

23

Im still not having any luck. Heres everything I have done. (I may have forgotten to list a few packages I installed along the way)

  • Flashed OpenWRT to NAND
  • Formatted 16GB USB as exFAT and copied over both: bpi-r4_emmc_bl2.img and bpi-r4_noble_6.12.6-main.img
  • Removed SD Card from the R4 and set the bootstrap switched to NAND.
  • Booted into OpenWRT and installed some needed packages:
    • apk update
    • apk add kmod-usb-core kmod-usb-storage kmod-fs-exfat util-linux util-linux-blkid exfat-fsck exfat-mkfs pv mmc-utils
  • mounted my usb drive in OpenWRT mount /dev/sda1 /mnt/usb
  • echo 0 > /sys/block/mmcblk0boot0/force_ro
  • pv /mnt/usb/bpi-r4_emmc_bl2.img | dd of=/dev/mmcblk0boot0 bs=4096 conv=sync,notrunc
  • pv /mnt/usb/bpi-r4_noble_6.12.6-main.img | dd of=/dev/mmcblk0 bs=4096 conv=sync,notrunc

Now Here I started to hit a snag:

  • mmc partconf 0 1 1 0 gave the following error ERROR: unknown command 'partconf'
  • mmc bootbus 0 0 0 0 gave a similar error ERROR: unknown command 'bootbus'

partconf doesn’t seem to be available in OpenWRT, bootbus is listed in the help file but it doesn’t work either.

  • Tried mmc bootpart enable 1 1 /dev/mmcblk0 from the official getting started guide. This ran without error. However it still didn’t work. I still ended up with the issue below after rebooting using eMMC:
F0: 102B 0000
FA: 1042 0000
FA: 1042 0000 [0200]
F9: 1041 0000
F3: 1006 0033 [0200]
F3: 4001 00E0 [0200]
F3: 4002 0000
F6: 102C 0000
F5: 1026 0000
00: 1005 0000
FA: 1042 0000
FA: 1042 0000 [0200]
F9: 1041 0000
F3: 4001 00E0 [0200]
F3: 4002 0000
F6: 102C 0000
01: 102A 0001
02: 1005 0000
BP: 0200 00C0 [0001]
EC: 0000 0000 [2000]
MK: 0000 0000 [0000]
T0: 0000 00E3 [0101]
System halt!
24

The mmc commands need to be run in uboot,not in openwrt. Not sure if (openwrt) mmc-utils can do this too.

25

But how?

I cant get to uboot when I boot in eMMC mode because I immediately get the System halt! error.

If I try to run the commands in SD Card mode I get:

BPI-R4> mmc partconf 0 1 1 0
PARTITION_CONFIG only exists on eMMC
BPI-R4> mmc bootbus 0 0 0 0
BOOT_BUS_WIDTH only exists on eMMC

And running in NAND mode I get:

MT7988> mmc partconf 0 1 1 0
MT7988> mmc bootbus 0 0 0 0
Set to BOOT_BUS_WIDTH = 0x0, RESET = 0x0, BOOT_MODE = 0x0

And in the end I still get:

F0: 102B 0000
FA: 1042 0000
FA: 1042 0000 [0200]
F9: 1041 0000
F3: 1006 0033 [0200]
F3: 4001 00E0 [0200]
F3: 4002 0000
F6: 102C 0000
F5: 1026 0000
00: 1005 0000
FA: 1042 0000
FA: 1042 0000 [0200]
F9: 1041 0000
F3: 4001 00E0 [0200]
F3: 4002 0000
F6: 102C 0000
01: 102A 0001
02: 1005 0000
BP: 0200 00C0 [0001]
EC: 0000 0000 [2000]
MK: 0000 0000 [0000]
T0: 0000 00E3 [0101]
System halt!
26

Nand is right,but on wonder why it doe not work…you can try reflashing bl2 from uboot.

If it does not work we could verify the bl2.img with hexdump. it should have boot_emmc as string nearly at beginning of file.

oh, seems issue in my build-pipeline…header from emmc bl2 is SDMMC_BOOT

mtk-atf is right when i do manually, i guess it silently fails to build because of not removing build-dir after using the newer atf-branch for 8G mode

$ hexdump -C ~/Downloads/bpi-r4_emmc_bl2.img |head -1
00000000  53 44 4d 4d 43 5f 42 4f  4f 54 00 ff 01 00 00 00  |SDMMC_BOOT......|

vs manual build

$ hexdump -C build/mt7988/release/bl2.img | head -1
00000000  45 4d 4d 43 5f 42 4f 4f  54 00 ff ff 01 00 00 00  |EMMC_BOOT.......|

yes, thats the case:

uploaded fix for ci-pipeline (currently building) and prepare a fix for build.sh in atf-branches to fail when make fails (looks like i did already for the 2025 atf, but not for the old one).

have not tested the new atf across all boards so still have the old one in pipeline…

looks good now:

$ hexdump -C ~/Downloads/bpi-r4_emmc_bl2_new.img |head -1
00000000  45 4d 4d 43 5f 42 4f 4f  54 00 ff ff 01 00 00 00  |EMMC_BOOT.......|
27

Flashed the new uboot image and definitely made some progress now, though still not booting.

Started fresh following the same directions I used before, except using uboot to run the mmc commands.

Ran these in nand boot:

MT7988> mmc partconf 0 1 1 0
MT7988> mmc bootbus 0 0 0 0
Set to BOOT_BUS_WIDTH = 0x0, RESET = 0x0, BOOT_MODE = 0x0
MT7988>

Switched to eMMC boot and this was the output:

F0: 102B 0000
FA: 1042 0000
FA: 1042 0000 [0200]
F9: 1041 0000
F3: 1006 0033 [0200]
F3: 4001 00E0 [0200]
F3: 0000 0000
V0: 0000 0000 [0001]
00: 0000 0000
BP: 0600 0041 [0000]
G0: 1190 0000
EC: 0000 0000 [2000]
MK: 0000 0000 [0000]
T0: 0000 0197 [0101]
Jump to BL

NOTICE:  BL2: v2.9(release):84bda613c emmc
NOTICE:  BL2: Built : 08:25:48, Apr 26 2025
NOTICE:  WDT: Cold boot
NOTICE:  WDT: disabled
NOTICE:  CPU: MT7988
NOTICE:  EMI: Using DDR unknown settings
NOTICE:  EMI: Detected DRAM size: 4096 MB
NOTICE:  EMI: complex R/W mem test passed
NOTICE:  BL2: Booting BL31
NOTICE:  BL31: v2.9(release):7154cf664 sdmmc
NOTICE:  BL31: Built : 09:54:58, Nov  3 2024


U-Boot 2024.10-bpi-gbaedb28ae9a8-dirty (Nov 03 2024 - 09:54:28 +0000)

CPU:   MediaTek MT7988
Model: mt7988-rfb
DRAM:  4 GiB
Core:  47 devices, 18 uclasses, devicetree: separate
MMC:   mmc@11230000: 0
Loading Environment from nowhere... OK
In:    serial@11000000
Out:   serial@11000000
Err:   serial@11000000
=> board_late_init...
bootmedia:emmc
Net:
Warning: ethernet@15100000 (eth0) using random MAC address - d2:e0:65:40:7f:e4
eth0: ethernet@15100000
Hit any key to stop autoboot:  0
Card did not respond to voltage select! : -110
mmc_init: -95, time 16
** Bad device specification mmc 0 **
Couldn't find partition mmc 0:5

  *** U-Boot Boot Menu ***

      1. Enter kernel-name to boot from SD/EMMC.
      2. Boot kernel from TFTP.
      3. Boot from SD/EMMC.
      4. Boot kernel from NVME.
      Exit


  Press UP/DOWN to move, ENTER to select, ESC to quit
## Error: "initrd" not defined
Card did not respond to voltage select! : -110
mmc_init: -95, time 16
sd available
bootconf=#conf-sd
fit=bpi-r4.itb
Card did not respond to voltage select! : -110
mmc_init: -95, time 15
** Bad device specification mmc 0 **
Couldn't find partition mmc 0:5
Can't set block device
BPI-R4>
28

Now you have to replace fip with emmc variant too as you have the sdcard dts in uboot active,but then it should work

Have documented way for r3 in my wiki,but should be same for r4 of course you have to use r4 files and adjust filename.

usb start
fatload usb 0:1 $loadaddr fip_emmc.bin
mmc write ${loadaddr} 0x3400 0x1000
29

Ok, so just to make sure I understand. I downloaded bpi-r4_emmc_fip.bin

In UBoot I run:

usb start
fatload usb 0:1 $loadaddr bpi-r4_emmc_fip.bin
mmc write ${loadaddr} 0x3400 0x1000
30

Right,you can do a

ls usb 0:1

After “usb start” (usb stick needs to be connected before that) to see the fip file before loading to $loadaddr :slight_smile:

31 
MT7988> usb start
starting USB...
Bus xhci@11200000: xhci-mtk xhci@11200000: hcd: 0x0000000011200000, ippc: 0x0000000011203e00
xhci-mtk xhci@11200000: ports disabled mask: u3p-0x0, u2p-0x0
xhci-mtk xhci@11200000: u2p:1, u3p:1
Register 200010f NbrPorts 2
Starting the controller
USB XHCI 1.10
scanning bus xhci@11200000 for devices... 5 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found
MT7988> ls usb 0:1

0 file(s), 0 dir(s)

My USB drive is formatted as exFAT, could that be why no files are found? Or am I missing something?

32

There is an easier way to nvme boot:

Use the emmc image and format nvme rootfs partition only. Copy emmc rootfs files over to nvme rootfs and change kernel cmdline to find the root on partition on nvme.

In a nutshell…

Then everything concerning boot stays on emmc and only rootfs is on nvme.

33

We want first getting boot from emmc working.

And yes uboot does not have exfat driver yet afaik.

Please use fat(32)

My uboot is prepared to have boot and root on nvme.so 2 partitions because kernel and modules are mapped together…it does not make much sense having itb on emmc where modules are only in nvme rootfs but not on emmc rootfs.

34

Ok,

  • used a completely empty fat32 usb drive this time.

  • Booted using NAND mode and entered the uboot console.

  • Ran usb start (Output Below)

MT7988> usb start
starting USB...
Bus xhci@11200000: xhci-mtk xhci@11200000: hcd: 0x0000000011200000, ippc: 0x0000000011203e00
xhci-mtk xhci@11200000: ports disabled mask: u3p-0x0, u2p-0x0
xhci-mtk xhci@11200000: u2p:1, u3p:1
Register 200010f NbrPorts 2
Starting the controller
USB XHCI 1.10
scanning bus xhci@11200000 for devices... 5 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found
  • Ran ls usb to find my drive. Mine ended up being 0:2
MT7988> ls usb 0:1

0 file(s), 0 dir(s)

MT7988> ls usb 1:1
** Bad device specification usb 1 **
Couldn't find partition usb 1:1
MT7988> ls usb 0:0
MT7988> ls usb 0:2
            .fseventsd/
            .Spotlight-V100/
   295461   bpi-r4_emmc_fip.bin
     4096   ._bpi-r4_emmc_fip.bin

2 file(s), 2 dir(s)
  • Ran the fatload and mmc write commands
MT7988> fatload usb 0:2 $loadaddr bpi-r4_emmc_fip.bin
295461 bytes read in 22 ms (12.8 MiB/s)
MT7988> mmc write ${loadaddr} 0x3400 0x1
MMC write: dev # 0, block # 13312, count 1 ... 1 blocks written: OK

After that I went back to the boot menu, powered down, switched back to eMMC boot, and got the following:

F0: 102B 0000WN to move, ENTER to select, ESC to quit
FA: 1042 0000
FA: 1042 0000 [0200]
F9: 1041 0000
F3: 1006 0033 [0200]
F3: 4001 00E0 [0200]
F3: 0000 0000
V0: 0000 0000 [0001]
00: 0000 0000
BP: 0600 0041 [0000]
G0: 1190 0000
EC: 0000 0000 [2000]
MK: 0000 0000 [0000]
T0: 0000 0225 [0101]
Jump to BL

NOTICE:  BL2: v2.9(release):84bda613c emmc
NOTICE:  BL2: Built : 08:25:48, Apr 26 2025
NOTICE:  WDT: Cold boot
NOTICE:  WDT: disabled
NOTICE:  CPU: MT7988
NOTICE:  EMI: Using DDR unknown settings
NOTICE:  EMI: Detected DRAM size: 4096 MB
NOTICE:  EMI: complex R/W mem test passed
NOTICE:  BL2: Booting BL31
NOTICE:  BL2: v2.9(release):84bda613c emmc
NOTICE:  BL2: Built : 08:25:48, Apr 26 2025
NOTICE:  WDT: Cold boot
NOTICE:  CPU: MT7988
NOTICE:  EMI: Using DDR unknown settings

So still no OS booting

35

you flashed only one block, not max of 0x1000

mmc write ${loadaddr} 0x3400 0x1000
36

Ok well I’ve definitely made more progress this time around. It seems like it’s booting, but Im not sure if it’s actually running or if I just don’t know how to get in to it.

Over the serial connection the boot process stops around:

[    5.879285] mtk_soc_eth 15100000.ethernet: generated random MAC address 20:08:02:00:00:00
[    5.887476] mtk_soc_eth 15100000.ethernet: generated random MAC address 20:08:02:00:00:00
[    5.899487] mtk_soc_eth 15100000.ethernet eth0: mediatek frame engine at 0xffffffc084c00000, irq 101
[    5.909151] mtk_soc_eth 15100000.ethernet eth1: mediatek frame engine at 0xffffffc084c00000, irq 101
[    5.918796] mtk_soc_eth 15100000.ethernet eth2: mediatek frame engine at 0xffffffc084c00000, irq 101
[    5.929524] MediaTek MT7988 PHY mt7530-0:00: probe with driver MediaTek MT7988 PHY failed with error -22
[    5.939383] MediaTek MT7988 PHY mt7530-0:01: probe with driver MediaTek MT7988 PHY failed with error -22
[    5.949190] MediaTek MT7988 PHY mt7530-0:02: probe with driver MediaTek MT7988 PHY failed with error -22
[    5.959080] MediaTek MT7988 PHY mt7530-0:03: probe with driver MediaTek MT7988 PHY failed with error -22
[    5.968642] mt7530-mmio 15020000.switch: configuring for fixed/internal link mode
[    5.976131] mt7530-mmio 15020000.switch: Link is Up - 10Gbps/Full - flow control rx/tx
[    5.976814] mt7530-mmio 15020000.switch wan (uninitialized): PHY [mt7530-0:00] driver [Generic PHY] (irq=POLL)
[    5.995116] mt7530-mmio 15020000.switch lan1 (uninitialized): PHY [mt7530-0:01] driver [Generic PHY] (irq=POLL)
[    6.006111] mt7530-mmio 15020000.switch lan2 (uninitialized): PHY [mt7530-0:02] driver [Generic PHY] (irq=POLL)
[    6.017073] mt7530-mmio 15020000.switch lan3 (uninitialized): PHY [mt7530-0:03] driver [Generic PHY] (irq=POLL)
[    6.027440] mtk_soc_eth 15100000.ethernet eth0: entered promiscuous mode
[    6.034167] DSA: tree 0 setup
[    6.037402] clk: Disabling unused clocks
[    6.042228] PM: genpd: Disabling unused power domains
[    6.047541] Waiting for root device /dev/mmcblk0p6...
[    6.118130] usb 2-1: new SuperSpeed USB device number 2 using xhci-mtk
[    6.140563] hub 2-1:1.0: USB hub found
[    6.144461] hub 2-1:1.0: 4 ports detected
[    6.267903] usb 1-1: new high-speed USB device number 2 using xhci-mtk
[    6.420157] hub 1-1:1.0: USB hub found
[    6.424067] hub 1-1:1.0: 5 ports detected
[    7.097901] usb 1-1.5: new high-speed USB device number 3 using xhci-mtk

The full log is a lot longer than this.

I’d check over SSH but it doesnt appear that any of the network ports are actually active.

So progress is being made, but im not sure where to go from here, I double checked GitHub and FW-Web Wiki but I didn’t see anything obvious that I could try.

Looking at the top of the output, this part seems like it’s probably more relevant.

## Error: "initrd" not defined
EXT_CSD[179], PARTITION_CONFIG:
BOOT_ACK: 0x1
BOOT_PARTITION_ENABLE: 0x1 (boot0)
PARTITION_ACCESS: 0x0 (user)
emmc available
bootconf=#conf-base#ov-emmc
fit=bpi-r4.itb
9299076 bytes read in 621 ms (14.3 MiB/s)
## Loading kernel from FIT Image at 46000000 ...
   Using 'conf-base' configuration
   Trying 'kernel-1' kernel subimage
     Description:  Linux Kernel 6.12.6-main
     Type:         Kernel Image
     Compression:  gzip compressed
     Data Start:   0x460000e8
     Data Size:    9254207 Bytes = 8.8 MiB
     Architecture: AArch64
     OS:           Linux
     Load Address: 0x44000000
     Entry Point:  0x44000000
     Hash algo:    sha1
     Hash value:   a987131870d87c8201df5861373cfed59469fac4
   Verifying Hash Integrity ... sha1+ OK
## Loading fdt from FIT Image at 46000000 ...
   Using 'conf-base' configuration
   Trying 'fdt-base' fdt subimage
     Description:  Flattened Device Tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x468d3724
     Data Size:    36744 Bytes = 35.9 KiB
     Architecture: AArch64
     Load Address: 0x47000000
     Hash algo:    sha1
     Hash value:   30a48a4b032dbbb65ab6abe7e12ee5b755a2c100
   Verifying Hash Integrity ... sha1+ OK
   Loading fdt from 0x468d3724 to 0x47000000
## Loading fdt from FIT Image at 46000000 ...
Could not find configuration node
load of <NULL> failed
   Booting using the fdt blob at 0x47000000
Working FDT set to 47000000
   Uncompressing Kernel Image to 44000000
   Loading Device Tree to 00000000fe7ef000, end 00000000fe7faf87 ... OK
Working FDT set to fe7ef000

Starting kernel ...
37

Resulting mmc dt node is missing properties (and not enabled) and cannot find device.

I guess your kernel does not have the ov-emmc part i had to add for 2.5g r4. I had to change uboot bootconf to use same overlays with 2 different base dt.

Try replace your kernel with actual 6.12

on my r3 i did these changes in uEnv.txt for nvme default boot:

bootnvme=run usenvme; if test "$device" = "nvme"; then run loadenv; printenv; run newboot;fi

bootmenu_3=4. Boot kernel from NVME.=run bootnvme
bootmenu_default=3

the loadenv here is to load uEnv.txt on nvme boot partition first

cat /mnt/nvme_boot/uEnv.txt
root=/dev/nvme0n1p2 rootwait
fit=....itb
38

I not going to lie, Im completely lost on that one. Im not sure where to place the .itb file, or how to even find the uEnv.txt. Plus at this point I still haven’t setup the nvme, im struggling just to get the eMMC boot working.

39

On emmc you should have a BPI-BOOT (where the uEnv.txt and itb should be placed) and a BPI-ROOT (where the debian/ubuntu rootfs is located and modules need to be placed) partition after you flashed the image to emmc userpartition.

See here how to install kernel+modules from prebuilt archives:

The files are in same repo as releases. For emmc the easiest way is unpack itb to usb stick,start from there (run useusb;run newboot) and then copy all like above. Or using initrd.

40 
## Error: "initrd" not defined
EXT_CSD[179], PARTITION_CONFIG:
BOOT_ACK: 0x1
BOOT_PARTITION_ENABLE: 0x1 (boot0)
PARTITION_ACCESS: 0x0 (user)
emmc available
bootconf=#conf-base#ov-emmc
fit=bpi-r4.itb
Can't set block device
BPI-R4>

Not sure where I went wrong.