Ubuntu 24.04 LTS on Banana RPI-R4 Pro 8X

[Michal_Zygmunt]

6.18-main branch or 6.18-r4pro branch?

[Michal_Zygmunt]

Assume 6.18-main, as I noticed this commit as last one: net: phy: as21: fix phy-id overwrite for all phys (breaking leds on m

[frank-w]

The 6.18-main…endusers should stay at main branches,mainly LTS (6.12,6.18,for R4pro and r4lite 6.18).

And yes this commit fixes the phy led issue

[ericwoud (Eric W.)]

See:

[BPI-R4] Imagebuilder R4 ArchlinuxArm

Added more boards to the script. I do not own this hardware (yet), so I can only partly check it for bugs. Added R4 Lite, R4 Pro 4E and R4 Pro 8X. This is in very early stage. mt7987 (R4 Lite) ATF and U-Boot not tested at all. R4 pro is mainly based on standard R4. Device tree overlays borrowed from R4, also the wired network setup, so this may need some manual changes to function properly. Expect many changes and some bugs…

[Michal_Zygmunt]

Hey Frank,

How do you exactly compile the 8X variant of r4-pro in the BPI-Router-Linux?

I am using 6.18-main branch.

build.conf:

uploaduser=$USER
uploadserver=r3
uploaddir=/var/lib/tftp
#uploaduser=root
#uploadserver=192.168.0.11
#uploaddir=/boot/bananapi/bpi-r2/linux

builddir=../build
ramdisksize=8G
#numproc=8

#board=bpi-r2
#board=bpi-r64
#board=bpi-r2pro
#board=bpi-r3
#board=bpi-r3mini
#board=bpi-r4
board=bpi-r4pro
#board=bpi-r4lite

#r64 with rtl8367
#boardversion=v0.1
#r2pro with rtl8367
#boardversion=v00

#mainline uboot for r64 (old ATF) needs 64bit uImage
#uimagearch=arm64

#grep whitelist filter for adding modules to initramfs
ownmodules='mt76\|bluetooth'

I am getting this:

cp: cannot stat '/work/bpi/build/arch/arm64/boot/dts/mediatek/mt7988a-bananapi-bpi-r4-pro.dtb': No such file or directory

I can only see those files there:

# ls -al /work/bpi/build/arch/arm64/boot/dts/mediatek/mt7988a-bananapi-bpi-r4-pro*.dtb
-rw-r--r-- 1 root root 35067 Feb 16 09:07 /work/bpi/build/arch/arm64/boot/dts/mediatek/mt7988a-bananapi-bpi-r4-pro-4e.dtb
-rw-r--r-- 1 root root 38422 Feb 16 09:07 /work/bpi/build/arch/arm64/boot/dts/mediatek/mt7988a-bananapi-bpi-r4-pro-8x.dtb
root@0b31aec1f51d:/work/bpi/BPI-Router-Linux#

Same later during package creation:

cp: cannot stat './bpi-r4.dtb': No such file or directory

Where the file on disk is actually called:

ls -al bpi-r4*.dtb
-rw-r--r-- 1 root root 38422 Feb 16 09:26 bpi-r4pro.dtb

I can ‘hack it up’ locally so it builds, but wondering if there is any official way to do this via build.conf.

Am I missing some flag there?

Where do you specify the 8x in the build.conf?

[Michal_Zygmunt]

also - side question - can you enable KEXEC by default in your build?

I am planning to use the following setup:

1. /boot on eMMC that has custom initramfw + custom boot.cfg
2. that initramfs has custom /init script with options of different kernels, different partitions etc
3. that /init in initramfs loads the kernel from specific partition and use kexec to move execution to this kernel

I am planning to use two-kernel boot approach to have my boot loaded doing some more advanced stuff (ex. checking from network which kernel to boot, and then booting either full ubuntu or openwrt etc).

That’s why kexec would be beneficial to have in the ubuntu kernel that is being used with uboot.

[frank-w]

You can just build with board=bpi-r4 in build.conf

The bpi-r4pro there is only an alias to set names for e.g. upload function,open dts source and such.seems there is a bug. Basicly the r4 defconfig is used for all R4 boards with all dts(o). Uboot chooses right config based on isr4pro var and i have only 8x support.

@Michal_Zygmunt

I am planning to use two-kernel boot approach to have my boot loaded doing some more advanced stuff (ex. checking from network which kernel to boot, and then booting either full ubuntu or openwrt etc).

You want to boot openwrt from running ubuntu? Or how can i imagine this? You can check this in uboot and either boot openwrt initramfs/recovery or ubuntu.

[ericwoud (Eric W.)]

I have tried kexec on the R3 before, it has never been done before and the drivers and/or hardware was not compatible. It did not work, they are not designed with kexec in mind, I think. At least two years ago…

[Michal_Zygmunt]

Thanks! Will try the r4 for the compilation variant. I just tried kexec on 6.18-main (with modifications).

It works for me. Can send you updated config. I see two kernels booting one after another. The only thing that I’m still ‘fixing’ is the kernel modules. This is just packaging problem.

So I’m very close to make it fully work.

TL;DR; i want to install your kernel in nand or eMMC. In a way that is ‘untouchable’ later (never need to worry about it) with custom initrd that has script that parses custom boot.config.

So in a way have high-end bootloader that have access to networking drivers, nvma, custom startup scripts etc. And then based on boot.config it can load any kernel from any location via kexec.

So I can then install Openwrt, different versions on different partitions, Ubuntu etc and can also update everything locally inside nvme and don’t need to worry if repackaging kernel back to eMMC.

So Ubuntu feels like full installed Ubuntu.

I’m also enabling wireguard on second kernel etc.

So far got everything working, including two kernels booting one after another. Just dealing with kernel modules packaging. But it’s mostly because the compilation of RPI Linux I’m doing in docker containers and first was getting this bin placing ‘cp’ issues (as I noticed config has pro variant so was trying to use it), and now have similar problems with modules packaging. But I would say 99% is solved:-)

The kexec works great on 6.18-main.

[Michal_Zygmunt]

what I am using for kexec:

CONFIG_CRASH_RESERVE=y
CONFIG_VMCORE_INFO=y
CONFIG_KEXEC_CORE=y
CONFIG_KEXEC=y
CONFIG_KEXEC_HANDOVER=y
CONFIG_CRASH_DUMP=n
CONFIG_KEXEC_FILE=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_TMPFS=y
CONFIG_ARM64_VA_BITS=48

Though not options are required, however I believe the ARM64_VA_BITS=48 is what before I had problems with. Once I set it to 48 the kexec started working for me.

[Michal_Zygmunt]

also - any reason why in r4 default config the ``` CONFIG_FRAME_WARN=

is set to 1024 instead of 2048 (which is the default for 64-bit arch)?

[Michal_Zygmunt]

well… it works for me on r4-pro-8x and 6.18-main (with the above modifications). At least I was able to make it work:)

have you tried this ARM64_VA_BITS to 48 modification?

[ericwoud (Eric W.)]

Good news it works. I tried about 2 years ago, if I remember correctly it was the PCIe driver that did not function anymore. Or perhaps even it was the PCIe device that was failing. Good news now it is working.

[Michal_Zygmunt]

@Michal_Zygmunt

CONFIG_ARM64_VA_BITS

yeah… the PCIe driver is not starting up after kexec. This is what I thought is module issue, but it seems kexec that is exiting kernel is putting the PCIe in some strange mode where it cannot reinitialize it.

So the PCIe - you were able to make it work after kexec? What exactly did you do? As this is last remaining piece that I am struggling with.

[Michal_Zygmunt]

to be honest, thinking now about disabling PCIe via some dtb overlay, so first kernel boots, but then using full dtb in second kernel. That would do the work, however first kernel would not be able to use the PCIe (and nvme). So this is not a go for me.

The only option I can think of now is possibly try to play with driver code itself and create some patch there.

but U-Boot @frank-w , @ericwoud , from what I noticed also does not support enumerating nvmes to boot directly from there, as the pcie scan and nvme scan does not return anything when going inside the u-boot console…

as with r4-pro and nvme that you can put there (ex. 1TB), it just “begs” to have the kernel boot directly from nvme via u-boot, and even to have more then 1 kernel there, so you can switch between ubuntu, openwrt, and even have nice way to upgrade ubuntu kernel without touching the main u-boot loaded in NAND/eMMC, by having kernel actually not part of itb, but being part of actual /boot partition of nvme drive.

[frank-w]

@Michal_Zygmunt

but U-Boot @frank-w , @ericwoud , from what I noticed also does not support enumerating nvmes to boot directly from there, as the pcie scan and nvme scan does not return anything when going inside the u-boot console…

this is a r4pro specific issue…it works for me when chainloading u-boot.bin, but packed same binary into bl3 and flashed it does not work…i guess something is blocking the gpio-hog or it is applied too late. did same changes in openwrt and there it works in uboot…really strange

got it…CONFIG_GPIO_HOG was missing, strange that it was working on chainloading

BPI-R4P> version
U-Boot 2026.01-bpi-00051-g89fed6af1cd4-dirty (Feb 17 2026 - 07:36:49 +0100)

aarch64-linux-gnu-gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
GNU ld (GNU Binutils for Ubuntu) 2.42
BPI-R4P> pcie enum
Unknown command 'pcie' - try 'help'
BPI-R4P> pci enum 
BPI-R4P> pci scan
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
00.00.00   0x14c3     0x7988     Bridge device           0x04
01.00.00   0x1e4b     0x1202     Mass storage controller 0x08
BPI-R4P> nvme scan
BPI-R4P> nvme info
Device 0: Vendor: 0x1e4b Rev: GTdecda1 Prod: 1OXUW7Y39WJPT7YGL4QE
            Type: Hard Disk
            Capacity: 122104.3 MB = 119.2 GB (250069680 x 512)
BPI-R4P>

and only cn14 works on R4Po as the other key-m slot requires xsphy driver

[ericwoud (Eric W.)]

@Michal_Zygmunt

So the PCIe - you were able to make it work after kexec? What exactly did you do? As this is last remaining piece that I am struggling with.

Nothing really.

I did experiment with arm-trusted-firmware some more, and that did work, but it is really hacking into arm-trusted-firmware. I already have a version that can boot a linux kernel directly, but then I hacked it into using a preloaded linux found in memory.

Anyway, that was experimental, it is not in my final atf version anymore. I saved it here:

https://github.com/ericwoud/arm-trusted-firmware/commits/bpir-kexec/

Reserved memory in linux with overlay:

/dts-v1/;
/plugin/;

&{/} {
  reserved-memory {
    // 256 bytes
    atf_data: atf-data@43030000 {
      no-map;
      reg = <0 0x43030000 0 0x100>;
    };
    // 32 Mbytes
    atf_buffer: atf-buffer@51000000 {
      no-map;
      reg = <0 0x51000000 0 0x2000000>;
    };
  };
};

Using this script to load kernel, initrd and dtb in memory:

bpir-kexec.sh
============
#!/usr/lib/initcpio/busybox ash

set -e

[ -f "/usr/lib/initcpio/busybox" ] && bb="/usr/lib/initcpio/busybox" || bb="busybox"

if [ $# -ne 3 ]; then
  echo "Usage $0 {image} {dtb} {initrd}"
  linux=$(cat "/boot/bootcfg/linux")
  dtb=$(cat "/boot/bootcfg/atfdtb")
  initrd=$(cat "/boot/bootcfg/initrd")
else
  linux="$1"
  dtb="$2"
  initrd="$3"
fi

function read_64 {
  echo "0x"$($bb hexdump -n8 -s$(($2*8)) -e'8/1 "%02x""\n"' $1)
}

function write_le64 {
  n=$(printf "%016x" $(($2)))
  echo -en "\x${n:14:2}\x${n:12:2}\x${n:10:2}\x${n:8:2}\x${n:6:2}\x${n:4:2}\x${n:2:2}\x${n:0:2}" >> $1
}

pagesz=$(( $(grep -i -m1 kernelpagesize /proc/self/smaps | grep -Eo "[0-9]+") * 1024))

t="/tmp/atf-data.bin"
rm -f $t

linux_sz="0x$( printf '%x\n' $(( $(du -b $linux  | cut -f1) )))"
initrd_sz="0x$(printf '%x\n' $(( $(du -b $initrd | cut -f1) )))"
dtb_sz="0x$(   printf '%x\n' $(( $(du -b $dtb    | cut -f1) )))"

linux_sz2=$((  ($linux_sz  + $pagesz-1) & -$pagesz ))
initrd_sz2=$(( ($initrd_sz + $pagesz-1) & -$pagesz ))
dtb_sz2=$((    ($dtb_sz    + $pagesz-1) & -$pagesz ))

buf=$(      read_64 /proc/device-tree/reserved-memory/atf-buffer@*/reg 0)
bufsz=$(    read_64 /proc/device-tree/reserved-memory/atf-buffer@*/reg 1)
atfdata=$(  read_64 /proc/device-tree/reserved-memory/atf-data@*/reg   0)
atfdatasz=$(read_64 /proc/device-tree/reserved-memory/atf-data@*/reg   1)

bl31_s=0
bl31_sz=0
linux_s=$(($buf))
initrd_s=$(($buf + $linux_sz2))
dtb_s=$(($initrd_s + $initrd_sz2))

pokefile $linux_s  $linux
pokefile $initrd_s $initrd
pokefile $dtb_s    $dtb

write_le64 $t ${bl31_s}   ; write_le64 $t ${bl31_sz}
write_le64 $t ${linux_s}  ; write_le64 $t ${linux_sz}
write_le64 $t ${initrd_s} ; write_le64 $t ${initrd_sz}
write_le64 $t ${dtb_s}    ; write_le64 $t ${dtb_sz}
j=$(($atfdatasz/8-1))
i=8; while [ ${i} -lt $j ]; do
  write_le64 $t 0; let $((i++))
done
crc="0x"$(cat $t | $bb crc32)
write_le64 $t "${crc}"
pokefile $atfdata $t
echo "CRC atf-data = ${crc}"
rm -f $t

Which in turn uses pokefile:

https://github.com/ericwoud/archlinuxarm-repo/blob/peekpoke-git/pokefile.c

So bpir-kexec.sh can be run from initrd (it runs from busybox), loading kernel and initrd and dtb into the reserved memory. Then you need to reboot, and arm-trusted-firmware will find them (checking with crc, which will be cleared after usage), then atf will boot this kernel.

It works, but it is a bit hacking

[frank-w]

Do you still need such komplex structure, now that nvme is detected in ubuntu uboot? Just boot from sdcard,create partitions/filesystem and copy all from sdcard and configure uboot to use the rootfs from nvme

I have already prepared most things in uboot builtin environment,so maybe you just need to update some vars there (uenv.txt in sdcard-boot partition).

[ericwoud (Eric W.)]

@Michal_Zygmunt

being part of actual /boot partition of nvme drive.

U-Boot also supports btrfs, so you can also choose to have the kernel in the /boot directory inside the root partition and mark this root partition as boot. . All in 1 partition (for Ubuntu/ArchLinuxARM/…)

[ericwoud (Eric W.)]

You could boot to an initramfs on nand, that manages an extlinux.conf file on the boot partition on nvme.

So from the initrd setup whatever is needed on the extlinux.conf, then do a reboot.

Btw, did I mention I prefer using extlinux.conf in U-Boot?

Edit:

Then when you want to enter the initrd on nand again, delete/rename the extlinux.conf file on nvme and reboot.

Or something like this…