NixOS on Banana Pi BPI-R3

Hi,

I would like to to install yet another distribution on BPI-R3 board - NixOS. NixOS is a versatile distro that has been already deployed on many different SoCs (see community supported devices under https://nixos.wiki/wiki/NixOS_on_ARM).

I won’t lie, this is something that greatly exceeds my knowledge but I hope that with some guidance I will be able to achieve my goal :slight_smile:

By using ATF from arm-trusted-firmware/mtk-openwrt I was able to compile uboot for mtk7986 as describe in mentioned nixos wiki. The wiki says:

This should build whatever is needed for, and then build U-Boot for the desired defconfig, then open a shell with the build in $buildInputs . Do note that this particular invocation may need more changes than only the defconfig if built for other than allwinner boards.

I then copied nixos-arm image on the sd-card using dd and then copied the u-boot.bin that I got from my compilation. Not surprisingly that didn’t wok.

I then thought that I could try to load the image directly from USB like it was described at the bottom of http://www.fw-web.de/dokuwiki/doku.php?id=en:bpi-r3:uboot However, the default uboot that is on the board seems to be less capable and I couldn’t figure out how to mount neither USB nor SD card.

I understand the general concept that the SoC expects some particular partition layout to find in the installation drive. I am also aware of mtk_gpt.py tool described in [Tutorial] Build, customize and use MediaTek open-source U-Boot and ATF

What I don’t understand is how to put nixos-arm.img into that layout, and even if I somehow managed to do that, will it be enough if I install precompiled fip and uboot files (https://drive.google.com/drive/folders/1fxDR-uL3DIO-8ebgqOJd5MoWORQTTm7Z) into the remaining partitions ?

In this post([BPI-R3] Imagebuilder R3 Archlinux) Eric wrote that he was able to boot directly into the linux kernel from ATF. I guess that would be also a good idea for me. But how can I build ATF in a such a way that it will skip uboot and proceed to kernel intialization?

Here is partition structure of nixos image:

mmcblk0           179:0    0 119.1G  0 disk
├─mmcblk0p1       179:1    0    30M  0 part  /run/media/kghost/FIRMWARE
└─mmcblk0p2       179:2    0   2.7G  0 part  /run/media/kghost/NIXOS_SD

Many thanks in advance,

2 Likes

You cannot use a image which is not defined. To support this board.

Basicly you need bootchain with atf/uboot+linux-kernel and rootfs. I guess nix-os is only rootfs. So you can use img file from my uboot-repo (to have gpt,atf and uboot),unpack aarch64 rootfs from nix-os to root partition and then kernel+modules from r3 package in my kernel-repo

No, NixOS has a 2 partions in image: u-boot and rootfs.

https://nixos.wiki/wiki/NixOS_on_ARM#NixOS_installation_.26_configuration

of course without an r3-compatible u-boot.

Today I’ll try to install NixOS on my r3.

My plan is to write u-boot to NoR, then try to boot into nixos image which will be stored on sd-card. If that works, I’ll create gpt table on eMMC with a single ext4 partition and instal nixos on it. then boot into installed system.

also, as explained in NixOS_on_ARM we can create image with a custom u-boot

This is one of the problems…default bl2 (first stage bootloader in atf) looks for partition “fip” to load uboot…i guess eric changed this in his atf when using atf to load kernel directly.

Then you have to make sure the right dtb is started with the kernel before userspace gets booted

I’m not sure that this is a good idea in NixOS. One of the feature of NixOS is atomic rollbacks. For x86 it generates entries in gurb/systemd-boot. For ARM there is an option to generate extlinux-compatible configuration files which works as entry points for u-boot, if I understand correctly.

Here is some information about nixos-arm.img. when I looked at this, I thought I could just write u-boot on NoR and nixos rootfs, which generates extlinux configurations in its /boot directory, in eMMC. I hope this will work

In short it would be something like this:

First, connect to uart to see what is going on.

You can download pre-build atf from bpir64-atf-git-xxxx-.pkg.gz at ftp://ftp.woudstra.mywire.org/repo/aarch64

It can use a boot partition to load the kernel. It is patched so that it searches partlabel bpir3-sdmmc-boot or bpir3-emmc-boot

You can download pre-build uboot from bpir-uboot-git-xxxx-.pkg.gz at ftp://ftp.woudstra.mywire.org/repo/aarch64

If you like to use it also… It uses distro-boot thus extlinux.conf. Just put u-boot.bin inside the boot partition. It is quite new, so may need some changes. Use excactly that name and it will be loaded without furter configuration of atf.

They are archlinux packages, but they are also simple archives.

Create sd card like so:

wipefs --all --force ${device}

parted -s -- ${device} unit MiB \
    mklabel gpt \
    mkpart primary 34s 1 \
    mkpart primary 1 128 \
    mkpart primary 128 100% \
    set 1 legacy_boot on \
    set 2 boot on \
    name 1 bpir3-sdmmc-atf \
    name 2 bpir3-sdmmc-boot \
    name 3 bpir3-sdmmc-root \
    print

(set 2 boot on only for u-boot distro-boot)

dd the atf binairy to partition 1.

mkfs.vfat -v -F 32 -S 512 -s 16 -n "BPIR3-BOOT" ${partition 2}

When directly booting kernel, setup config for ATF as follows:

cd {bootpartition}/bootcfg
echo -n "/boot/Image" >./linux
echo -n "root=PARTLABEL=bpir3-sdmmc-root console=ttyS0,115200 debug=7 rw rootwait audit=0" >./cmdline
echo -n "/boot/dtbs/the-dtb-you-want-to-use.dtb") >./atfdtb
echo -n "/boot/initramfs-linux-bpir64-git.img") >./initrd
echo -n "DUMMYBYT" > {bootpartition}/initramfs-linux-bpir64-git.img

Use of initrd is optional and needs to be edited in .dtb. Come to think if it, probably need a dummy file. I still need to change that in the source of ATF so it will not fail when there is no initrd.

When using u-boot, you need to write an extlinux.conf

Then you need a kernel… Which is also prebuild available on the same site

1 Like

Uboot needs to have distroboot (standard boot) enabled to load extlinux.conf,which my uboot is not

My pre-build u-boot has distroboot enabled :wink:

Edit: See:

mt7xxx.h - aur.git - AUR Package Repositories (archlinux.org)

Hi guys, First of all thank you for all the answers :slight_smile:

@ericwoud I followed what you wrote and something worked though not everything:

export device=/dev/mmcblk0
wipefs --all --force ${device}
/dev/mmcblk0: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/mmcblk0: 8 bytes were erased at offset 0x1dc55ffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/mmcblk0: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
parted -s -- ${device} unit MiB \
    mklabel gpt \
    mkpart primary 34s 1 \
    mkpart primary 1 128 \
    mkpart primary 128 100% \
    set 1 legacy_boot on \
    set 2 boot on \
    name 1 bpir3-sdmmc-atf \
    name 2 bpir3-sdmmc-boot \
    name 3 bpir3-sdmmc-root \
    print
Warning: The resulting partition is not properly aligned for best performance: 34s % 2048s != 0s
Model: SD SE128 (sd/mmc)
Disk /dev/mmcblk0: 121942MiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start    End        Size       File system  Name              Flags
 1      0.02MiB  1.00MiB    0.98MiB                 bpir3-sdmmc-atf   legacy_boot
 2      1.00MiB  128MiB     127MiB                  bpir3-sdmmc-boot  boot, esp
 3      128MiB   121941MiB  121813MiB               bpir3-sdmmc-root

sudo dd if=boot/bpir3-atf-sdmmc-atf.bin of=/dev/mmcblk0p1                                                           
447+1 records in
447+1 records out
229209 bytes (229 kB, 224 KiB) copied, 0.100459 s, 2.3 MB/s
 sudo mkfs.vfat -v -F 32 -S 512 -s 16 -n "BPIR3-BOOT" /dev/mmcblk0p2                                                 13:12:38
mkfs.fat 4.2 (2021-01-31)
WARNING: Number of clusters for 32 bit FAT is less then suggested minimum.
/dev/mmcblk0p2 has 4 heads and 16 sectors per track,
hidden sectors 0x0800;
logical sector size is 512,
using 0xf8 media descriptor, with 260096 sectors;
drive number 0x80;
filesystem has 2 32-bit FATs and 16 sectors per cluster.
FAT size is 128 sectors, and provides 16238 clusters.
There are 32 reserved sectors.
Volume ID is 0aab83df, volume label BPIR3-BOOT.
sudo mount /dev/mmcblk0p2 /mnt/disk   
cd /mnt/disk/bootcfg
echo -n "/boot/Image" >./linux
echo -n "root=PARTLABEL=bpir3-sdmmc-root console=ttyS0,115200 debug=7 rw rootwait audit=0" >./cmdline
echo -n "/boot/initramfs-linux-bpir64-git.img") >./initrd
echo -n "/boot/dtbs/mt7986a-bananapi-bpi-r3.dtb" >./atfdtb
echo -n "DUMMYBYT" > /mnt/disk/initramfs-linux-bpir64-git.img
cp boot/u-boot-bpir3.bin /mnt/disk    

Use excactly that name and it will be loaded without furter configuration of atf.

I wasn’t sure which name so I put it twice there :slight_smile:

.
├── bootcfg
│   ├── atfdtb
│   ├── cmdline
│   ├── initrd
│   └── linux
├── initramfs-linux-bpir64-git.img
├── u-boot.bin
└── u-boot-bpir3.bin

Then I downloaded linux-bpir64-git-6.2.15.bpi-1-aarch64.pkg.tar.xz, extracted and copied all the files to the partition2 (using cp -RL to resolve all the symlinks).

ls -la /mnt/disk/                                                                                                    
.rwxr-xr-x  12k root 13 May 13:29 .BUILDINFO
.rwxr-xr-x  303 root 13 May 13:28 .INSTALL
.rwxr-xr-x  32k root 13 May 13:29 .MTREE
.rwxr-xr-x  645 root 13 May 13:29 .PKGINFO
drwxr-xr-x    - root 13 May 13:26 boot
drwxr-xr-x    - root 13 May 13:25 bootcfg
drwxr-xr-x    - root 13 May 13:26 etc
.rwxr-xr-x    8 root 13 May 13:17 initramfs-linux-bpir64-git.img
.rwxr-xr-x 465k root 13 May 13:22 u-boot-bpir3.bin
.rwxr-xr-x 465k root 13 May 13:22 u-boot.bin
drwxr-xr-x    - root 13 May 13:26 usr
> sudo mkfs.vfat -v -F 32 -S 512 -s 16 -n "rootfs" /dev/mmcblk0p3   

mkfs.fat 4.2 (2021-01-31)
mkfs.fat: Warning: lowercase labels might not work properly on some systems
/dev/mmcblk0p3 has 4 heads and 16 sectors per track,
hidden sectors 0x40000;
logical sector size is 512,
using 0xf8 media descriptor, with 249473024 sectors;
drive number 0x80;
filesystem has 2 32-bit FATs and 16 sectors per cluster.
FAT size is 121696 sectors, and provides 15576850 clusters.
There are 32 reserved sectors.
Volume ID is 8793d905, volume label rootfs.

Then I mounted nix-sd-image into a loop device and copied content of rootfs partition into /dev/mmclblk0p3

After putting the card into bananaPI I got:

MT7986>
F0: 102B 0000
FA: 1040 0000
FA: 1040 0000 [0200]
F9: 103F 0000
F3: 1001 0000 [0200]
F3: 1001 0000
F6: 300C 0028
F5: 0000 0000
V0: 0000 0000 [0001]
00: 0000 0000
BP: 2400 0041 [0000]
G0: 1190 0000
EC: 0000 0000 [3000]
T0: 0000 02B1 [010F]
Jump to BL

NOTICE:  BL2: v2.8(release):v2.8-538-g2f158d215-dirty
NOTICE:  BL2: Built : 11:26:21, May  9 2023
INFO:    BL2: Doing platform setup
NOTICE:  WDT: disabled
NOTICE:  CPU: MT7986 (2000MHz)
NOTICE:  EMI: Using DDR4 settings
NOTICE:  EMI: Detected DRAM size: 2048MB
NOTICE:  EMI: complex R/W mem test passed
INFO:    MediaTek MMC/SD Card Controller ver 20200520, eco 0
INFO:    Located GPT partition 'fip/fat' at 0x100000, size 0x7f00000
INFO:    BL2: Loading image id 3
INFO:    Reading filename from bootcfg/bl31
WARNING: fat_file_open: failed opening bootcfg/bl31
WARNING: Failed to access image id=3 (-2)
INFO:    Image id=3 copied: 0x43001000 - 0x43009061
INFO:    BL2: Loading image id 5
INFO:    Opened u-boot.bin
INFO:    Loading image id=5 at address 0x44000000
INFO:    Image id=5 loaded: 0x44000000 - 0x44071778
INFO:    BL2: Loading image id 22
INFO:    Loaded BL33 image is not linux kernel image, not loading DTB/INITRD
INFO:    BL2: Loading image id 27
INFO:    Loaded BL33 image is not linux kernel image, not loading DTB/INITRD
NOTICE:  BL2: Booting BL31
INFO:    Entry point address = 0x43001000
INFO:    SPSR = 0x3cd
INFO:    Total CPU count: 4
INFO:    MCUSYS: Disable 512KB L2C shared SRAM
INFO:    GICv3 without legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    Maximum SPI INTID supported: 671
INFO:    SPMC: Changed to SPMC mode
NOTICE:  BL31: v2.8(release):v2.8-538-g2f158d215-dirty
NOTICE:  BL31: Built : 11:26:21, May  9 2023
INFO:    [MPU](Region0)sa:0x0300, ea:0x0302
INFO:    [MPU](Region0)apc0:0x80b6db69, apc1:0x00b6db6d
INFO:    [MPU](Region1)sa:0x0000, ea:0x0000
INFO:    [MPU](Region1)apc0:0x00000000, apc1:0x00000000
INFO:    [MPU](Region2)sa:0x0000, ea:0x0000
INFO:    [MPU](Region2)apc0:0x00000000, apc1:0x00000000
INFO:    [MPU](Region3)sa:0x0000, ea:0x0000
INFO:    [MPU](Region3)apc0:0x00000000, apc1:0x00000000
INFO:    [DEVAPC] devapc_init done
INFO:    BL31: Initializing runtime services
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x44000000
INFO:    SPSR = 0x3c9


U-Boot 2023.04-dirty (May 09 2023 - 12:27:31 +0000)

CPU:   MediaTek MT7986
Model: mt7986-rfb
DRAM:  2 GiB
Core:  38 devices, 14 uclasses, devicetree: separate
MMC:   mmc@11230000: 0
Loading Environment from nowhere... OK
In:    serial@11002000
Out:   serial@11002000
Err:   serial@11002000
Net:
Warning: ethernet@15100000 (eth0) using random MAC address - 56:8a:58:5c:9c:c3
eth0: ethernet@15100000
Hit any key to stop autoboot:  0
MMC Device 1 not found
no mmc device at slot 1
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:2...
MT7986>

Any idea where the problem could be?

@frank-w In parallel I would also try to use your approach. I couldn’t figure out how to tell createimg command to use something else then the loop device, or what to do with that loop device next.

@name_snrl great to hear that I am not alone :slight_smile: let me know if you will have any progress with this. Idk but maybe it will be useful for you https://github.com/NixOS/nixpkgs/commit/4f477624428c8907365d4350fafa659ab96bdcfe

Looks good!

Now install your rootfs.

Write a /boot/extlinux/extlinux.conf and u-boot will load and start your kernel.

Edit:

When you supply a /boot/u-boot.bin then you do not need the bootcfg folder and files. Also do not need the dummy initramfs-linux-bpir64-git.img

From the linux package you only put the Image file and the .dtb files in the boot partition, thus the contents of the boot folder in the package… The rest of the kernel goes in the rootfs parition.

Edit2: Have not tested this u-boot for applying overlays, so use this .dtb for now (sdmmc and compatible with direct atf->kernel boot): mt7986a-bananapi-bpi-r3-atf.dtb (23.8 KB)

Edit 3: For partition 3 (rootfs) I do not recommend fat filesystem. Use ext4, f2fs or btrfs. ext4 is most difficult to setup correctly for sdmmc, using the correct erase-size.

at the moment I only have this snippet to patch the kernel, but I haven’t tested it

  boot.kernelPatches = [{
    name = "patches_for_Banana_r3";
    patch = pkgs.fetchpatch {
      url = "https://github.com/frank-w/BPI-Router-Linux/compare/fbe58b8..379c024.patch";
      hash = "sha256-+dkorhIe3PlhK+xUycoAsssu7BywiaWyZfecXq5p2P0=";
    };
  }];

nixos.iso contains these dtbs. As I understand they are built as described in the kernel. Are they correct if booting via uboot?

name_snrl in ../media/name_snrl/NIXOS_SD/boot/nixos/h46bpi1q2bs90yj08isdxnjcwk4ykxdg-linux-6.3.1-dtbs [RO]
> ls -la mediatek/mt7986*
.r-xr-xr-x 477 root  1 Jan  1970 mediatek/mt7986a-bananapi-bpi-r3-emmc.dtbo
.r-xr-xr-x 913 root  1 Jan  1970 mediatek/mt7986a-bananapi-bpi-r3-nand.dtbo
.r-xr-xr-x 956 root  1 Jan  1970 mediatek/mt7986a-bananapi-bpi-r3-nor.dtbo
.r-xr-xr-x 327 root  1 Jan  1970 mediatek/mt7986a-bananapi-bpi-r3-sd.dtbo
.r-xr-xr-x 20k root  1 Jan  1970 mediatek/mt7986a-bananapi-bpi-r3.dtb
.r-xr-xr-x 18k root  1 Jan  1970 mediatek/mt7986a-rfb.dtb
.r-xr-xr-x 15k root  1 Jan  1970 mediatek/mt7986b-rfb.dtb

so the extlinux.conf file should be on the boot partition, right?

What should be there? For now I only have this one which was generated for nixos:

# Generated file, all changes will be lost on nixos-rebuild!

# Change this to e.g. nixos-42 to temporarily boot to an older configuration.
DEFAULT nixos-default

MENU TITLE ------------------------------------------------------------
TIMEOUT 50

LABEL nixos-default
  MENU LABEL NixOS - Default
  LINUX ../nixos/r80r91zqfg25516i7wz49vd5paiwd1g5-linux-5.15.110-Image
  INITRD ../nixos/gcvd9iaqkzlq20hv76shf15marqkr2sd-initrd-linux-5.15.110-initrd
  APPEND init=/nix/store/369walk9f0qm2h66r3zq71524wys0iik-nixos-system-nixos-22.11.4087.7629f9b0680/init console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0 nohibernate loglevel=7
  FDTDIR ../nixos/r80r91zqfg25516i7wz49vd5paiwd1g5-linux-5.15.110-dtbs

Will it just work if I put it into the boot partition?

So I tried also removing u-boot.bin to force direct initialization atf->kernel but then I got:

NOTICE:  BL2: v2.8(release):v2.8-538-g2f158d215-dirty
NOTICE:  BL2: Built : 11:26:21, May  9 2023
INFO:    BL2: Doing platform setup
NOTICE:  WDT: disabled
NOTICE:  CPU: MT7986 (1998MHz)
NOTICE:  EMI: Using DDR4 settings
NOTICE:  EMI: Detected DRAM size: 2048MB
NOTICE:  EMI: complex R/W mem test passed
INFO:    MediaTek MMC/SD Card Controller ver 20200520, eco 0
INFO:    Located GPT partition 'fip/fat' at 0x100000, size 0x7f00000
INFO:    BL2: Loading image id 3
INFO:    Reading filename from bootcfg/bl31
WARNING: fat_file_open: failed opening bootcfg/bl31
WARNING: Failed to access image id=3 (-2)
INFO:    Image id=3 copied: 0x43001000 - 0x43009061
INFO:    BL2: Loading image id 5
INFO:    Reading filename from bootcfg/linux
INFO:    Opening (/boot/Image)
WARNING: fat_file_open: failed opening /boot/Image
WARNING: Failed to access image id=5 (-2)
ERROR:   BL2: Failed to load image id 5 (-2)

Though the file is there and its checksum is correct.

It cannot find the file Image inside the root of the boot partition.

You have not specified any dtb in the extlinux.conf

Use the dtb I supplied. If it works then experiment with others. Otherwise I can’t say why it doesn’t work.

I also think that linux image won’t work…

You could just use my script and build an archlinux image. There you can see exactly what you need and how it looks like. Then read [BPI-R3] Imagebuilder R3 Archlinux

I think you can use the example and apply it to nixos

It cannot find the file Image inside the root of the boot partition.

Indeed, I got confused and put both the Image and dtbs folder under /boot on the boot partition while they should be on root. After moving them there, it tries to load kernel but it fails with:

[    1.405024] FAT-fs (mmcblk0p3): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[    1.414989] VFS: Mounted root (vfat filesystem) on device 179:3.
[    1.421052] devtmpfs: error mounting -2
[    1.426198] Freeing unused kernel memory: 3264K
[    1.488942] Run /sbin/init as init process
[    1.493032]   with arguments:
[    1.495988]     /sbin/init
[    1.498681]   with environment:
[    1.501819]     HOME=/
[    1.504167]     TERM=linux
[    1.507094] Run /etc/init as init process
[    1.511095]   with arguments:
[    1.514049]     /etc/init
[    1.516656]   with environment:
[    1.519794]     HOME=/
[    1.522141]     TERM=linux
[    1.528479] Run /bin/init as init process
[    1.532491]   with arguments:
[    1.535445]     /bin/init
[    1.538052]   with environment:
[    1.541186]     HOME=/
[    1.543533]     TERM=linux
[    1.546322] Run /bin/sh as init process
[    1.550151]   with arguments:
[    1.553104]     /bin/sh
[    1.555537]   with environment:
[    1.558663]     HOME=/
[    1.561015]     TERM=linux
[    1.563760] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation.

I know, I am still using FAT, but is it because of that?

Full stack trace
F0: 102B 0000
FA: 1040 0000
FA: 1040 0000 [0200]
F9: 103F 0000
F3: 1001 0000 [0200]
F3: 1001 0000
F6: 300C 0028
F5: 0000 0000
V0: 0000 0000 [0001]
00: 0000 0000
BP: 2400 0041 [0000]
G0: 1190 0000
EC: 0000 0000 [3000]
T0: 0000 01E1 [010F]
Jump to BL

NOTICE:  BL2: v2.8(release):v2.8-538-g2f158d215-dirty
NOTICE:  BL2: Built : 11:26:21, May  9 2023
INFO:    BL2: Doing platform setup
NOTICE:  WDT: disabled
NOTICE:  CPU: MT7986 (2000MHz)
NOTICE:  EMI: Using DDR4 settings
NOTICE:  EMI: Detected DRAM size: 2048MB
NOTICE:  EMI: complex R/W mem test passed
INFO:    MediaTek MMC/SD Card Controller ver 20200520, eco 0
INFO:    Located GPT partition 'fip/fat' at 0x100000, size 0x7f00000
INFO:    BL2: Loading image id 3
INFO:    Reading filename from bootcfg/bl31
WARNING: fat_file_open: failed opening bootcfg/bl31
WARNING: Failed to access image id=3 (-2)
INFO:    Image id=3 copied: 0x43001000 - 0x43009061
INFO:    BL2: Loading image id 5
INFO:    Reading filename from bootcfg/linux
INFO:    Opening (/boot/Image)
INFO:    Loading image id=5 at address 0x44000000
INFO:    Image id=5 loaded: 0x44000000 - 0x4525fa00
INFO:    BL2: Loading image id 22
INFO:    Loaded BL33 image is linux kernel image, loading DTB/INITRD
INFO:    Reading filename from bootcfg/initrd
INFO:    Opening (/boot/initramfs-linux-bpir64-git.img)
INFO:    Loading image id=22 at address 0x48000000
INFO:    Image id=22 loaded: 0x48000000 - 0x48000008
INFO:    BL2: Loading image id 27
INFO:    Loaded BL33 image is linux kernel image, loading DTB/INITRD
INFO:    Reading filename from bootcfg/atfdtb
INFO:    Opening (/boot/dtbs/mt7986a-bananapi-bpi-r3-atf.dtb)
INFO:    Loading image id=27 at address 0x4fd00000
INFO:    Image id=27 loaded: 0x4fd00000 - 0x4fd05f59
NOTICE:  BL2: Booting BL31
INFO:    Entry point address = 0x43001000
INFO:    SPSR = 0x3cd
INFO:    Total CPU count: 4
INFO:    MCUSYS: Disable 512KB L2C shared SRAM
INFO:    EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE KERNELLLLLLLLLLLLLL
INFO:    GICv3 without legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    Maximum SPI INTID supported: 671
INFO:    SPMC: Changed to SPMC mode
NOTICE:  BL31: v2.8(release):v2.8-538-g2f158d215-dirty
NOTICE:  BL31: Built : 11:26:21, May  9 2023
INFO:    [MPU](Region0)sa:0x0300, ea:0x0302
INFO:    [MPU](Region0)apc0:0x80b6db69, apc1:0x00b6db6d
INFO:    [MPU](Region1)sa:0x0000, ea:0x0000
INFO:    [MPU](Region1)apc0:0x00000000, apc1:0x00000000
INFO:    [MPU](Region2)sa:0x0000, ea:0x0000
INFO:    [MPU](Region2)apc0:0x00000000, apc1:0x00000000
INFO:    [MPU](Region3)sa:0x0000, ea:0x0000
INFO:    [MPU](Region3)apc0:0x00000000, apc1:0x00000000
INFO:    [DEVAPC] devapc_init done
INFO:    BL31: Initializing runtime services
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x44000000
INFO:    SPSR = 0x3c9
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 6.2.15-bpi (nathalieeneric@rk3588) (gcc (GCC) 12.1.0, GNU ld (GNU Binutils) 2.38) #13 SMP PREEMPT3
[    0.000000] Machine model: Bananapi BPI-R3
[    0.000000] earlycon: uart8250 at MMIO32 0x0000000011002000 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] OF: fdt: Reserved memory: failed to reserve memory for node 'wo-emi@4fd00000': base 0x000000004fd00000, size 0 B
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000040000000-0x00000000bfffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000040000000-0x0000000042ffffff]
[    0.000000]   node   0: [mem 0x0000000043000000-0x000000004302ffff]
[    0.000000]   node   0: [mem 0x0000000043030000-0x000000004fbfffff]
[    0.000000]   node   0: [mem 0x000000004fc00000-0x000000004fcfffff]
[    0.000000]   node   0: [mem 0x000000004fd00000-0x000000004fd3ffff]
[    0.000000]   node   0: [mem 0x000000004fd40000-0x000000004ffbffff]
[    0.000000]   node   0: [mem 0x000000004ffc0000-0x00000000bfffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x00000000bfffffff]
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] percpu: Embedded 27 pages/cpu s72616 r8192 d29784 u110592
[    0.000000] pcpu-alloc: s72616 r8192 d29784 u110592 alloc=27*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: ARM erratum 845719
[    0.000000] alternatives: applying boot alternatives
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 516096
[    0.000000] Kernel command line: root=PARTLABEL=bpir3-sdmmc-root earlycon=uart8250,mmio32,0x11002000 console=ttyS0,115200 d0
[    0.000000] audit: disabled (until reboot)
[    0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 2025352K/2097152K available (9792K kernel code, 1982K rwdata, 3668K rodata, 3264K init, 419K bss, 71800)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] trace event string verifier disabled
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000]  Trampoline variant of Tasks RCU enabled.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 640 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GICv3: GICv3 features: 16 PPIs
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x000000000c080000
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 timer(s) running at 13.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x2ff89eacb, max_idle_ns: 440795202429 ns
[    0.000000] sched_clock: 56 bits at 13MHz, resolution 76ns, wraps every 4398046511101ns
[    0.008333] Console: colour dummy device 80x25
[    0.012824] Calibrating delay loop (skipped), value calculated using timer frequency.. 26.00 BogoMIPS (lpj=130000)
[    0.023239] pid_max: default: 32768 minimum: 301
[    0.028004] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.035453] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.044080] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    0.050731] cblist_init_generic: Setting adjustable number of callback queues.
[    0.057993] cblist_init_generic: Setting shift to 2 and lim to 1.
[    0.064186] cblist_init_generic: Setting shift to 2 and lim to 1.
[    0.070450] rcu: Hierarchical SRCU implementation.
[    0.075274] rcu:     Max phase no-delay instances is 1000.
[    0.081044] EFI services will not be available.
[    0.085823] smp: Bringing up secondary CPUs ...
[    0.090819] Detected VIPT I-cache on CPU1
[    0.090891] cacheinfo: Unable to detect cache hierarchy for CPU 1
[    0.090901] GICv3: CPU1: found redistributor 1 region 0:0x000000000c0a0000
[    0.090938] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[    0.091412] Detected VIPT I-cache on CPU2
[    0.091466] cacheinfo: Unable to detect cache hierarchy for CPU 2
[    0.091472] GICv3: CPU2: found redistributor 2 region 0:0x000000000c0c0000
[    0.091489] CPU2: Booted secondary processor 0x0000000002 [0x410fd034]
[    0.091912] Detected VIPT I-cache on CPU3
[    0.091964] cacheinfo: Unable to detect cache hierarchy for CPU 3
[    0.091970] GICv3: CPU3: found redistributor 3 region 0:0x000000000c0e0000
[    0.091985] CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
[    0.092041] smp: Brought up 1 node, 4 CPUs
[    0.167013] SMP: Total of 4 processors activated.
[    0.171736] CPU features: detected: 32-bit EL0 Support
[    0.176896] CPU features: detected: CRC32 instructions
[    0.182109] CPU features: emulated: Privileged Access Never (PAN) using TTBR0_EL1 switching
[    0.190504] CPU: All CPU(s) started at EL2
[    0.194637] alternatives: applying system-wide alternatives
[    0.201161] devtmpfs: initialized
[    0.207940] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.217864] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.224882] pinctrl core: initialized pinctrl subsystem
[    0.230446] DMI not present or invalid.
[    0.234755] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.241162] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations
[    0.248274] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.256098] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.264476] thermal_sys: Registered thermal governor 'fair_share'
[    0.264480] thermal_sys: Registered thermal governor 'bang_bang'
[    0.270618] thermal_sys: Registered thermal governor 'step_wise'
[    0.276651] thermal_sys: Registered thermal governor 'user_space'
[    0.282813] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.295827] ASID allocator initialised with 65536 entries
[    0.312845] cryptd: max_cpu_qlen set to 1000
[    0.318273] iommu: Default domain type: Translated
[    0.323194] iommu: DMA domain TLB invalidation policy: strict mode
[    0.329733] SCSI subsystem initialized
[    0.333679] libata version 3.00 loaded.
[    0.337647] usbcore: registered new interface driver usbfs
[    0.343179] usbcore: registered new interface driver hub
[    0.348548] usbcore: registered new device driver usb
[    0.353966] i2c-gpio i2c-gpio-0: using lines 528 (SDA) and 529 (SCL)
[    0.360511] i2c-gpio i2c-gpio-1: using lines 530 (SDA) and 531 (SCL)
[    0.366936] mc: Linux media interface: v0.10
[    0.371250] videodev: Linux video capture interface: v2.00
[    0.376792] pps_core: LinuxPPS API ver. 1 registered
[    0.381782] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    0.390971] PTP clock support registered
[    0.395865] vgaarb: loaded
[    0.398883] clocksource: Switched to clocksource arch_sys_counter
[    0.411503] NET: Registered PF_INET protocol family
[    0.416659] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.425652] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)
[    0.434341] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.442158] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.450194] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)
[    0.457883] TCP: Hash tables configured (established 16384 bind 16384)
[    0.464577] UDP hash table entries: 1024 (order: 3, 32768 bytes, linear)
[    0.471346] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes, linear)
[    0.478711] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.484434] PCI: CLS 0 bytes, default 64
[    0.488626] Unpacking initramfs...
[    0.489733] Initialise system trusted keyrings
[    0.492079] Initramfs unpacking failed: invalid magic at start of compressed archive
[    0.496614] workingset: timestamp_bits=46 max_order=19 bucket_order=0
[    0.507773] Freeing initrd memory: 8232K
[    0.511135] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.520616] jffs2: version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc.
[    0.549953] Key type asymmetric registered
[    0.554077] Asymmetric key parser 'x509' registered
[    0.559036] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    0.566471] io scheduler mq-deadline registered
[    0.571027] io scheduler kyber registered
[    0.586416] Serial: 8250/16550 driver, 3 ports, IRQ sharing disabled
[    0.594512] printk: console [ttyS0] disabled
[    0.618973] 11002000.serial: ttyS0 at MMIO 0x11002000 (irq = 116, base_baud = 2500000) is a ST16650V2
[    0.628298] printk: console [ttyS0] enabled
[    0.628298] printk: console [ttyS0] enabled
[    0.636663] printk: bootconsole [uart8250] disabled
[    0.636663] printk: bootconsole [uart8250] disabled
[    0.669931] 11003000.serial: ttyS1 at MMIO 0x11003000 (irq = 117, base_baud = 1625000) is a ST16650V2
[    0.700680] 11004000.serial: ttyS2 at MMIO 0x11004000 (irq = 118, base_baud = 1625000) is a ST16650V2
[    0.710876] mtk_rng 1020f000.rng: registered RNG driver
[    0.711153] random: crng init done
[    0.716451] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    0.728988] loop: module loaded
[    0.735359] sfp sfp-1: Host maximum power 1.0W
[    0.740515] sfp sfp-2: Host maximum power 1.0W
[    0.761188] mtk_soc_eth 15100000.ethernet: generated random MAC address 3a:d5:e1:db:85:dc
[    0.770679] mtk_soc_eth 15100000.ethernet eth0: mediatek frame engine at 0xffffffc009c80000, irq 123
[    0.779881] mtk_soc_eth 15100000.ethernet: generated random MAC address 36:e3:98:b2:72:22
[    0.789330] mtk_soc_eth 15100000.ethernet eth1: mediatek frame engine at 0xffffffc009c80000, irq 123
[    0.798862] usbcore: registered new interface driver rtl8150
[    0.804779] usbcore: registered new interface driver usb-storage
[    0.811131] i2c_dev: i2c /dev entries driver
[    0.817209] mtk-wdt 1001c000.watchdog: Watchdog enabled (timeout=31 sec, nowayout=0)
[    0.827439] ledtrig-cpu: registered to indicate activity on CPUs
[    0.833687] SMCCC: SOC_ID: ID = jep106:0426:7986 Revision = 0x00000000
[    0.840331] usbcore: registered new interface driver usbhid
[    0.845890] usbhid: USB HID core driver
[    0.850269] NET: Registered PF_INET6 protocol family
[    0.855934] Segment Routing with IPv6
[    0.859623] In-situ OAM (IOAM) with IPv6
[    0.863567] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    0.869974] NET: Registered PF_PACKET protocol family
[    0.875138] 8021q: 802.1Q VLAN Support v1.8
[    0.886467] mmc0: host does not support reading read-only switch, assuming write-enable
[    0.893092] Loading compiled-in X.509 certificates
[    0.899703] mmc0: new high speed SDXC card at address e624
[    0.906188] mmcblk0: mmc0:e624 SE128 119 GiB
[    0.907904] mtk-pcie-gen3 11280000.pcie: host bridge /soc/pcie@11280000 ranges:
[    0.917928] mtk-pcie-gen3 11280000.pcie:      MEM 0x0020000000..0x002fffffff -> 0x0020000000
[    0.919069]  mmcblk0: p1 p2 p3
[    1.138917] mtk-pcie-gen3 11280000.pcie: PCIe link down, current LTSSM state: detect.active (0x1000001)
[    1.148337] mtk-pcie-gen3: probe of 11280000.pcie failed with error -110
[    1.178846] mt7530 mdio-bus:1f: configuring for fixed/2500base-x link mode
[    1.186018] mt7530 mdio-bus:1f: Link is Up - 2.5Gbps/Full - flow control rx/tx
[    1.195219] mt7530 mdio-bus:1f wan (uninitialized): PHY [mt7530-0:00] driver [MediaTek MT7531 PHY] (irq=129)
[    1.215308] mt7530 mdio-bus:1f lan0 (uninitialized): PHY [mt7530-0:01] driver [MediaTek MT7531 PHY] (irq=130)
[    1.235181] mt7530 mdio-bus:1f lan1 (uninitialized): PHY [mt7530-0:02] driver [MediaTek MT7531 PHY] (irq=131)
[    1.255040] mt7530 mdio-bus:1f lan2 (uninitialized): PHY [mt7530-0:03] driver [MediaTek MT7531 PHY] (irq=132)
[    1.274898] mt7530 mdio-bus:1f lan3 (uninitialized): PHY [mt7530-0:04] driver [MediaTek MT7531 PHY] (irq=133)
[    1.286223] device eth0 entered promiscuous mode
[    1.290897] DSA: tree 0 setup
[    1.294390] xhci-mtk 11200000.usb: supply vbus not found, using dummy regulator
[    1.301790] xhci-mtk 11200000.usb: supply vusb33 not found, using dummy regulator
[    1.310281] xhci-mtk 11200000.usb: xHCI Host Controller
[    1.315516] xhci-mtk 11200000.usb: new USB bus registered, assigned bus number 1
[    1.325985] xhci-mtk 11200000.usb: hcc params 0x01403f99 hci version 0x110 quirks 0x0000000000210010
[    1.335156] xhci-mtk 11200000.usb: irq 134, io mem 0x11200000
[    1.340901] xhci-mtk 11200000.usb: xHCI Host Controller
[    1.346115] xhci-mtk 11200000.usb: new USB bus registered, assigned bus number 2
[    1.353504] xhci-mtk 11200000.usb: Host supports USB 3.2 Enhanced SuperSpeed
[    1.361099] hub 1-0:1.0: USB hub found
[    1.364866] hub 1-0:1.0: 2 ports detected
[    1.369280] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    1.377820] hub 2-0:1.0: USB hub found
[    1.381608] hub 2-0:1.0: 1 port detected
[    1.386561] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[    1.393675] of_cfs_init
[    1.396193] of_cfs_init: OK
[    1.405024] FAT-fs (mmcblk0p3): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[    1.414989] VFS: Mounted root (vfat filesystem) on device 179:3.
[    1.421052] devtmpfs: error mounting -2
[    1.426198] Freeing unused kernel memory: 3264K
[    1.488942] Run /sbin/init as init process
[    1.493032]   with arguments:
[    1.495988]     /sbin/init
[    1.498681]   with environment:
[    1.501819]     HOME=/
[    1.504167]     TERM=linux
[    1.507094] Run /etc/init as init process
[    1.511095]   with arguments:
[    1.514049]     /etc/init
[    1.516656]   with environment:
[    1.519794]     HOME=/
[    1.522141]     TERM=linux
[    1.528479] Run /bin/init as init process
[    1.532491]   with arguments:
[    1.535445]     /bin/init
[    1.538052]   with environment:
[    1.541186]     HOME=/
[    1.543533]     TERM=linux
[    1.546322] Run /bin/sh as init process
[    1.550151]   with arguments:
[    1.553104]     /bin/sh
[    1.555537]   with environment:
[    1.558663]     HOME=/
[    1.561015]     TERM=linux
[    1.563760] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation.
[    1.577897] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 6.2.15-bpi #13
[    1.584234] Hardware name: Bananapi BPI-R3 (DT)
[    1.588750] Call trace:
[    1.591184]  dump_backtrace+0xe0/0x124
[    1.594928]  show_stack+0x18/0x30
[    1.598231]  dump_stack_lvl+0x64/0x80
[    1.601882]  dump_stack+0x18/0x34
[    1.605185]  panic+0x130/0x300
[    1.608229]  kernel_init+0x110/0x124
[    1.611793]  ret_from_fork+0x10/0x20
[    1.615359] SMP: stopping secondary CPUs
[    1.619270] Kernel Offset: disabled
[    1.622743] CPU features: 0x00000,01000108,0000420b
[    1.627604] Memory Limit: none
[    1.630647] Rebooting in 1 seconds..

Looks good so far. Fat could be the issue, your rootfs is not as it should be. But now this will be nixos specific …

Right, I can add there something like:

fdt /path/to/dtb_file.dtb

but what bothers me the most is that my nixos kernel and the rest of the stuff that extlinux.conf refers to is on a different partition (extlinux.conf is on boot while rest is on rootfs)

When I tried to boot it the process failed with:

Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
------------------------------------------------------------
1:      NixOS - Default
Enter choice: 1:        NixOS - Default
Retrieving file: /boot/extlinux/../nixos/r80r91zqfg25516i7wz49vd5paiwd1g5-linux-5.15.110-Image
Skipping nixos-default for failure retrieving kernel
EXTLINUX FAILED: continuing...

Which is somehow not a surprise. Could you tell me how can I refer to the rootfs partition in that file? Or maybe I could merge both boot and rootfs partitions and then I wouldn’t have that problem at all?

The entire purpose of the boot partition in fat is that the bootloader can access it, and the rootfs partition can be anything, not readable by bootloader

However, uboot can also read ext4, but I’m not sure if distro-boot reads from ext4.

You need to parted set boot on then also on the rootfs partition, formatted ext4. It will then hopefully search it also. I think it could find the /extlinux/extlinux.conf file on that partition too. But first remove it from boot partition before trying, as this will be searched earlier

Or you make nixos copy the files to the boot partition mounted at /boot