NixOS on Banana Pi BPI-R3

Finally some success :slight_smile:

CPU:   MediaTek MT7986
Model: mt7986-rfb
DRAM:  2 GiB
Core:  38 devices, 14 uclasses, devicetree: separate
MMC:   mmc@11230000: 0
Loading Environment from nowhere... OK
In:    serial@11002000
Out:   serial@11002000
Err:   serial@11002000
Net:
Warning: ethernet@15100000 (eth0) using random MAC address - 3e:5a:18:68:97:ce
eth0: ethernet@15100000
Hit any key to stop autoboot:  0
MMC Device 1 not found
no mmc device at slot 1
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:2...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
------------------------------------------------------------
1:      NixOS - Default
Enter choice: 1:        NixOS - Default
Retrieving file: /boot/extlinux/../nixos/wbv7qr9q5bqlr4dn5srjqmhxppp2sxgn-linux-6.3.1-Image
Retrieving file: /boot/extlinux/../nixos/slnriwqj7rw29dqp1fh04736j7vwfqn9-initrd-linux-6.3.1-initrd
append: init=/nix/store/9ifmb0qa9jm6bl95walaf0gn1w7nsfn7-nixos-system-nixos-23.05pre483514.790a208ab94/init console=ttyS0,11527
Retrieving file: /boot/extlinux/../nixos/wbv7qr9q5bqlr4dn5srjqmhxppp2sxgn-linux-6.3.1-dtbs/mt7986a-sd-rfb
## Flattened Device Tree blob at bf7fdcd0
   Booting using the fdt blob at 0xbf7fdcd0
Working FDT set to bf7fdcd0
   Loading Ramdisk to bec3b000, end bf7f79c7 ... OK
   Loading Device Tree to 000000004fcfa000, end 000000004fcff0df ... OK
Working FDT set to 4fcfa000

Starting kernel ...

However it hangs here. The problem might be the one described here: https://nixos.wiki/wiki/NixOS_on_ARM#Enable_UART

If you try to use UART to log on NixOS, you might hang on the line ā€œStarting kernel ā€¦ā€. To enable UART, you will need to add at the end of the line that contains loglevel4 in the file /extlinux/extlinux.conf the text:

Breeze-text-x-plain.png

/extlinux/extlinux.conf

console=ttyAMA0,115200n8

Breeze-text-x-plain.png

/extlinux/extlinux.conf

console=ttyS0,115200n8

However I already have in my extlinux.conf this:

LABEL nixos-default
  MENU LABEL NixOS - Default
  LINUX ../nixos/wbv7qr9q5bqlr4dn5srjqmhxppp2sxgn-linux-6.3.1-Image
  INITRD ../nixos/slnriwqj7rw29dqp1fh04736j7vwfqn9-initrd-linux-6.3.1-initrd
  APPEND init=/nix/store/9ifmb0qa9jm6bl95walaf0gn1w7nsfn7-nixos-system-nixos-23.05pre483514.790a208ab94/init console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0 loglevel=7
  FDTDIR ../nixos/wbv7qr9q5bqlr4dn5srjqmhxppp2sxgn-linux-6.3.1-dtbs

Can I bruteforce it and add S1,S2 and S3?

Perhapse someone at nixos can help you with the kernel.

Is it configured for the appropriate mediatek devices?

Edit: i found out that I cannot use the standard archlinuxarm linux kernel, because many mtk config options are not enabled. Mediatek boards are not included. It is why I need to use my own linux package.

It doesnā€™t apply any patching to code, only some changes to devicetree. Then I added Dynamic overlay support, which is an addition to the code. I use Dynamic overlay to access Emmc at hs200 when booting from sdmmc

Is it configured for the appropriate mediatek devices?

Not sure what do you mean by ā€œconfiguredā€. I used latest version of unstable channel that contains the latest kernel and I can see dtb files for mediatek devices in the nixos image:

āÆ ls mediatek                                                                                                         
mt2712-evb.dtb                     mt8173-elm-hana.dtb                     mt8183-kukui-kakadu.dtb
mt6755-evb.dtb                     mt8173-elm.dtb                          mt8183-kukui-kodama-sku16.dtb
mt6779-evb.dtb                     mt8173-evb.dtb                          mt8183-kukui-kodama-sku32.dtb
mt6795-evb.dtb                     mt8183-evb.dtb                          mt8183-kukui-kodama-sku272.dtb
mt6795-sony-xperia-m5.dtb          mt8183-kukui-jacuzzi-burnet.dtb         mt8183-kukui-kodama-sku288.dtb
mt6797-evb.dtb                     mt8183-kukui-jacuzzi-cozmo.dtb          mt8183-kukui-krane-sku0.dtb
mt6797-x20-dev.dtb                 mt8183-kukui-jacuzzi-damu.dtb           mt8183-kukui-krane-sku176.dtb
mt7622-bananapi-bpi-r64.dtb        mt8183-kukui-jacuzzi-fennel-sku1.dtb    mt8183-pumpkin.dtb
mt7622-rfb1.dtb                    mt8183-kukui-jacuzzi-fennel-sku6.dtb    mt8186-evb.dtb
mt7986a-bananapi-bpi-r3-emmc.dtbo  mt8183-kukui-jacuzzi-fennel-sku7.dtb    mt8192-asurada-hayato-r1.dtb
mt7986a-bananapi-bpi-r3-nand.dtbo  mt8183-kukui-jacuzzi-fennel14-sku2.dtb  mt8192-asurada-spherion-r0.dtb
mt7986a-bananapi-bpi-r3-nor.dtbo   mt8183-kukui-jacuzzi-fennel14.dtb       mt8192-evb.dtb
mt7986a-bananapi-bpi-r3-sd.dtbo    mt8183-kukui-jacuzzi-juniper-sku16.dtb  mt8195-cherry-tomato-r1.dtb
mt7986a-bananapi-bpi-r3.dtb        mt8183-kukui-jacuzzi-kappa.dtb          mt8195-cherry-tomato-r2.dtb
mt7986a-rfb.dtb                    mt8183-kukui-jacuzzi-kenzo.dtb          mt8195-cherry-tomato-r3.dtb
mt7986b-rfb.dtb                    mt8183-kukui-jacuzzi-willow-sku0.dtb    mt8195-demo.dtb
mt8167-pumpkin.dtb                 mt8183-kukui-jacuzzi-willow-sku1.dtb    mt8195-evb.dtb
mt8173-elm-hana-rev7.dtb           mt8183-kukui-kakadu-sku22.dtb           mt8516-pumpkin.dtb

i found out that I cannot use the standard archlinuxarm linux kernel, because many mtk config options are not enabled. Mediatek boards are not included. It is why I need to use my own linux package.

Ah that might be it. What do you mean not enabled? How can I enable it? Also what do you mean ā€œmediatek boards are not includedā€?

It doesnā€™t apply any patching to code, only some changes to devicetree. Then I added Dynamic overlay support, which is an addition to the code. I use Dynamic overlay to access Emmc at hs200 when booting from sdmmc

You mean that even though mediatek devices are in the devicetree, there are no patches applied to kernel? Sorry, I am not sure if I understand.

Anyway, I decided to try out the dtb file that you sent earlier. I modified the extlinux.conf and changed

  FDTDIR ../nixos/wbv7qr9q5bqlr4dn5srjqmhxppp2sxgn-linux-6.3.1-dtbs

to

FDT ../dtbs/mt7986a-bananapi-bpi-r3-atf.dtb

And it worked (sort of):

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 6.3.1 (nixbld@localhost) (gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.40) #1-NixOS SMP Sun Apr 30 23
[    0.000000] Machine model: Bananapi BPI-R3
[    0.000000] efi: UEFI not found.
[    0.000000] OF: reserved mem: 0x0000000015194000..0x0000000015194fff (4 KiB) nomap non-reusable wo-boot@15194000
[    0.000000] OF: reserved mem: 0x00000000151e0000..0x00000000151e7fff (32 KiB) nomap non-reusable wo-ilm@151e0000
[    0.000000] OF: reserved mem: 0x00000000151e8000..0x00000000151e9fff (8 KiB) nomap non-reusable wo-dlm@151e8000
[    0.000000] OF: reserved mem: 0x00000000151f0000..0x00000000151f7fff (32 KiB) nomap non-reusable wo-ilm@151f0000
[    0.000000] OF: reserved mem: 0x00000000151f8000..0x00000000151f9fff (8 KiB) nomap non-reusable wo-dlm@151f8000
[    0.000000] OF: reserved mem: 0x0000000043000000..0x000000004302ffff (192 KiB) nomap non-reusable secmon@43000000
[    0.000000] OF: reserved mem: 0x000000004fc00000..0x000000004fcfffff (1024 KiB) nomap non-reusable wmcpu-reserved@4fc00000
[    0.000000] OF: reserved mem: 0x000000004fd00000..0x000000004fd3ffff (256 KiB) nomap non-reusable wo-emi@4fd00000
[    0.000000] OF: reserved mem: 0x000000004fd40000..0x000000004fd7ffff (256 KiB) nomap non-reusable wo-emi@4fd40000
[    0.000000] OF: reserved mem: 0x000000004fd80000..0x000000004ffbffff (2304 KiB) nomap non-reusable wo-data@4fd80000
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x00000000bfffffff]
[    0.000000] NUMA: NODE_DATA [mem 0xbfbe88c0-0xbfbebfff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000040000000-0x00000000bfffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000040000000-0x0000000042ffffff]
[    0.000000]   node   0: [mem 0x0000000043000000-0x000000004302ffff]
[    0.000000]   node   0: [mem 0x0000000043030000-0x000000004fbfffff]
[    0.000000]   node   0: [mem 0x000000004fc00000-0x000000004ffbffff]
[    0.000000]   node   0: [mem 0x000000004ffc0000-0x00000000bfffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x00000000bfffffff]
[    0.000000] cma: Reserved 32 MiB at 0x00000000bac00000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] percpu: Embedded 31 pages/cpu s89320 r8192 d29464 u126976
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: ARM erratum 845719
[    0.000000] alternatives: applying boot alternatives
[    0.000000] Fallback order for Node 0: 0
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 516096
[    0.000000] Policy zone: DMA
[    0.000000] Kernel command line: init=/nix/store/9ifmb0qa9jm6bl95walaf0gn1w7nsfn7-nixos-system-nixos-23.05pre483514.790a2088
[    0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
[    0.000000] Memory: 1947424K/2097152K available (21376K kernel code, 5812K rwdata, 19504K rodata, 12736K init, 740K bss, 11)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] ftrace: allocating 67185 entries in 263 pages
[    0.000000] ftrace: allocated 263 pages with 4 groups
[    0.000000] trace event string verifier disabled
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=4.
[    0.000000]  Rude variant of Tasks RCU enabled.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 640 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GICv3: GICv3 features: 16 PPIs
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x000000000c080000
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 timer(s) running at 13.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x2ff89eacb, max_idle_ns: 440795202429 ns
[    0.000000] sched_clock: 56 bits at 13MHz, resolution 76ns, wraps every 4398046511101ns
[    0.000454] Console: colour dummy device 80x25
[    0.000461] printk: console [tty0] enabled
[    0.000808] Calibrating delay loop (skipped), value calculated using timer frequency.. 26.00 BogoMIPS (lpj=52000)
[    0.000822] pid_max: default: 32768 minimum: 301
[    0.000997] LSM: initializing lsm=capability,landlock,yama,integrity,bpf
[    0.001047] landlock: Up and running.
[    0.001053] Yama: becoming mindful.
[    0.001092] LSM support for eBPF active
[    0.001215] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.001233] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.002671] cblist_init_generic: Setting adjustable number of callback queues.
[    0.002696] cblist_init_generic: Setting shift to 2 and lim to 1.
[    0.002753] cblist_init_generic: Setting shift to 2 and lim to 1.
[    0.002902] rcu: Hierarchical SRCU implementation.
[    0.002909] rcu:     Max phase no-delay instances is 1000.
[    0.004863] EFI services will not be available.
[    0.005189] smp: Bringing up secondary CPUs ...
[    0.005614] Detected VIPT I-cache on CPU1
[    0.005658] GICv3: CPU1: found redistributor 1 region 0:0x000000000c0a0000
[    0.005693] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[    0.006130] Detected VIPT I-cache on CPU2
[    0.006159] GICv3: CPU2: found redistributor 2 region 0:0x000000000c0c0000
[    0.006175] CPU2: Booted secondary processor 0x0000000002 [0x410fd034]
[    0.006554] Detected VIPT I-cache on CPU3
[    0.006578] GICv3: CPU3: found redistributor 3 region 0:0x000000000c0e0000
[    0.006591] CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
[    0.006636] smp: Brought up 1 node, 4 CPUs
[    0.006682] SMP: Total of 4 processors activated.
[    0.006689] CPU features: detected: 32-bit EL0 Support
[    0.006694] CPU features: detected: 32-bit EL1 Support
[    0.006700] CPU features: detected: CRC32 instructions
[    0.006751] CPU: All CPU(s) started at EL2
[    0.006769] alternatives: applying system-wide alternatives
[    0.008707] devtmpfs: initialized
[    0.012373] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.012411] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.013292] pinctrl core: initialized pinctrl subsystem
[    0.014426] DMI not present or invalid.
[    0.014993] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.015907] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations
[    0.015995] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.016137] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.016217] audit: initializing netlink subsys (disabled)
[    0.016345] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1
[    0.017189] thermal_sys: Registered thermal governor 'step_wise'
[    0.017194] thermal_sys: Registered thermal governor 'power_allocator'
[    0.017261] cpuidle: using governor ladder
[    0.017284] cpuidle: using governor menu
[    0.017518] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.017598] ASID allocator initialised with 65536 entries
[    0.018864] Serial: AMBA PL011 UART driver
[    0.030299] KASLR disabled due to lack of seed
[    0.054645] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
[    0.054673] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page
[    0.054680] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages
[    0.054686] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page
[    0.054693] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    0.054699] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page
[    0.054705] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages
[    0.054711] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page
[    0.498535] fbcon: Taking over console
[    0.498586] ACPI: Interpreter disabled.
[    0.500397] iommu: Default domain type: Translated
[    0.500420] iommu: DMA domain TLB invalidation policy: strict mode
[    0.500726] SCSI subsystem initialized
[    0.501017] usbcore: registered new interface driver usbfs
[    0.501039] usbcore: registered new interface driver hub
[    0.501062] usbcore: registered new device driver usb
[    0.501571] pps_core: LinuxPPS API ver. 1 registered
[    0.501578] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    0.501600] PTP clock support registered
[    0.501678] EDAC MC: Ver: 3.0.0
[    0.502082] scmi_core: SCMI protocol bus registered
[    0.502519] FPGA manager framework
[    0.502613] Advanced Linux Sound Architecture Driver Initialized.
[    0.503384] vgaarb: loaded
[    0.512627] clocksource: Switched to clocksource arch_sys_counter
[    0.512975] VFS: Disk quotas dquot_6.6.0
[    0.513027] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.513231] pnp: PnP ACPI: disabled
[    0.517978] NET: Registered PF_INET protocol family
[    0.518195] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.519845] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)
[    0.519902] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.519917] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.519993] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)
[    0.520378] TCP: Hash tables configured (established 16384 bind 16384)
[    0.520725] MPTCP token hash table entries: 2048 (order: 3, 49152 bytes, linear)
[    0.520835] UDP hash table entries: 1024 (order: 3, 32768 bytes, linear)
[    0.520869] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes, linear)
[    0.521025] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.521509] RPC: Registered named UNIX socket transport module.
[    0.521526] RPC: Registered udp transport module.
[    0.521532] RPC: Registered tcp transport module.
[    0.521537] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.521551] NET: Registered PF_XDP protocol family
[    0.521573] PCI: CLS 0 bytes, default 64
[    0.521848] Trying to unpack rootfs image as initramfs...
[    0.529333] kvm [1]: IPA Size Limit: 40 bits
[    0.531837] kvm [1]: vgic-v2@c420000
[    0.531884] kvm [1]: GIC system register CPU interface enabled
[    0.531930] kvm [1]: vgic interrupt IRQ9
[    0.531956] kvm [1]: Hyp mode initialized successfully
[    0.533422] Initialise system trusted keyrings
[    0.533795] workingset: timestamp_bits=42 max_order=19 bucket_order=0
[    0.533886] zbud: loaded
[    0.534595] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.535050] NFS: Registering the id_resolver key type
[    0.535102] Key type id_resolver registered
[    0.535109] Key type id_legacy registered
[    0.535142] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.535151] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.535330] 9p: Installing v9fs 9p2000 file system support
[    0.557740] Key type asymmetric registered
[    0.557818] Asymmetric key parser 'x509' registered
[    0.557927] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[    0.558193] io scheduler mq-deadline registered
[    0.558211] io scheduler kyber registered
[    0.566109] EINJ: ACPI disabled.
[    0.582601] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.584995] printk: console [ttyS0] disabled
[    0.605193] 11002000.serial: ttyS0 at MMIO 0x11002000 (irq = 116, base_baud = 2500000) is a ST16650V2
[    0.605284] printk: console [ttyS0] enabled
[    0.856144] Freeing initrd memory: 12016K
[    0.882126] 11003000.serial: ttyS1 at MMIO 0x11003000 (irq = 117, base_baud = 1625000) is a ST16650V2
[    1.783840] 11004000.serial: ttyS2 at MMIO 0x11004000 (irq = 118, base_baud = 1625000) is a ST16650V2
[    1.793875] SuperH (H)SCI(F) driver initialized
[    1.798721] msm_serial: driver initialized
[    1.808158] loop: module loaded
[    1.812099] megasas: 07.725.01.00-rc1
[    1.820983] tun: Universal TUN/TAP device driver, 1.6
[    1.826601] thunder_xcv, ver 1.0
[    1.829862] thunder_bgx, ver 1.0
[    1.833100] nicpf, ver 1.0
[    1.836492] hns3: Hisilicon Ethernet Network Driver for Hip08 Family - version
[    1.843707] hns3: Copyright (c) 2017 Huawei Corporation.
[    1.849036] hclge is initializing
[    1.852355] e1000: Intel(R) PRO/1000 Network Driver
[    1.857223] e1000: Copyright (c) 1999-2006 Intel Corporation.
[    1.862970] e1000e: Intel(R) PRO/1000 Network Driver
[    1.867923] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    1.873853] igb: Intel(R) Gigabit Ethernet Network Driver
[    1.879240] igb: Copyright (c) 2007-2014 Intel Corporation.
[    1.884817] igbvf: Intel(R) Gigabit Virtual Function Network Driver
[    1.891072] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    1.897205] sky2: driver version 1.30
[    1.902853] usbcore: registered new interface driver usb-storage
[    1.910364] i2c_dev: i2c /dev entries driver
[    1.918240] mtk-wdt 1001c000.watchdog: Watchdog enabled (timeout=31 sec, nowayout=0)
[    1.926401] ghes_edac: GHES probing device list is empty
[    1.927665] sdhci: Secure Digital Host Controller Interface driver
[    1.939176] sdhci: Copyright(c) Pierre Ossman
[    1.944004] Synopsys Designware Multimedia Card Interface Driver
[    1.950841] sdhci-pltfm: SDHCI platform and OF driver helper
[    1.958178] ledtrig-cpu: registered to indicate activity on CPUs
[    1.965472] SMCCC: SOC_ID: ID = jep106:0426:7986 Revision = 0x00000000
[    1.972378] hid: raw HID events driver (C) Jiri Kosina
[    1.977799] usbcore: registered new interface driver usbhid
[    1.983364] usbhid: USB HID core driver
[    1.991374] drop_monitor: Initializing network drop monitor service
[    2.011035] mmc0: host does not support reading read-only switch, assuming write-enable
[    2.024406] mmc0: new high speed SDXC card at address e624
[    2.031185] mmcblk0: mmc0:e624 SE128 119 GiB
[    2.045824]  mmcblk0: p1 p2
[    2.054396] NET: Registered PF_INET6 protocol family
[    2.061363] Segment Routing with IPv6
[    2.065102] In-situ OAM (IOAM) with IPv6
[    2.069084] NET: Registered PF_PACKET protocol family
[    2.074298] 9pnet: Installing 9P2000 support
[    2.078629] Key type dns_resolver registered
[    2.089447] registered taskstats version 1
[    2.093841] Loading compiled-in X.509 certificates
[    2.098846] zswap: loaded using pool lzo/zbud
[    2.109093] Key type .fscrypt registered
[    2.113038] Key type fscrypt-provisioning registered
[    2.126598] xhci-mtk 11200000.usb: supply vbus not found, using dummy regulator
[    2.134123] xhci-mtk 11200000.usb: supply vusb33 not found, using dummy regulator
[    2.142515] xhci-mtk 11200000.usb: xHCI Host Controller
[    2.147748] xhci-mtk 11200000.usb: new USB bus registered, assigned bus number 1
[    2.158152] xhci-mtk 11200000.usb: hcc params 0x01403f99 hci version 0x110 quirks 0x0000000000210010
[    2.167326] xhci-mtk 11200000.usb: irq 124, io mem 0x11200000
[    2.173183] xhci-mtk 11200000.usb: xHCI Host Controller
[    2.178404] xhci-mtk 11200000.usb: new USB bus registered, assigned bus number 2
[    2.185792] xhci-mtk 11200000.usb: Host supports USB 3.2 Enhanced SuperSpeed
[    2.193272] hub 1-0:1.0: USB hub found
[    2.197040] hub 1-0:1.0: 2 ports detected
[    2.201307] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    2.209689] hub 2-0:1.0: USB hub found
[    2.213453] hub 2-0:1.0: 1 port detected
[    2.218234] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[    2.225776] ALSA device list:
[    2.228763]   No soundcards found.
[    2.241244] Freeing unused kernel memory: 12736K
[    2.264663] Run /init as init process
[    2.388830] device-mapper: ioctl: 4.47.0-ioctl (2022-07-28) initialised: [email protected]
[    2.612802] usb 1-1: new high-speed USB device number 2 using xhci-mtk
[    2.765579] hub 1-1:1.0: USB hub found
[    2.769531] hub 1-1:1.0: 4 ports detected
[    9.464635] random: crng init done
[   33.673466] /dev/disk/by-label/NIXOS_SD: Can't open blockdev

The error obviously stems from the fact that I am not using partition layout from the original nixos image but rather my modification using atf etc. But that seems to be mostly concern of nixos configuration.

However, I wonder why it worked after switching to that file ? Could you explain what kind of changes have you made there comparing to the original file?

edit:

Have not tested this u-boot for applying overlays, so use this .dtb for now (sdmmc and compatible with direct atf->kernel boot): mt7986a-bananapi-bpi-r3-atf.dtb (23.8 KB)

Maybe that was the reason :slight_smile: wdyt?

See def config of @frank-w

https://github.com/frank-w/BPI-Router-Linux/blob/6.3-main/arch/arm64/configs/mt7986a_bpi-r3_defconfig

And/or mine:

https://aur.archlinux.org/cgit/aur.git/tree/defconfig?h=linux-bpir64-git

It selects which drivers are build into the kernel.

With a glance at your log looks like at least some mtk support build in.

The dtb I shared has overlays applied, so you can try that to start with, not worrying about overlays.

Just set label on your rootfs partition :slight_smile:

@frank-w if that was only that easy :slight_smile: tried that (keep in mind that I merged boot and rootfs partitions for the sake of testing):

NOTICE:  BL2: v2.8(release):v2.8-538-g2f158d215-dirty
NOTICE:  BL2: Built : 11:26:21, May  9 2023
INFO:    BL2: Doing platform setup
NOTICE:  WDT: disabled
NOTICE:  CPU: MT7986 (2000MHz)
NOTICE:  EMI: Using DDR4 settings
NOTICE:  EMI: Detected DRAM size: 2048MB
NOTICE:  EMI: complex R/W mem test passed
INFO:    MediaTek MMC/SD Card Controller ver 20200520, eco 0
ERROR:   Partition 'fip/boot' not found
PANIC at PC : 0x0000000000204bf8

I guess that @ericwoudā€™s atf requires u-boot partition to be called exactly bpir3-sdmmc-boot. I will try to un-merge boot and rootfs partitions now.

edit: Although it also makes me wonder because the nixos installation disk does not have any labels assigned:

sudo parted /dev/loop1                                                                                               19:02:09
GNU Parted 3.5
Using /dev/loop1
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: Loopback device (loopback)
Disk /dev/loop1: 7340MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      8389kB  39.8MB  31.5MB  primary  fat16
 2      39.8MB  2597MB  2557MB  primary  ext4         boot

(parted)

How could that be?

@ericwoud

The dtb I shared has overlays applied, so you can try that to start with, not worrying about overlays.

That was the result of trying it. The regular one doesnā€™t work which I think indicates that the u-boot I got from ftp doesnā€™t support overlays.

I think overlay would work as i have not changed config for it in u-boot. But it is more difficult to get it right and be able to boot. And I have not used it. Worry about it when you have booting ok.

The partlabel indeed needs to be exactly bpir3-sdmmc-boot. You can leave u-boot on boot partition, save all kernel files on rootfs partition. So keep 2 partitions separate. ATF cannot read ext4.

You can checkout partlabels with

lsblk -o name,partlabel

Edit:

But any normal os will have the ability to save extlinux.conf, initrd and Image Image to the boot partition mounted at /boot

Worry about it when you have booting ok.

good idea :slight_smile:

You can checkout partlabels with

Yes, lsblk was the answer. parted only shows labels for gpt partition table while that was msdos partition table, then I had to use dosfslabel to change the label name. After that the system booted without any errors up to the point:

[    2.179931] xhci-mtk 11200000.usb: Host supports USB 3.2 Enhanced SuperSpeed
[    2.187419] hub 1-0:1.0: USB hub found
[    2.191187] hub 1-0:1.0: 2 ports detected
[    2.195463] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    2.203838] hub 2-0:1.0: USB hub found
[    2.207602] hub 2-0:1.0: 1 port detected
[    2.212389] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[    2.219940] ALSA device list:
[    2.222925]   No soundcards found.
[    2.235408] Freeing unused kernel memory: 12736K
[    2.256557] Run /init as init process
[    2.381067] device-mapper: ioctl: 4.47.0-ioctl (2022-07-28) initialised: [email protected]
[    2.616592] usb 1-1: new high-speed USB device number 2 using xhci-mtk
[    2.769458] hub 1-1:1.0: USB hub found
[    2.773457] hub 1-1:1.0: 4 ports detected
[    9.636530] random: crng init done

where it hung. I briefly looked looked that last line over the internet and it might indicate some problem with mounting rootfs. Maybe it tries to mount the same disk for the second time and fail?

Another reason to split them into boot and rootfs :slight_smile: I might also try to increase log level in extlinux.conf

But any normal os will have the ability to save extlinux.conf, initrd and Image Image to the boot partition mounted at /boot

I actually thought about putting all of them in the rootfs. Is there any preference for one over the other?

Whatever works best for you.

Why do you choose for nixos? It is definitely more complicated in some aspects. Iā€™ve chosen Archlinux because of the Rolling updates and the pkgbuild system

Afaik bl2 only looks for gpt partition fip to load uboot from and rootfs should be an ext2/3/4 or similar handling linux filesystem permissionsā€¦so not fat

I put it like you said on the boot partition. It barely fits.

This time I moved a little bit further:

[    9.760811] random: crng init done
[    9.972578] EXT4-fs (mmcblk0p3): mounted filesystem dd449cee-85e1-406a-b521-2a6a36a9bb5d with ordered data mode. Quota mode.
[    9.987582] EXT4-fs (mmcblk0p3): re-mounted dd449cee-85e1-406a-b521-2a6a36a9bb5d. Quota mode: none.
[   10.697666] EXT4-fs (mmcblk0p3): re-mounted dd449cee-85e1-406a-b521-2a6a36a9bb5d. Quota mode: none.
[   10.714095] booting system configuration /nix/store/9ifmb0qa9jm6bl95walaf0gn1w7nsfn7-nixos-system-nixos-23.05pre483514.790a4
[   13.532271] EXT4-fs (mmcblk0p3): resizing filesystem from 31184128 to 31184379 blocks
[   13.540349] EXT4-fs (mmcblk0p3): resized filesystem to 31184379
[   15.498508] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00
[   15.506169] CPU: 1 PID: 1 Comm: init Not tainted 6.3.1 #1-NixOS
[   15.512078] Hardware name: Bananapi BPI-R3 (DT)
[   15.516594] Call trace:
[   15.519030]  dump_backtrace+0xa8/0x138
[   15.522776]  show_stack+0x20/0x38
[   15.526079]  dump_stack_lvl+0x48/0x60
[   15.529733]  dump_stack+0x18/0x28
[   15.533038]  panic+0x388/0x3a0
[   15.536084]  do_exit+0x8d4/0xa10
[   15.539301]  do_group_exit+0x3c/0xa8
[   15.542865]  __arm64_sys_exit_group+0x20/0x28
[   15.547209]  invoke_syscall+0x50/0x128
[   15.550949]  el0_svc_common.constprop.0+0x4c/0x100
[   15.555729]  do_el0_svc+0x40/0xa8
[   15.559034]  el0_svc+0x34/0xd8
[   15.562082]  el0t_64_sync_handler+0xf4/0x120
[   15.566339]  el0t_64_sync+0x190/0x198
[   15.569992] SMP: stopping secondary CPUs
[   15.573906] Kernel Offset: disabled
[   15.577382] CPU features: 0x000000,08000400,0200421b
[   15.582333] Memory Limit: none
[   15.585379] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00 ]---

Any idea what that means?

This is how my extlinux.conf looks like (maybe there is some obvious bug in it):

DEFAULT nixos-default

MENU TITLE ------------------------------------------------------------
TIMEOUT 50

LABEL nixos-default
  MENU LABEL NixOS - Default
  LINUX ../nixos/wbv7qr9q5bqlr4dn5srjqmhxppp2sxgn-linux-6.3.1-Image
  INITRD ../nixos/slnriwqj7rw29dqp1fh04736j7vwfqn9-initrd-linux-6.3.1-initrd
  APPEND root=LABEL=NIXOS_SD rootwait rootfstype=ext4 init=/nix/store/9ifmb0qa9jm6bl95walaf0gn1w7nsfn7-nixos-system-nixos-23.05pre483514.790a208ab94/init console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0 loglevel=7
  FDT ../dtbs/mt7986a-bananapi-bpi-r3-atf.dtb

Update: putting files on the ext4 rootfs partition also works but it ends the same - with kernel panic.

Regarding your question:

Why do you choose for nixos?

I donā€™t want to pollute that topic so I will try to keep the answer short. If not nixos I wouldnā€™t even bother with stuff like banana PI. Nixos reproduceability allows me to replicate my configuration across multiple different machines/devices and do it in a confident way thanks to its deterministic nature. Sure, it seems to be more complicated, however it is not. In todayā€™s world everything is complicated, with nix you just need to tackle that complexity from the first go, instead of waiting till it slowly cripples you.

Iā€™ve already done this, it works but the 8GB is a bit tight. Iā€™m booting from SPI and have a standard NixOS extlinux-style loader setup. https://github.com/lorenz/nixpkgs/tree/bpi-r3 is my old setup, Iā€™ve since improved it a bit (updated to 6.3, some other fixes). I donā€™t really have much time now to explain how all of it works, but you can look at the setup there already.

2 Likes

Wow @lorenz this is awesome! Let me go through it, I am sure I will have many questions.

Could you briefly explain what is the usability state of it? I mean does everything work?

Good the hear that all of this has already been figured out.

I was wondering though, it does not seem to use extlinux.conf, or does it? Which other way do you use to boot kernel from u-boot?

Thatā€™s why I want to install uboot on NoR and rootfs on m.2. This is possible by adding the appropriate BOOT_TARGET_DEVICES when building uboot, right?

Where is kernel located? Uboot cannot access nvme

But in this documentation nvme is present :thinking:

Yes uboot can access nvme in general,but not on r3 as pcie driver is missing for mt7986

You can, but you need to setup a boot partition on emmc, where U-Boot can load the kernel Image from.

But then it also wouldnā€™t hurt to just use emmc for ATF and U-Boot also, instead of NOR. That only needs 1MB.

Bit short on time this week, will have more on the weekend to explain things and upload code. Everything works (including LTE, WiFi, Ethernet with VLANs/VXLAN/whatever, also have a custom ZigBee HAT with Zigbee2MQTT working as well as RTC shenanigans), but there are issues with the stock NixOS modules (for example the official hostapd module only allows one hostapd instance). The code on that branch is also not up-to-date, I have some newer stuff locally based on 6.3 which has quite a few less patches. Iā€™ve also been trying to upstream some of my kernel changes.

Iā€™m actually quite interested in just using kexec to boot these, i.e. TF-A -> Linux without u-boot (or some minimal shim, maybe coreboot? not sure) and then kexec to the main environment. This would fix the issue that u-boot needs drivers for everything.