[BPI-R3] [BPI-R3-Mini] [BPI-R4] NAND distro-boot + linux-recovery

Banana Pi Router design
1

[ Alpha experimental phase ]

This section is about installing a nand image that contains:

U-Boot that scans emmc/nvme/nand for (/boot)/extlinux/extlinux.conf and boots it. The partition that contains this file needs to have the boot flag set.

If it fails on emmc and nvme, it boots the linux initramdisk on nand. It contains all basic utilities needed to setup an internet connection and install any distro.

There are some basic utilities on the initrd, but perhaps it is still missing some tools and/or kernel modules. It contains my bpi router scripts bpir-build bpir-toolbox, but also: debootstrap wget curl nano parted mkfs-btrfs tar xz gzip zstd, etc, etc.

Note: All images (sdmms/emmc/nvme/uart) all have the same initramdisk. You can interrupt normal boot by keeping ‘x’ pressed during early linux booting. You will then enter a bash shell from initramdisk.

Setup my archlinuxarm image on sd-card, via my script or prebuild image. Another possibility is to use my uartboot image.

After booting from uartboot, first make internet connection with:

bpir-dhcpc <interfacename>

bpir-toolbox needs to download some files needed to build the image.

When running archlinuxarm from sd-card or the initrd from uartboot on the BPI-R3/R3M/R4, you can:

bpir-toolbox --nand-format

This will format and install the image.

I need to add more documentation about ‘bpir-toolbox’, but you can look into the file to see which options to use.

‘bpir-build’ to install archlinuxarm (or experimental ubuntu) on nvme is added pretty recently also, so also needs documentation and testing. Basically the steps are:

‘bpir-build -F’ and go through the menu.

But of course you can just manually use parted and debootstrap to install any other distro. Add extlinux.conf and set the bootflag. You will need to find a suitable linux kernel then also.

If you have used bpir-build to build the nvme/emmc rootfs, you can also use the same tool when running from the initramdisk to enter it via chroot. Just run the command without arguments.

This all needs more testing…

Note: Use archlinuxarm (not ubuntu) for now to build the initramdisk image. There is still a small issue in ubuntu, which is missing the bpi-r3m airoha firmware files in the standard linux-firmware package.

2

For those who are interested, this is the source that does the nand writing magic:



function mtdnr {
  cat /proc/mtd | grep '"'$1'"' | cut -d':' -f1 | tr -d [:alpha:]
}

function ubivol {
  for u in /sys/class/ubi/${ubidev/"/dev/"/}_*/name; do
    if [[ "$(cat $u)" == "$1" ]]; then
      echo $(basename $(dirname $u))
      return
    fi
  done
}

function ubiupdate {
  if ! diff -q "$1" "$2"; then
    echo "Updating "$(basename $1)" on NAND:"
    cp -vf "$1" "$2"
  else
    echo "Skipping "$(basename $1)" on NAND"
  fi
}

function get_ubidev {
  local ubidevice
  ubidevice="$(basename /sys/class/mtd/mtd${1}/ubi*)"
  [ ! -e "/sys/class/ubi/$ubidevice" ] && return 1
  echo "/dev/$ubidevice"
  return 0
}


###############
# Format NAND #
###############
if [[ $@ =~ "--nand-format" ]]; then
  [[ "$(tr -d '\0' 2>/dev/null </proc/device-tree/compatible)" != "bananapi,"* ]] && exit 1
  i=0
  ubinr=$(mtdnr ubi)
  ubidetach -p /dev/mtd${ubinr}
  ubiformat -y /dev/mtd${ubinr}
  ubiattach -p /dev/mtd${ubinr}
  ubidev=$(get_ubidev ${ubinr})
  [[ $? != 0 ]] && exit 1
  [[ "$ubidev" == "/dev/ubi_ctrl" ]] && exit 1
  ubimkvol ${ubidev} -n $i -N fip       -s 1MiB   -t static
  ((i++))
  ubimkvol ${ubidev} -n $i -N ubootenv  -s 128KiB
  ((i++))
  ubimkvol ${ubidev} -n $i -N ubootenv2 -s 128KiB
  ((i++))
  ubimkvol ${ubidev} -n $i -N rootfs    -m
  while [ ! -e "${ubidev}_$i" ]; do sleep 0.1; done
  mkfs.ubifs ${ubidev}_$i
fi


###############
# Update NAND #
###############
if [[ $@ =~ "--nand-update" ]] || [[ $@ =~ "--nand-format" ]]; then
  [[ "$(tr -d '\0' 2>/dev/null </proc/device-tree/compatible)" != "bananapi,"* ]] && exit 1
  ubinr=$(mtdnr ubi)
  bl2nr=$(mtdnr bl2)
  atfbin="/usr/share/bpir-atf/${target}-atf-spim-nand-atf.bin"
  dd if=/dev/mtdblock${bl2nr} of="$tmp/dump.bin" bs=$(du -b "$atfbin" | cut -f1) count=1 >/dev/null 2>&1
  if ! diff "$tmp/dump.bin" "$atfbin"; then
    echo "Updating bl2 on NAND:"
    dd if="$atfbin" of=/dev/mtdblock${bl2nr}
  else
    echo "Skipping bl2 on NAND"
  fi
  ubidev=$(get_ubidev ${ubinr})
  if [[ $? != 0 ]]; then
    ubiattach -p /dev/mtd${ubinr}
    ubidev=$(get_ubidev ${ubinr})
    [[ $? != 0 ]] && exit 1
  fi
  if [[ "$ubidev" == "/dev/ubi_ctrl" ]] || [[ "$ubidev" == "/dev/ubi*" ]]; then exit 1; fi
  ubifip=$(ubivol fip)
  fiptool --verbose create $tmp/fip.bin --nt-fw "/usr/share/bpir-uboot/u-boot-${target}-emmc.bin"
  fiptool info $tmp/fip.bin
  dd if="/dev/${ubifip}" of="$tmp/dump.bin"
  if ! diff "$tmp/dump.bin" "$tmp/fip.bin"; then
    echo "Updating fip on NAND"
    ubiupdatevol "/dev/${ubifip}" $tmp/fip.bin
  else
    echo "Skipping fip on NAND"
  fi
  ubirootfs=$(ubivol rootfs)
  mkdir -p $tmp/mnt
  mount -t ubifs ${ubirootfs} $tmp/mnt
  while ! mountpoint -q $tmp/mnt; do sleep 0.1; done
  mkdir -p $tmp/mnt/boot/extlinux $tmp/mnt/boot/dtbs
  [ -f "${linux}.gz" ] && ubootlinux="${linux}.gz" || ubootlinux="${linux}"
  write_dtb "$tmp/atf.dtb" \
            "root=" \
            "/boot/dtbs/${dtb}.dtb" \
	    /usr/share/buildR64arch/boot/${target^^}/*.dts \
	    /usr/share/buildR64arch/boot/${target^^}-EMMC/*.dts
  ubiupdate "$tmp/atf.dtb" "$tmp/mnt${atfdtb}"
  write_extlinux "$tmp/extlinux.conf" \
                 "${ubootlinux}" \
                 "${initrd}" \
                 "${atfdtb}"
  ubiupdate "$tmp/extlinux.conf" "$tmp/mnt/boot/extlinux/extlinux.conf"
  ubiupdate "$ubootlinux" "$tmp/mnt$ubootlinux"
  ubiupdate "$initrd" "$tmp/mnt$initrd"
  sync
  ls -lR $tmp/mnt
  sync
  while mountpoint -q $tmp/mnt; do umount $tmp/mnt; sleep 0.1; done
fi

Note that this code can still be changed. See:

https://github.com/ericwoud/buildR64arch/blob/main/rootfs/bin/bpir-toolbox

Note that everything in the update function first reads the data and only writes if it is changed. That is done for the mtd device (bl2), also the ubi volume (fip) and also for files on the ubifs (kernel), which is mounted on the rootfs ubi volume.

Small note: I use an ATF that has BL31 included in BL2 image, so you will not find it in the fip image.

3

I wonder where you write the kernel (itb/Image.gz+dtb) and initrd.

I see you create these volumes

ubimkvol ${ubidev} -n $i -N fip       -s 1MiB   -t static
  ((i++))
  ubimkvol ${ubidev} -n $i -N ubootenv  -s 128KiB
  ((i++))
  ubimkvol ${ubidev} -n $i -N ubootenv2 -s 128KiB
  ((i++))
  ubimkvol ${ubidev} -n $i -N rootfs    -m

But ubiupdate (i guess this writes files to the partitions) use some vars i do not see their definition.

4

The script is a bit larger.

Files go like this (after the ubimkvol and mkfs.ubifs calls):

[root@bpir3 ~]# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 07a80000 00020000 "ubi"
mtd1: 00100000 00020000 "bl2"
[root@bpir3 ~]# ubiattach -p /dev/mtd0
UBI device number 0, total 980 LEBs (124436480 bytes, 118.6 MiB), available 0 LEBs (0 bytes), LEB size 126976 bytes (124.0 KiB)
[root@bpir3 ~]# ubinfo /dev/ubi0 -N rootfs
Volume ID:   3 (on ubi0)
Type:        dynamic
Alignment:   1
Size:        943 LEBs (119738368 bytes, 114.1 MiB)
State:       OK
Name:        rootfs
Character device major/minor: 243:4
[root@bpir3 ~]# mkdir /tmp/mnt
[root@bpir3 ~]# mount -t ubifs /dev/ubi0_3 /tmp/mnt
[root@bpir3 ~]# ls -lR /tmp/mnt
/tmp/mnt:
total 0
drwxr-xr-x 4 root root 448 Jun 22 14:42 boot

/tmp/mnt/boot:
total 22768
-rwx------ 1 root root  7758117 Jun 22 14:42 Image.gz
drwxr-xr-x 2 root root      248 Jun 22 14:42 dtbs
drwxr-xr-x 2 root root      232 Jun 22 14:42 extlinux
-rwx------ 1 root root 15549126 Jun 22 14:42 initramfs-bpir.img

/tmp/mnt/boot/dtbs:
total 24
-rw-r--r-- 1 root root 24055 Jun 22 14:42 mt7986a-bananapi-bpi-r3-atf.dtb

/tmp/mnt/boot/extlinux:
total 4
-rw-r--r-- 1 root root 251 Jun 22 14:42 extlinux.conf
[root@bpir3 ~]# cat /tmp/mnt/boot/extlinux/extlinux.conf 
DEFAULT linux-bpir-git
  MENU title U-Boot menu
  PROMPT 0
  TIMEOUT 50
LABEL linux-bpir-git
  MENU LABEL Archlinux AUR package BananaPi Routers
  LINUX /boot/Image.gz
  INITRD /boot/initramfs-bpir.img
  FDT /boot/dtbs/mt7986a-bananapi-bpi-r3-atf.dtb

And now you can copy/overwrite files to /tmp/mnt/ using cp or cat command.

So only Image.gz dtb initrd and extlinux.conf needed for distroboot, no itb. The extlinux.conf can also refer to dtb-overlays to apply, but I have it build to 1 dtb via my script.

BL2 is written directly to /dev/mtdblock1 with dd

UBOOT (+ BL31) is written to:

[root@bpir3 ~]# ubinfo /dev/ubi0 -N fip
Volume ID:   0 (on ubi0)
Type:        static
Alignment:   1
Size:        9 LEBs (1142784 bytes, 1.0 MiB)
Data bytes:  767336 bytes (749.3 KiB)
State:       OK
Name:        fip
Character device major/minor: 243:1

So to /dev/ubi0_0 with ubiupdatevol, but I think you can just dd it as well.

5

Ok,so you have kernel and initrd as files in the rootfs ubi Volume.

6

The ubi rootfs volume is formatted with:

mkfs.ubifs /dev/ubi0_3

Then mount ubifs and copy over the files to the ubifs.

U-Boot’s distro-boot will look for (/boot)/extlinux/extlinux.conf and do the rest. Can specify to U-Boot the name ubi and rootfs, which I did, because the original was UBI and I think it is case dependant.

1 Like
7

Can distroboot also do fit and overlays? Have only seen the Image.gz+dtb way

8

I do not think it does FIT, this is another method. extlinux.conf can do overlays, cmdline and has a menu to choose which kernel (or same kernel with different options) to boot. Distroboot also can use boot.scr instead of extlinux.conf.

This is done to make distroboot possible:

include/configs/mt7xxx.h


/* SPDX-License-Identifier: GPL-2.0 */
/*
 * Configuration for MediaTek MT7622/MT7986 SoC
 *
 */

#ifndef __MT7XXX_H
#define __MT7XXX_H

/* Uboot definition */
#define CFG_SYS_UBOOT_BASE		CONFIG_TEXT_BASE

/* SPL -> Uboot */
#define CFG_SYS_UBOOT_START		CONFIG_TEXT_BASE

/* DRAM */
#define CFG_SYS_SDRAM_BASE		0x40000000

/* This is needed for kernel booting */
#define FDT_HIGH			"0x4fd00000"

#define ENV_MEM_LAYOUT_SETTINGS				\
	"fdt_high=" FDT_HIGH "\0"			\
	"kernel_addr_r=0x44000000\0"			\
	"ramdisk_addr_r=0x48000000\0"			\
	"kernel_comp_addr_r=0x4c000000\0"		\
	"kernel_comp_size=0x03c00000\0"			\
	"scriptaddr=0x4fc00000\0"			\
	"fdt_addr_r=" FDT_HIGH "\0"			\
	"fdtfile=" CONFIG_DEFAULT_FDT_FILE "\0"

#ifdef CONFIG_DISTRO_DEFAULTS

#define BOOT_TARGET_DEVICES(func)	\
	func(MMC, mmc, 1)		\
	func(MMC, mmc, 0)		\
	func(NVME, nvme, 0)		\
	func(UBIFS, ubifs, 0, ubi, rootfs)

#include <config_distro_bootcmd.h>

/* Extra environment variables */
#define CFG_EXTRA_ENV_SETTINGS	\
	ENV_MEM_LAYOUT_SETTINGS		\
	BOOTENV

#endif /* ifdef CONFIG_DISTRO_DEFAULTS*/

#endif

diff --git a/include/configs/mt7622.h b/include/configs/mt7622.h
index 4a056954bf8..c61f7b00794 100644
--- a/include/configs/mt7622.h
+++ b/include/configs/mt7622.h
@@ -9,4 +9,6 @@
 #ifndef __MT7622_H
 #define __MT7622_H
 
+#include "mt7xxx.h"
+
 #endif
diff --git a/include/configs/mt7986.h b/include/configs/mt7986.h
index 090b4c6833e..e75674f69c1 100644
--- a/include/configs/mt7986.h
+++ b/include/configs/mt7986.h
@@ -9,4 +9,6 @@
 #ifndef __MT7986_H
 #define __MT7986_H
 
+#include "mt7xxx.h"
+
 #endif
diff --git a/include/configs/mt7987.h b/include/configs/mt7987.h
index 18ed3c7a55b..bd6f6d0d27a 100644
--- a/include/configs/mt7987.h
+++ b/include/configs/mt7987.h
@@ -11,4 +11,6 @@
 
 #define CFG_MAX_MEM_MAPPED		0xC0000000
 
+#include "mt7xxx.h"
+
 #endif
diff --git a/include/configs/mt7988.h b/include/configs/mt7988.h
index e63825a5a19..616f111aea1 100644
--- a/include/configs/mt7988.h
+++ b/include/configs/mt7988.h
@@ -11,4 +11,6 @@
 
 #define CFG_MAX_MEM_MAPPED		0xC0000000
 
+#include "mt7xxx.h"
+
 #endif