[BPI-R3] Debian Bullseye Image

Banana Pi Router design BPI-R3/R3 Mini(MT7986)
1

Hi,

i’ve created a basic debian-image with kernel 6.1.0 (from my main-branch) for sdcard-usage (uboot contains write-functions to flash nand and emmc from usb, files in bpi-r3_uboot.tar.gz), but have not tested again (will do when i get my new board).

login: root pw: bananapi

wan: dhcp-client

lanports: bridged 192.168.1.1/24 + dhcp-server

Network config is done using systemd-networkd (+resolved for dns)…see files in /etc/systemd/network/

wifi: via wifi.sh script in /root (5g needs to set reg-domain first e.g. “iw reg set DE”)

there is still a bug with gpt which is not readable via parted (reports gpt broken) and may cause errors in uboot on reboot (power-cycle works for me).

https://drive.google.com/drive/folders/1fxDR-uL3DIO-8ebgqOJd5MoWORQTTm7Z?usp=sharing

bpi-r3_sdmmc_bullseye.img.gz (+ bpi-r3_sdmmc_bullseye.img.gz.md5)

3 Likes
2

i could fix the gpt-problem with sgdisk (exporting gpt and importing again).

sudo sgdisk --backup=bpi-r3_sgdisk.gpt /dev/sdb
sudo sgdisk --load-backup=bpi-r3_sgdisk.gpt /dev/sdb

after that i can use parted and board boots after reboot too :slight_smile:

edit: found a way to create the gpt via script, but not yet tested if it boots creategpt.sh (1,1 KB)

1 Like
3

Looks good and boots, however there are several kernel modules missing to allow for iptables/nft.

Your default network is using the wan interface, not sure if you want to set it up with a wan bridge to include sfp2 (not eth1) to allow usage of wan or sfp2 for the WAN.

Kernel has a reference link to /media/data_nvme/git/kernel/build for the source. Would rebuild the kernel for the modules, but not sure which repo it’s from.

Does your kernel include the latest MT patches for packet offloading?

4

Kernel-source:

The source link is automaticly created On the host system…so it is invalid on target-system…also i made a out-of-tree build,so there is no source too. I have recently fixed deb creation,but not yet tested the resulting debs. They are built via ci on my repo,but use same defconfig. But you can use ./build.sh build_debs to create debian packes for source too.

R3 is currently basic support, have not added nft yet.

Network-config is basic too…if someone will bridge anything it can be done like this

https://wiki.fw-web.de/doku.php?id=en:linux:systemd#bridge

But sfp2 is designed as lan-sfp so Sfp1 should be bridged to wan if Doing such thing. I know that there are still some problems reported for sfp1 (vlan not working and some sfps not recogized).

Wed is still in bugfixing, hwnat should work if you enable nft+flowtables

edit: have added networking and virtualization-options to r3 defconfig, ci pipeline currently running (takes around 1,5h to complete), then you could replace kernel with the one github-releases

edit: seems like fixing the gpt via sgdisk (or creating it with sgdisk instead of flashing the gpt-img from python-script) solves the reboot-issue (where uboot cannot read the gpt)

Edit2: seems my systemd config pevents adding vlans on wan-port. They are created,but do not work (tagged arp not received on other side). So i added an issue to systemd github

5

Issue with systemd is linux driver issue with mt753x dsa driver.

This patch fixes it:

https://patchwork.kernel.org/project/linux-mediatek/patch/[email protected]/

I also managed to get both wifi configured by systemd

https://drive.google.com/drive/mobile/folders/1WLWAR1FC-rF4n2SgFecBlU1ym_XKqAR_/15Y5Y3NAOwg_IMmN3k6hdb7pAQj9oTVTl/1fxDR-uL3DIO-8ebgqOJd5MoWORQTTm7Z/1kST9ZOv8xQWFfo9GUNIIpKjD8QuYMj8i

6

What happened to the old one?

7

It is still the first version (after fixing the gpt afair)…not yet tested emmc/nand on v1.1

I still have the v1.0 for testing (full system configured and lte card in m.2 slot and both sfp plugged in. My v1.1 has a basic system with systemd config i uploaded to my gdrive,but here mounted wifi cables

8

Congratulations. The image working perfect.

I have question in terms of NVMe . Because the write speed on MMC on my end is ~15MB/sec I decide to combine it with NVMe disk , the speed there is around 300MB/sec which is perfect. My idea was to copy most of the things from /dev/mmcblk0p6 to the NVMe disk. I created couple partitions on the NVMe with idea to mount each of these partitions to directory: etc , home, tmp usr var. I used rsync to copy the content from each directory on mmcblk0p6 to the NVme partitions and finally defined mounts in the /etc/fstab. On first look the things working, the problem is that on boot BPI-R3 use the folders from mmcblk0p6 and then remount these from NVMe. As expected during the boot systemd using the files from mmcblk0p6 and if some service was stopped/disable after normal boot in NVMe it appears again on next reboot. Any idea how can solve this. As I already understood boot from NVMe is not possible , that’s why I tried such workaround.

Thank you,

9

Hi,

Yes boot from nvme is not possible and also access from uboot to it not.

Imho the preferred way would be using lvm volumes. There you can easily extend them later. Configure them in running system is tricky…normally this is done at install,but here we have no real install.

I guess your mounts failing because the mountpoint is not empty…you have to configure lvm and mounts without running this system (maybe booting initrd to make dirs empty and adding mounts for it in fstab).

10

LVM is already in place, I rebuild the kernel and enable LVM support. The mounts not failing , just LVM mounts happen on later stage. Looks like systemd first boot with the files from mmcblk0p6 ,the services start loading then mounts from LVM happen too, which doing re-mount but systemd already loaded the configurations from mmcblk0p6 partition.

11

Mhm,somewhere the config needs to be stored…so moving etc to nvme does not make much sense…maybe there is a way to tell kernel to mount lvm before by defining something in cmdline (uboot).

Only know /var,/home,…also /bin, /usr/bin can be loaded before the mount happens.

12

You only need to have /boot on mmc as uboot doesn’t support nvme. But you can boot with root on nvme, just change bootargs in uboot. Then mount mmc somewhere and install kernels to mmc-mountpoint/boot.

13

Yes,that was the easiest way…don’t thought at it in this moment. Just copy full rootfs to nvme (i guess you already did that) and set “root” var in uEnv.txt to the nvme device and partition

14

Perfect. Thank you very much. I removed the LVM and decide to use entire NVMe. Now bpi-r3 really work very well. Only with MMC was not good.

root@bpi-r3:~# hdparm -Tt --direct /dev/nvme0n1

/dev/nvme0n1:
 Timing O_DIRECT cached reads:   942 MB in  2.00 seconds = 470.62 MB/sec
 Timing O_DIRECT disk reads: 1890 MB in  3.00 seconds = 629.92 MB/sec

root@bpi-r3:~# cat /boot/uEnv.txt
fit=bpi-r3.itb
root=/dev/nvme0n1p1 rootfstype=ext4 rootwait

root@bpi-r3:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       469G  965M  444G   1% /
devtmpfs        993M     0  993M   0% /dev
tmpfs           995M     0  995M   0% /dev/shm
tmpfs           398M  1.6M  397M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/mmcblk0p5  100M  8.3M   92M   9% /boot
tmpfs           199M     0  199M   0% /run/user/0

root@bpi-r3:~# mount
/dev/nvme0n1p1 on / type ext4 (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=1016064k,nr_inodes=254016,mode=755)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=407220k,nr_inodes=819200,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/mmcblk0p5 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=203608k,nr_inodes=50902,mode=700)
15

What was not good with mmc? Sdcards are slower than nvme/ssd,but maybe there is anything we can tune here.

16

I tried to use SanDisk 128GB Extreme microSDXC UHS-I/U3 A2 Micro SD Card. Vendor’s spec say: Up to 190MB/s read spead Up to 90MB/s write speed On bpi-r3 I get write speed ~5MB/sec which is frustrating to use :slight_smile: . Having in mind 512GB NVMe price is 30USD no reason to take bad experience with bpi-r3. It good enough that there are option for workaround with NVMe. Actually the NVMe is cheaper than mmc in ratio usd per gigabyte.

Anyway your Debian image working perfect.

17

To be honest, i have not done speed test on mmc. Settings for it i got from mtk. @ericwoud @dangowrt have you seen similar speed? I try to find some time tomorrow to test with my card.

18

I have not checked sd speed (yet), I was focusing on the emmc hs400 crc errors.

When I was accessing the emmc with sdmmc devicetree settings, I got that write speed of 5MB/s using dd (sequential), indicating that it is communicating with some compatibility mode. With HS200 on emmc I remember write speed is roughly 25 to 30MB/s write using dd (sequential). That felt a bit disapointing, but I do still need to check what to expect. The maximum theoretical data throughput of the eMMC bus in HS200 mode is 200MBytes/s when using 8-bit wide data bus

What speed check program is best to use? Should test for sequencial r/w speeds, but definitely also random r/w speeds.

I recently ran in to this article, also interesting for some BPI boards: It also compares to NVMe in the last row.

Best microSD Cards for the Rock 5B - bret.dk

Compared to the table in the article, random write on SD at 5MB/s is not so bad.

Also note that ext4 is not really best suitable for sd cards. Although it can be tuned to perform a little better. I used the following code for ext4, before I changed over to f2fs.

# https://github.com/bradfa/flashbench.git, running multiple times:
# sudo ./flashbench -a --count=64 --blocksize=1024 /dev/sda
# Shows me that times increase at alignment of 8k
# On f2fs it is used for wanted-sector-size, but sector size is stuck at 512
SD_BLOCK_SIZE_KB=8                   # in kilo bytes
# When the SD card was brand new, formatted by the manufacturer, parted shows partition start at 4MiB
# 1      4,00MiB  29872MiB  29868MiB  primary  fat32        lba
# Also, once runnig on BPIR64 execute:
# bc -l <<<"$(cat /sys/block/mmcblk1/device/preferred_erase_size) /1024 /1024"
# bc -l <<<"$(cat /sys/block/mmcblk1/queue/discard_granularity) /1024 /1024"
SD_ERASE_SIZE_MB=4                   # in Mega bytes

ROOTFS_EXT4_OPTIONS=""
#ROOTFS_EXT4_OPTIONS="-O ^has_journal"  # No journal is faster, but you can get errors after powerloss

  [[ $SD_BLOCK_SIZE_KB -lt 4 ]] && blksize=$SD_BLOCK_SIZE_KB || blksize=4
    stride=$(( $SD_BLOCK_SIZE_KB / $blksize ))
    stripe=$(( ($SD_ERASE_SIZE_MB * 1024) / $SD_BLOCK_SIZE_KB ))
    mkfs.ext4 -v $ROOTFS_EXT4_OPTIONS -b $(( $blksize * 1024 ))  -L "SOMELABEL" \
                    -E stride=$stride,stripe-width=$stripe /dev/XXX
19

There is a way to use iptables on this image today ?

Thanks

20

Afair have not added all needed modules for it to keep my compiletime short but as most is now merged to mainline i can add some to my defconfig and then you can upgrade kernel or create new image with my image-builder repo