[BPI-R4] TODO List

[jcdutton]

Hi,

I think most things now work on vanilla 6.9.x kernel (with some patches) for the BPI-R4. For supporting BPI-R4 on a vanilla 6.9.x kernel what else is needed, i.e. what is the definition of done?:

1. Network RSS (Attempting to get iperf3 UDP > 1 Gbps, TCP > 5 Gbps and closer to 10 Gbps)

Note: I tried working at this item but have given up. Someone else is welcome to try.

Note: RSS uses a hash to spread the incoming traffic across more than one CPU. The hash is only different if the src/destination IP address and TCP port are different. So on its own it will not improve iperf3, but it will mean one can run 2 iperf3 at the same time, on different ports, to get two tcp stream or 5 Gbps totaling 10 Gbps. I don’t know if mediatek’s implementation also understands VLANs or not.

2. Network LRO (Attempting to get iperf3 TCP > 5 Gbps, and closer to 10 Gbps)
3. UBOOT change so that Linux kernel is more portable across platforms.
4. General SFP improvements.
5. Wifi (BE14 Readily available, BE19 Status Unknown)
6. GPIO - Control of the GPIO Header features.
7. Upstream the above so they get into mainline kernel
8. hw qos/speed limit support. It seems should be implemented using tc flower api. Downstream implementation here: autobuild_mac80211_release/target/linux/mediatek/patches-5.4/999-3012-flow-offload-add-mtkhnat-qdma-qos.patch - openwrt/feeds/mtk-openwrt-feeds - Gitiles
9. Hardware flow-offloading if using all 4 GiB of RAright now it only works with mem=2048M appended to kernel cmdline)
10. pcie@0x11280000 doesn’t work (probably missing clock)
11. Ethernet driver still misses RSS and LRO for better RX-to-CPU performance (eg. when running iperf3) by aggregating RX packets
12. No driver for TOPS yet (ie. no hardware tunnel encap/decap offloading)
13. No driver for EIP-197 yet (no IPSec, TLS or DTLS in hardware, all work has to be done by CPU)
14. Only one out of three PPE is being used
15. ebpf xdp support
16. DS-Lite, 6RD, MAP-E/T and 464xlat hardware offload support
17. Anything else.

[frank-w]

@jcdutton

Network LTO (Attempting to get iperf3 TCP > 5 Gbps, and closer to 10 Gbps)

You mean LRO (large receive offload),as daniel told me it is a receive issue not a transmit one. But he is also working on it,told me today)

My last state for lro was this (upported patches from @balika011): Commits · frank-w/BPI-Router-Linux · GitHub

For wifi i created this thread:

I have not yet verified gpio header,but biggest parts are listed so far…and of course the upstreaming which is much because code mostly has to be changed to get it accepted. @dangowrt do a great work here.

Definition of done in my eyes if the changes work stable and are upstreamed…maybe backported to openwrt kernels.

[totkeks (Norman Dankert)]

@jcdutton great idea. Could you link to the respective forum threads or single posts for each of those topics? Frank already listed two in his reply, I think. Then this could be a good overview if pinned.

@frank-w can you pin this post or do we need someone like @simon or @sinovoip to do it?

[frank-w]

I have not found a way to pin a thread.

[Thanks]

Anything else by @dangowrt

@dangowrt

However, there are still a few problems which will have to be addressed before vanilla OpenWrt is ready for production on the BPi-R4:

• hardware flow-offloading if using all 4 GiB of RAright now it only works with mem=2048M appended to kernel cmdline)
• pcie@0x11280000 doesn’t work (probably missing clock)
• Ethernet driver still misses RSS and LRO for better RX-to-CPU performance (eg. when running iperf3) by aggregating RX packets
• no driver for TOPS yet (ie. no hardware tunnel encap/decap offloading)
• no driver for EIP-197 yet (no IPSec, TLS or DTLS in hardware, all work has to be done by CPU)
• only one out of three PPE is being used

To give the things which already work more testing it makes more sense to continue development on the main branch (which apart from providing us with downloadable binaries also reduces my workload of constantly doing git rebase origin/main on that large series)

[frank-w]

Pcie2 is working now in linux (patch should be already merged)…clock depends on pcie3.

[ericwoud (Eric W.)]

@jcdutton

GPIO - Control of the GPIO Header features.

Just control the state of the gpio pins?

On linux nowadays use libgpiod, for example as in here:

GPIO Programming: Exploring the libgpiod Library | ICS

[frank-w]

I have upported dma handling from sdk and rss/lro as far as i had to 6.9

Commits · frank-w/BPI-Router-Linux

Only checked if dma-patch does not break mt7988 (iperf good now),not yet tested rss/lro (upported patch from @balika011),but in older version there was something missing,but it’s a start

[dangowrt (Daniel Golle)]

@frank-w I’d love to import the patches from your tree to OpenWrt but can’t because they are missing a Signed-off-by: line. Can you add that to each commit currently missing it?

[dangowrt (Daniel Golle)]

@ericwoud

Just control the state of the gpio pins?

pinctrl should allow to use all pins as GPIOs. Maybe some groups will have to be split for more granularity, but generally it should all work.

[frank-w]

net: mediatek: make MTK_DMA_SIZE soc specific

and

arm64: dts: make pwm-fan work on r4

have now SoB…the other 2 are not really tested (upported patches for RSS/LRO)

[jcdutton]

In my initial work on RSS, I have found a lot to problems with the current drivers for 6.9-rc1. So, I think I am going to try to fix those before finally doing RSS.

In summary, If the SOC supports netsys_V2 or above, in some places it is trying to use V1 registers with a V2 SOC and in other places it is using V2 registers with a V2 SOC.

We need to fix this all so that the driver uses V2 registers with a V2 SOC and visa-versa. So that it matches the openwrt drivers that correctly does this. Unfortunately, the openwrt does it in such a way that it is unlikely to be able to be up-streamed to mainline.

[jcdutton]

I notice that the current mtk_eth_soc.c just uses the device tree to detect which chip it is. Is there not a more robust way to detect which chip we have by reading an ID register or similar?

[frank-w]

@jcdutton

I have found a lot to problems with the current drivers for 6.9-rc1.

@jcdutton

In summary, If the SOC supports netsys_V2 or above, in some places it is trying to use V1 registers with a V2 SOC

Where do you see this? Maybe v2 has also v1 registers?

Have you looked in my code? Balika has done some cleanups before rss/lro

Dma patch only tested on r4 over switch yet

[Stat_headcrabed (Shengyu Qu)]

Is bridging between 2xSFP+s accelerated by hardware?

[Stat_headcrabed (Shengyu Qu)]

Currently enablement of all 3 PPEs and above 2G ram support is being worked on here: GitHub - eladwf/openwrt at bpi_r4_ppe

[EasyNet]

Hi @jcdutton

What did you do for the compilation of the Kernel to make it work? I was trying 6.9.9 vanilla, @frank-w 6.9.0 git kernel and all of them are not booting at all on BPI-R4.

I was using this procedure:

1. Build the Kernel
2. Create a .its file:

/dts-v1/;

/ {
        description = "EasyOS Banana Pi R4 MT7988 FIT Image";
        #address-cells = <1>;

        images {
                kernel-1 {
                        description = "EasyOS Linux Kernel 6.9.0 BPi-R4";
                        data = /incbin/("Image.gz-6.9.0-bpi-r4");
                        type = "kernel";
                        arch = "arm64";
                        os = "linux";
                        compression = "gzip";
                        load = <0x44000000>;
                        entry = <0x44000000>;
                };
        };

        configurations {
                default = "conf-1";
                opt = "conf-2";

                conf-1 {
                        description = "EasyOS Standard Boot";
                        kernel = "kernel-1";
                        arch = "arm64";
                };
        };

};

3. make a uImage file:

mkimage -E -B 0x1000 -p 0x1000 -f Linux-6.9.0-bpi-r4.its uImage-6.9.0-bpi-r4

4. Try to boot the OS from U-boot using these parameters:

# Keep 32MB space between Kernel and FTD for bigger kernels.
setenv kaddr 0x46000000
setenv dtaddr 0x48000000

bootp ${dtaddr} 10.42.0.1:6.9.0-bpi-r4/mediatek/mt7988a-bananapi-bpi-r4.dtb
bootp ${kaddr} 10.42.0.1:uImage-6.9.0-bpi-r4
fdt addr ${dtaddr}
iminfo ${kaddr}
bootm ${kaddr} - ${dtaddr}

As soon as the message Starting kernel ... appears the system is freezing.

MT7988> setenv consoleargs "earlycon=uart8250,mmio32,0x11000000 console=ttyS0,115200n1 earlyprintk debug=7 loglevel=8 pci=pcie_
bus_perf consoleblank=0"
MT7988> setenv rootfs "root=/dev/mmcblk0p6 rootfstype=ext4 rootwait"
MT7988> setenv bootargs $consoleargs $rootfs
MT7988> 
MT7988> # Keep 32MB space between Kernel and FTD.
MT7988> setenv kaddr 0x46000000
MT7988> setenv dtaddr 0x48000000
MT7988> bootp ${dtaddr} 10.42.0.1:6.9.0-bpi-r4/mediatek/mt7988a-bananapi-bpi-r4.dtb
BOOTP broadcast 1
DHCP client bound to address 10.42.0.145 (2 ms)
Using ethernet@15100000 device
TFTP from server 10.42.0.1; our IP address is 10.42.0.145
Filename '6.9.0-bpi-r4/mediatek/mt7988a-bananapi-bpi-r4.dtb'.
Load address: 0x48000000
Loading: ##
         13.3 MiB/s
done
Bytes transferred = 27815 (6ca7 hex)
MT7988> bootp ${kaddr} 10.42.0.1:uImage-6.9.0-bpi-r4
BOOTP broadcast 1
DHCP client bound to address 10.42.0.145 (3 ms)
Using ethernet@15100000 device
TFTP from server 10.42.0.1; our IP address is 10.42.0.145
Filename 'uImage-6.9.0-bpi-r4'.
Load address: 0x46000000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #############
         19.9 MiB/s
done
Bytes transferred = 8773632 (85e000 hex)
MT7988> iminfo ${kaddr}

## Checking Image at 46000000 ...
   FIT image found
   FIT description: EasyOS Banana Pi R4 MT7988 FIT Image
    Image 0 (kernel-1)
     Description:  EasyOS Linux Kernel 6.9.0 BPi-R4
     Type:         Kernel Image
     Compression:  gzip compressed
     Data Start:   0x46001000
     Data Size:    8768737 Bytes = 8.4 MiB
     Architecture: AArch64
     OS:           Linux
     Load Address: 0x44000000
     Entry Point:  0x44000000
    Default Configuration: 'conf-1'
    Configuration 0 (conf-1)
     Description:  EasyOS Standard Boot
     Kernel:       kernel-1
## Checking hash(es) for FIT Image at 46000000 ...
   Hash(es) for Image 0 (kernel-1): 
MT7988> bootm ${kaddr} - ${dtaddr}
## Loading kernel from FIT Image at 46000000 ...
   Using 'conf-1' configuration
   Trying 'kernel-1' kernel subimage
     Description:  EasyOS Linux Kernel 6.9.0 BPi-R4
     Type:         Kernel Image
     Compression:  gzip compressed
     Data Start:   0x46001000
     Data Size:    8768737 Bytes = 8.4 MiB
     Architecture: AArch64
     OS:           Linux
     Load Address: 0x44000000
     Entry Point:  0x44000000
   Verifying Hash Integrity ... OK
## Flattened Device Tree blob at 48000000
   Booting using the fdt blob at 0x48000000
Working FDT set to 48000000
   Uncompressing Kernel Image to 44000000
   Loading Device Tree to 00000000ff7ed000, end 00000000ff7f6ca6 ... OK
Working FDT set to ff7ed000

Starting kernel ...

I was able to boot it without issue with GitHub - BPI-SINOVOIP/BPI-R4-bsp-6.1 but patched to be able to compile the kernel using make ARCH=arm64 O=build-mt7988 and I have my repository here: GitHub - EasyNetDev/BPI-R4-bsp-6.1

But I have no luck with 6.9.x. For 6.1.73 I did the same procedure just to load the DTB for 6.1.73

[frank-w]

You did strange things…when using my repo just change build.conf to board=bpi-r4 (uncomment by removing the sharp)

Then use build.sh and it does all the magic

./build.sh importconfig
./build.sh config
./build.sh

Mainline kernel misses some patches like pinctrl to get it working

[jcdutton]

I use my own kernel. It has many of Frank’s patches in it.

GitHub - jcdutton/linux-bpi at v6.9-rc1-bpi

Use the v6.9-rc1-bpi branch. I build my kernel slightly differently from Frank. Look at the arm64a.sh and arm64b.sh for a guide. My workflow covers arm, riscv and x86, so I wanted the workflow to be as similar as possible across them all and as similar as possible to the way mainline kernel developers work. Its is just my preference. Frank’s kernels should work just as well.

[EasyNet]

I just did traditional steps for kernel compilation:

export BUILD=build-mt7988
make ARCH=arm64 mrproper
make O=${BUILD}/ ARCH=arm64 mt7988a_bpi-r4_defconfig
rm -rf ${BUILD}/INSTALL && make O=${BUILD}/ ARCH=arm64 -j24 && make O=${BUILD}/ ARCH=arm64 INSTALL_PATH=INSTALL INSTALL_MOD_PATH=INSTALL INSTALL_DTS_PATH=INSTALL INSTALL_HDR_PATH=INSTALL/headers all headers_install dtbs_install modules_install Image install zinstall -j24

Then I took the Image and used with .its script to build the final uImage. Same I did with 6.1.73. I have no clue if your build.sh script is still applying any additional patches. I will give a try using the script to see if I’m getting any results.