Which iperf version do you have? I had limited throughput with default from debian. LRO needed iperf2 in my tests to reach the 9.4 gbit (max)
$ iperf --version:
iperf version 2.1.8 (12 August 2022) pthreads
After dealing with the nfs error:
$ journalctl -b -u nfs-server
Jul 22 06:14:45 nas systemd[1]: Dependency failed for nfs-server.service - NFS server and services.
Jul 22 06:14:45 nas systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.
I was re-compile kernel with the nfs directly in the kernel instead of module, and nfs-server started working.
- dd if=/dev/zero of=/mnt/nvme/test.img bs=10M count=1000 oflag=dsync
10370416640 bytes (10 GB, 9.7 GiB) copied, 39 s, 266 MB/s
-
dd if=/dev/zero of=/mnt/raid0/test.img bs=10M count=1000 oflag=dsync
10349445120 bytes (10 GB, 9.6 GiB) copied, 61 s, 169 MB/s -
dd if=/mnt/nvme/test.img of=/dev/null bs=10M iflag=direct
9961472000 bytes (10 GB, 9.3 GiB) copied, 14 s, 711 MB/s
- dd if=/mnt/raid0/test.img of=/dev/null bs=10M iflag=direct
9856614400 bytes (9.9 GB, 9.2 GiB) copied, 13 s, 757 MB/s
Results between 6.12.32 without rss/lro and 6.16.0 with rss/lro looks not as big as expected, but, I thing that in case of using BPI-R4 as a NAS server:
- writing to NAS means reading data by ethernet/sfp which utilise rss/lro functionality, but is probably limited by
-
- write to the NVME and SATA drives is limited because of the technology - these drivers have to clear the space before writing a new data which makes writing slower, and if I remember it correctly, it reads the data to verify succesfull write.
- reading from NAS drives is much more faster, as expected, and I am little confused here, because I did not set-up smp_affinity for TX frames.
It should make sense, but I have to ask: Does this means, that linux core utilise all cores for TX frames by default?
edit: nfs-server error added
I did test with the 3GB of ram drive:
- dd if=/dev/zero of=/mnt/ramdrive/test.img bs=10M count=300 oflag=dsync
2883584000 bytes (2.9 GB, 2.7 GiB) copied, 5 s, 575 MB/s
- dd if=/mnt/ramdrive/test.img of=/dev/null bs=10M iflag=direct
2170552320 bytes (2.2 GB, 2.0 GiB) copied, 2 s, 1.1 GB/s
Why is writing âonlyâ 50% of the maximum speed?
I assume that nobody knows even an idea. I will perform similar tests on new compiled kernels in the future.
Hello again @frank-w ,
I am wondering how to persist rss & lro setup? I created a script to automate it and I can create âone shotâ service to execute it, but are there any other better options?
When I was playing with it, I realize, that in my performance tests dd with the parameter --oflag=dsync lower the performance a bit, therefore I will update these test soon.
I donât know any specific setting via systemd or any other init. So i guess creating a acript and adding it as postup/predown into the networkd units is the best way.
ls /sys/class/net/sfp-lan/queues/ rx-0 tx-0 tx-1 tx-10 tx-11 tx-12 tx-13 tx-14 tx-15 tx-2 tx-3 tx-4 tx-5 tx-6 tx-7 tx-8 tx-9 one rx vs 10 tx
1 Like
It is not related to the RSS/LRO, but as I am testing everything on nvme or raid arrays, and next tests only on this raid 6 array from these days, I would like share with everyone interested thinking about using bpi-r4 as its own nas. Here is the on-going re-check performance of my raid6 array:
$ cat /proc/mdstat
Personalities : [raid4] [raid5] [raid6]
md127 : active raid6 sde[4] sdf[6] sdd[2] sdb[5] sdc[0] sda[1] sdg[7]
14650675200 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/7] [UUUUUUU]
[==============>......] check = 74.8% (2193851400/2930135040) finish=117.3min speed=104553K/sec
bitmap: 0/22 pages [0KB], 65536KB chunk
I have to say, that (as I created the array about more than 10 years ago) the array have probably NOT well optimalised stripe and stride, therefore performance of re-sync and working in degraded mode is lower. But I think that re-check is going quite fine.
Loadavg is about 2:
And kernel info:
$ uname -a
Linux nas 6.16.0-rc1-bpi-r4-rsslro #1 SMP Wed Aug 13 11:33:51 CEST 2025 aarch64 GNU/Linux
working with a kernel built from 6.16-rsslro. I have two questions.
a) ethtool -K eth0 lro on gets me an error âCould not change any device featuresâ, is there some magic I need first? a kernel CONFIG_* flag? does the order in which ethtool flags are set matter?
b) if I try to set more than two RSS rules [e.g., use loc 2 or loc 3] I get EINVAL. Do I just need to choose the two dst-ips that matter the most for performance, or is there a way I can enable RSS for each VLAN?
Afair the âlro onâ via ethtool was dropped at some state because needs to be defined by dest-ip. So afaik lro is working on layer 3 (mapping fastpath via ip header).
I have only tested with 1 rule,but afaik 4 should be possible
running debian bookworm
root@ratatosk:~# dpkg -l |grep ethtool
ii ethtool 1:6.1-1 arm64 display or change Ethernet device settings
root@ratatosk:~# ethtool -n eth0
4 RX rings available
Total 1 rules
rxclass: Cannot get RX class rules: Message too long
RX classification rule retrieval failed
root@ratatosk:~# ethtool -n ethLAN
4 RX rings available
Total 2 rules
Filter: 0
Rule Type: TCP over IPv4
Src IP addr: 0.0.0.0 mask: 0.0.0.0
Dest IP addr: 192.168.88.253 mask: 255.255.255.255
TOS: 0x0 mask: 0x0
Src port: 0 mask: 0x0
Dest port: 0 mask: 0x0
Action: Direct to queue 0
Filter: 1
Rule Type: TCP over IPv4
Src IP addr: 0.0.0.0 mask: 0.0.0.0
Dest IP addr: 192.168.80.253 mask: 255.255.255.255
TOS: 0x0 mask: 0x0
Src port: 0 mask: 0x0
Dest port: 0 mask: 0x0
Action: Direct to queue 0
root@ratatosk:~# ethtool -N ethLAN flow-type tcp4 dst-ip 192.168.100.253 action 0 loc 2
rmgr: Cannot insert RX class rule: Invalid argument
is there something I did wrong in my attempt to push the rules?
Maybe try the previous version afaik 6.15-rsslro. maybe i/we did something wrong while porting after the sram/irq changes.
testing with 6.14-rsslro.
note there is no 6.15-rsslro branch.
Can try testing with 6.14-rsslro2 next.
root@bananapir4:~# ethtool -N eth0 flow-type tcp4 dst-ip 192.168.89.92 action 0 loc 0
root@bananapir4:~# ethtool -N eth0 flow-type tcp4 dst-ip 192.168.89.91 action 0 loc 1
root@bananapir4:~# ethtool -N eth0 flow-type tcp4 dst-ip 192.168.89.90 action 0 loc 2
rmgr: Cannot insert RX class rule: Invalid argument
root@bananapir4:~# ethtool -n eth0
4 RX rings available
Total 2 rules
Filter: 0
Rule Type: TCP over IPv4
Src IP addr: 0.0.0.0 mask: 0.0.0.0
Dest IP addr: 192.168.89.92 mask: 255.255.255.255
TOS: 0x0 mask: 0x0
Src port: 0 mask: 0x0
Dest port: 0 mask: 0x0
Action: Direct to queue 0
Filter: 1
Rule Type: TCP over IPv4
Src IP addr: 0.0.0.0 mask: 0.0.0.0
Dest IP addr: 192.168.89.91 mask: 255.255.255.255
TOS: 0x0 mask: 0x0
Src port: 0 mask: 0x0
Dest port: 0 mask: 0x0
Action: Direct to queue 0
root@bananapir4:~# uname -a
Linux bananapir4 6.14.0-rc1-edge-filogic #1 SMP Sun Mar 16 16:40:16 UTC 2025 aarch64 GNU/Linux
It will let me use ethtool -N eth0 flow-type tcp4 dst-ip 192.168.89.90 action 1 loc 0 â note sending to queue 1
but ethtool -n eth0 does not show the rule, so I canât prove [trivially] that it works.
6.14-rsslro2 compiling with the same config I used for 6.14-rsslro & 6.16-rsslro did not produce a kernel that sees the network interfaces. itâs 4AM here so I may not fix the config til later.
using the mt7988a_bpi-r4_defconfig [admittedly filtered through the Armbian build framework] still has not produced a usable kernel. itâs now quarter to 5AM EDT.
You should use defconfig in the same branch. In 6.16 the sram was changed from allocating in ethernet driver to dedicated sram-mmio driver. I guess this is missing when you use older defconfig.
Not sure if i tested the 6.16 tree again after rebasing rss/lro patches on the irq/sram changes.
for this post, I did use that branchâs defconfig, as noted filtered through Armbianâs build framework.
Meanwhile, this post [different thread] suggests Iâm barking up the wrong features. Assuming my primary/expected use-case for this board is router, which tree and which CONFIG_ should I be using?
Afair routing part (ppe+flowtable offload) should be mainline. But you have to define a flowtable with nftablesâŚi did this only for hwnat some time ago and not with r4 yet as i have not much 10G equipment for testing routing.
This is for hwnat,but routing should be similar
https://wiki.fw-web.de/doku.php?id=en:bpi-r2:network:nftables#hwnat