ls /sys/class/net/sfp-lan/queues/ rx-0 tx-0 tx-1 tx-10 tx-11 tx-12 tx-13 tx-14 tx-15 tx-2 tx-3 tx-4 tx-5 tx-6 tx-7 tx-8 tx-9 one rx vs 10 tx
1 Like
It is not related to the RSS/LRO, but as I am testing everything on nvme or raid arrays, and next tests only on this raid 6 array from these days, I would like share with everyone interested thinking about using bpi-r4 as its own nas. Here is the on-going re-check performance of my raid6 array:
$ cat /proc/mdstat
Personalities : [raid4] [raid5] [raid6]
md127 : active raid6 sde[4] sdf[6] sdd[2] sdb[5] sdc[0] sda[1] sdg[7]
14650675200 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/7] [UUUUUUU]
[==============>......] check = 74.8% (2193851400/2930135040) finish=117.3min speed=104553K/sec
bitmap: 0/22 pages [0KB], 65536KB chunk
I have to say, that (as I created the array about more than 10 years ago) the array have probably NOT well optimalised stripe and stride, therefore performance of re-sync and working in degraded mode is lower. But I think that re-check is going quite fine.
Loadavg is about 2:
And kernel info:
$ uname -a
Linux nas 6.16.0-rc1-bpi-r4-rsslro #1 SMP Wed Aug 13 11:33:51 CEST 2025 aarch64 GNU/Linux
working with a kernel built from 6.16-rsslro. I have two questions.
a) ethtool -K eth0 lro on gets me an error “Could not change any device features”, is there some magic I need first? a kernel CONFIG_* flag? does the order in which ethtool flags are set matter?
b) if I try to set more than two RSS rules [e.g., use loc 2 or loc 3] I get EINVAL. Do I just need to choose the two dst-ips that matter the most for performance, or is there a way I can enable RSS for each VLAN?
Afair the “lro on” via ethtool was dropped at some state because needs to be defined by dest-ip. So afaik lro is working on layer 3 (mapping fastpath via ip header).
I have only tested with 1 rule,but afaik 4 should be possible
running debian bookworm
root@ratatosk:~# dpkg -l |grep ethtool
ii ethtool 1:6.1-1 arm64 display or change Ethernet device settings
root@ratatosk:~# ethtool -n eth0
4 RX rings available
Total 1 rules
rxclass: Cannot get RX class rules: Message too long
RX classification rule retrieval failed
root@ratatosk:~# ethtool -n ethLAN
4 RX rings available
Total 2 rules
Filter: 0
Rule Type: TCP over IPv4
Src IP addr: 0.0.0.0 mask: 0.0.0.0
Dest IP addr: 192.168.88.253 mask: 255.255.255.255
TOS: 0x0 mask: 0x0
Src port: 0 mask: 0x0
Dest port: 0 mask: 0x0
Action: Direct to queue 0
Filter: 1
Rule Type: TCP over IPv4
Src IP addr: 0.0.0.0 mask: 0.0.0.0
Dest IP addr: 192.168.80.253 mask: 255.255.255.255
TOS: 0x0 mask: 0x0
Src port: 0 mask: 0x0
Dest port: 0 mask: 0x0
Action: Direct to queue 0
root@ratatosk:~# ethtool -N ethLAN flow-type tcp4 dst-ip 192.168.100.253 action 0 loc 2
rmgr: Cannot insert RX class rule: Invalid argument
is there something I did wrong in my attempt to push the rules?
Maybe try the previous version afaik 6.15-rsslro. maybe i/we did something wrong while porting after the sram/irq changes.
testing with 6.14-rsslro.
note there is no 6.15-rsslro branch.
Can try testing with 6.14-rsslro2 next.
root@bananapir4:~# ethtool -N eth0 flow-type tcp4 dst-ip 192.168.89.92 action 0 loc 0
root@bananapir4:~# ethtool -N eth0 flow-type tcp4 dst-ip 192.168.89.91 action 0 loc 1
root@bananapir4:~# ethtool -N eth0 flow-type tcp4 dst-ip 192.168.89.90 action 0 loc 2
rmgr: Cannot insert RX class rule: Invalid argument
root@bananapir4:~# ethtool -n eth0
4 RX rings available
Total 2 rules
Filter: 0
Rule Type: TCP over IPv4
Src IP addr: 0.0.0.0 mask: 0.0.0.0
Dest IP addr: 192.168.89.92 mask: 255.255.255.255
TOS: 0x0 mask: 0x0
Src port: 0 mask: 0x0
Dest port: 0 mask: 0x0
Action: Direct to queue 0
Filter: 1
Rule Type: TCP over IPv4
Src IP addr: 0.0.0.0 mask: 0.0.0.0
Dest IP addr: 192.168.89.91 mask: 255.255.255.255
TOS: 0x0 mask: 0x0
Src port: 0 mask: 0x0
Dest port: 0 mask: 0x0
Action: Direct to queue 0
root@bananapir4:~# uname -a
Linux bananapir4 6.14.0-rc1-edge-filogic #1 SMP Sun Mar 16 16:40:16 UTC 2025 aarch64 GNU/Linux
It will let me use ethtool -N eth0 flow-type tcp4 dst-ip 192.168.89.90 action 1 loc 0 ← note sending to queue 1
but ethtool -n eth0 does not show the rule, so I can’t prove [trivially] that it works.
6.14-rsslro2 compiling with the same config I used for 6.14-rsslro & 6.16-rsslro did not produce a kernel that sees the network interfaces. it’s 4AM here so I may not fix the config til later.
using the mt7988a_bpi-r4_defconfig [admittedly filtered through the Armbian build framework] still has not produced a usable kernel. it’s now quarter to 5AM EDT.
You should use defconfig in the same branch. In 6.16 the sram was changed from allocating in ethernet driver to dedicated sram-mmio driver. I guess this is missing when you use older defconfig.
Not sure if i tested the 6.16 tree again after rebasing rss/lro patches on the irq/sram changes.
for this post, I did use that branch’s defconfig, as noted filtered through Armbian’s build framework.
Meanwhile, this post [different thread] suggests I’m barking up the wrong features. Assuming my primary/expected use-case for this board is router, which tree and which CONFIG_ should I be using?
Afair routing part (ppe+flowtable offload) should be mainline. But you have to define a flowtable with nftables…i did this only for hwnat some time ago and not with r4 yet as i have not much 10G equipment for testing routing.
This is for hwnat,but routing should be similar
https://wiki.fw-web.de/doku.php?id=en:bpi-r2:network:nftables#hwnat
mainline as of which kernel version?
Looks like 6.6
Hello @frank-w and @tabrisnet
sorry for the delay as I had working on hardware issues with my sfp nas solution… I wrote script which is running as a systemd service to configure rss/lro every startup.
It should work for more IP aliases, and because several issues appeared, I was trying to comment every non-stright solution. Script should produce easy readable output like:
OK irq 104 smp_affinity set to 1
OK irq 105 smp_affinity set to 2
OK irq 106 smp_affinity set to 4
OK irq 107 smp_affinity set to 8
OK rps value set to 0
OK HW LRO RX rule for sfp2 with dest ip 192.168.2.161 created
OK LRO for sfp2 enabled
OK HW LRO RX rule for sfp1 with dest ip 192.168.2.129 created
OK LRO for sfp1 enabled
or also ERROR in case of any problem. rsslro_set.sh (4.7 KB)
Please try it and let me know if it helps or not.
Have you bridged sfps or are subnets small enough to have their own?
Thank you very much for the script,i’m not sure about the lro state in ethtool.
on which kernel-version do you have it running? how are the results of your tests?
I am using 6.16.0-rc1-bpi-r4-rsslro, and bridged are only lan1-3 ports, both sfp and wan are routed.
Hmm, just looked into the script shortly, and instead of parsing dmesg output, I would rather use /sys structure today. I did not know how to work with it when the script is created. 
Here’s my script: BPI-R4 script for Receive-Side-Scaling & interrupt balancing · GitHub
it has issues for sure… but it does use the /sys/devices [and no part of dmesg] and I hope it’s a bit more readable. BUT… it also has an issue or 3 in that it more or less operates via assumed-state rather than rsslro_set.sh’s better state verification.
But I also don’t see how it gets around how many rules you can have. Mind you, my assumptions are much more complex. I’m trying to have half a dozen VLANs.
e.g.
tabris@ratatosk:~$ ip a|grep -A2 ethLAN|egrep '(ethLAN|inet )'
4: ethLAN: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
inet 192.168.88.253/23 brd 192.168.89.255 scope global ethLAN
10: guest@ethLAN: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.80.253/24 brd 192.168.80.255 scope global guest
11: iot@ethLAN: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.82.253/24 brd 192.168.82.255 scope global iot
12: modem@ethLAN: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.100.253/24 brd 192.168.100.255 scope global modem
13: surv@ethLAN: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.81.253/24 brd 192.168.81.255 scope global surv