Hi @frank-w is LRO/RSS etc upstreamed do you know ? and on which kernel version?
thank you
1 Like
Not yet,but recently iāve send a RFC to netdev Mailinglist
https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/
But no comments yet. I have added these patches to my 6.18-rc branch so it will be included in my 6.18-main and my upcoming LTS branch.
3 Likes
Thank you for your post, raised speed from 4.5 gbit\s to 8.4 with this RSS option that you described
LRO is not started but anyway it sound much better
If someone is interested, here is a link to a Linux kernel patch set taken from a recent mtk-openwrt-feeds branch and adjusted for recent Linux branch: linux_6.19-pre-rc1_37bb2e72__mtk-openwrt-feeds_eb48282b_patches_+rss+lro+nft_flow_offload+64bit~almost_stock_mt7996.tar.xz
If Iāll have more time in the future, then maybe Iāll fork the Linux kernel git archive somewhere in public available repository with the patches applied.
- I keep the quilt format since it us used by mtk-openwrt-feeds, thus makes it easier to keep on track as well as cherry pick what is really needed.
- What is applied can be checked by opening the patches/series file which also has some comments; for now most of MediaTek provided mac80211 and mt76 patches are not applied, since even if they were modified to apply, they bring certain problems (e.g. recently since Linux kernel > 6.16 Iāve noticed that WiFi stops receiving traffic just after some time and/or amount of traffic served; I have a nearly full set of MediaTek patches applied to the Linux kernel v6.16-rc1 and working if someone is interested).
- What is tested is given after ā+ā signs in the archive name.
- MLO is not tested as I do not own a station type of device that is compatible with it.
- It uses @frank-w build scripts, yet I provide a trimmed kernel config file that should be just copied to the build directory and if necessary adjusted to some custom needs.
How to apply and build:
cd <path_to_linux_kernel>
git checkout <linux_commit_from_patch_archive_file>
tar -xJf <path_to_patch_archive_file>
quilt push -a
sed -ri 's/^(board=bpi-.*)/#\1/' build.conf
sed -ri 's/^#(board=bpi-r4)/\1/' build.conf
bash build.sh importconfig
cp patches/config ../build/.config
bash build.sh build
bash build.sh pack
Which kernel do you use as base? I got message about requesting for access,please check rights on your gdrive.
I try to send a v3 today. Noticed that the part i dropped in v2 is needed for mt7986 rss and i try to get lro on mt7986 working (current patch seems to break lro on mt7988)
Sorry for the permissions, should be fixed by now.
I use todayās Linux 6.19 (pre-rc1) commit 37bb2e72, I try to keep precise info in the header of patches/series file.
BTW I had to revert your patch about sram (04c7aacc) since it was committed, it was causing all the time:
mt7530-mmio 15020000.switch: Failed to register DSA switch: -517
here is the boot log for reference: debian_6.18-rc1master_1760779590.log (54.0 KB)
Using a base in tovalds master in merge window is no good idea for this,either use rc1 or the last stable.
04c7aacc is no commit in my repo (not found by github,so iām not sure why this is caused. But when you talk about sram you have to use dts and driver code from mainline and not simply overwrite with sdk files. The sram handling is different in mainline (dedicated sram node) and sdk (uses second reg property in eth node).
Currently i have some issues with mt7622 and non rss/lro related patches. 1 is after 6.18 (mmc probe order) and one is needed for mt7988,but breaks mt7622 network.
I was referring to 04c7aacc + 65f0e397 from Linux repository (btw, that is one of the reasons I do not use your BPI-Router-Linux repo: it is not a fork of the torvalds repo, but that is your choice, same as mine is to stick to the upstream version and not to a specific release or rc tag).
Here is a patch set for linux v6.18 tag: linux_6.18_7d0a66e4__mtk-openwrt-feeds_eb48282b_patches_+wed+gfp_dma32+rss+lro+nft_flow_offload~almost_stock_mt7996.tar.xz
The set of patches is tested with BPI-R4+BPI-BE14. I canāt tell much about mt7622, I do not own that hardware.
The hardware LRO in MT7988A is more like a marketing gimmick rather than a generally useful offload feature.
The limitations of hardware LRO in MT7988A (flagship SoC from MediaTek):
- limited to 2 IP addresses which you have to manually set up by hacking into so called āflow-tableā (flow redirector) through ethtool
- possible to extend to support 4 IP addresses by changing the code (which is not whatās code in whatsoever patches you could find from MTK feed)
- the IP addresses have to terminate on the router; nothing useful for forwarding traffic
- one application I can think of is BPI-R4 as a NAS server which is likely will benefit from hardware LRO but nothing else
So I would recommend focus the energy on upstreaming something else than hardware LRO.
The hardware LRO in MT7988A is very restrictive and limitedā¦
1 Like
If you have a patch for extending functionality iām open,but i have only 1 other device sending/receiving 10G traffic so i cannot test routing (which should already be mainline via ppe). RSS and LRO were the only way to get the 10G terminating traffic working. Yes it is limited to 4 adresses (i only used one).
@matt1606 yes my repo is (no more) a fork of torvalds,but created branches are always having it as base till stable release,then stable tree. The rc branch is mostly based on rc1ā¦also torvalds/linux. It does not need to be a fork. I do not like rolling release as it prevents testing against older versions for finding bugs
I donāt have a patch unfortunately. I spent 15 mins trying and gave up for the reasons I posted above. Apparently itās not a simple change of redefining a MACRO from 2 to 4.
RSS is crucial. LRO is not. I believe RSS is sufficient for hitting 10G for forwarding and terminating in the router.
regarding the MTK_MAX_LRO_IP_CNT const i got an answer from mtk:
The original design was likely due to the fact that we only have four DIP registers (LRO_RINGX_DIP_DWX), and these four DIP registers are shared among eth0/1/2. Thatās why the mtk_hwlro_add_ipaddr_idx function searches for an available DIP register. This design simply reserves two IP addresses for each netdev (hwlro_ip), but in reality, any IP address written to a DIP register will take effect.
and only with LRO i was able to reach the 10GBit/s on R4ā¦
regarding the first 1-2 iperf3 rounds are not reaching the 10GBit/sā¦third round does
HWLRO does require some startup time. The register LRO2_CTRL_DW2 is the minimum threshold to become a candidate for LRO aggregation.
1 Like
Here is a more complete patch set with lots of more applied than previously: linux_6.19-rc5_0f61b186__mtk-openwrt-feeds_42be1aec_patches.tar.xz
- for now I can say that it booted and networking is working =)
Iām astonished it went so fast this time, only two evenings spent on upstreaming the mtk-openwrt-feeds patches to linux v6.19-rc5, it is like MediaTek picked up my earlier patches, but never mentioned me as a contributor 
Guys, what is the present state of RSS/LRO? @frank-w are your kernels enabled with this by default?
Is you kernel 7.* ok for the non PRO version?
Kernel itself is for all boards. Lro/RSS is part of latest branches,but not yet tested on 7.x
For upstreaming some AI messages have to be answered and fixed. Got some informtion from mtk, but some issues are still unresolved (e.g. some corner cases not clean in current state).
1 Like
HW LRO on MT7988A: the engine picks up only a tiny fraction of a learned flow, and every pickup causes a TCP retransmit
I have been running this series (via frank-wās BPI-Router-Linux 7.1-main, which carries it) on a BPI-R4 Pro (MT7988A) router and measured HW LRO behaviour rather than throughput. This is not a race condition as far as I can tell. The problem is related to packet ordering, inside the hardware.
To reproduce: one TCP stream from a 10G host to the board, board receiving, about 4.5 Gbit/s and 380 000 segments per second. One LRO rule for the boardās own IP, programmed with ethtool -N.
What I see:
- The flow appears in
hw_lro_auto_tlbwithin seconds and stays there for the whole test. Total agginhw_lro_statsreaches only 0.002 ā 0.05 % of the flowās segments. The other 99.95 % arrive on the RSS rings.- With the rule programmed, the sender retransmits 13 ā 218 segments per GiB. Without it, 0.4 ā 1.9. The extra retransmits equal
Total agg. No error or drop counter moves.
What I believe happens: the engine holds up to 8 segments (the 12 KiB SDL) to build a superframe. While it holds them, the next segments of the same flow go straight to an RSS ring and reach TCP first. TCP sees a hole the size of one superframe and the sender retransmits it. When the superframe arrives, it is a duplicate. If the engine took every segment of the flow there would be no hole. It takes 0.02 %.
Question for MediaTek: after a flow is learned, should all of its segments go to the LRO ring? If yes, what makes the engine skip 99.95 % of them? If no, then TCP is reordered by design and HW LRO cannot be used for traffic terminating on the board.
To test it, on the board (eth0, address 192.168.1.1):
ethtool -K eth0 lro on
ethtool -N eth0 flow-type tcp4 dst-ip 192.168.1.1 loc 0
echo "5 1" > /proc/mtketh/hw_lro_auto_tlb
echo > /proc/mtketh/hw_lro_stats
On the peer: iperf3 -c 192.168.1.1 -t 60. It prints the senderās retransmits and the bytes sent. Then on the board:
grep -A1 "^Total agg" /proc/mtketh/hw_lro_stats
Total agg is the number of segments the LRO rings took; bytes Ć· 1448 is the number the flow had. Repeat after ethtool -N eth0 delete 0: retransmits go back to almost zero.
Measured so far on a BPI-R4 Pro through the DSA conduit (tagged frames).