Hi @frank-w is LRO/RSS etc upstreamed do you know ? and on which kernel version?
thank you
1 Like
Not yet,but recently iāve send a RFC to netdev Mailinglist
https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/
But no comments yet. I have added these patches to my 6.18-rc branch so it will be included in my 6.18-main and my upcoming LTS branch.
3 Likes
Thank you for your post, raised speed from 4.5 gbit\s to 8.4 with this RSS option that you described
LRO is not started but anyway it sound much better
If someone is interested, here is a link to a Linux kernel patch set taken from a recent mtk-openwrt-feeds branch and adjusted for recent Linux branch: linux_6.19-pre-rc1_37bb2e72__mtk-openwrt-feeds_eb48282b_patches_+rss+lro+nft_flow_offload+64bit~almost_stock_mt7996.tar.xz
If Iāll have more time in the future, then maybe Iāll fork the Linux kernel git archive somewhere in public available repository with the patches applied.
- I keep the quilt format since it us used by mtk-openwrt-feeds, thus makes it easier to keep on track as well as cherry pick what is really needed.
- What is applied can be checked by opening the patches/series file which also has some comments; for now most of MediaTek provided mac80211 and mt76 patches are not applied, since even if they were modified to apply, they bring certain problems (e.g. recently since Linux kernel > 6.16 Iāve noticed that WiFi stops receiving traffic just after some time and/or amount of traffic served; I have a nearly full set of MediaTek patches applied to the Linux kernel v6.16-rc1 and working if someone is interested).
- What is tested is given after ā+ā signs in the archive name.
- MLO is not tested as I do not own a station type of device that is compatible with it.
- It uses @frank-w build scripts, yet I provide a trimmed kernel config file that should be just copied to the build directory and if necessary adjusted to some custom needs.
How to apply and build:
cd <path_to_linux_kernel>
git checkout <linux_commit_from_patch_archive_file>
tar -xJf <path_to_patch_archive_file>
quilt push -a
sed -ri 's/^(board=bpi-.*)/#\1/' build.conf
sed -ri 's/^#(board=bpi-r4)/\1/' build.conf
bash build.sh importconfig
cp patches/config ../build/.config
bash build.sh build
bash build.sh pack
Which kernel do you use as base? I got message about requesting for access,please check rights on your gdrive.
I try to send a v3 today. Noticed that the part i dropped in v2 is needed for mt7986 rss and i try to get lro on mt7986 working (current patch seems to break lro on mt7988)
Sorry for the permissions, should be fixed by now.
I use todayās Linux 6.19 (pre-rc1) commit 37bb2e72, I try to keep precise info in the header of patches/series file.
BTW I had to revert your patch about sram (04c7aacc) since it was committed, it was causing all the time:
mt7530-mmio 15020000.switch: Failed to register DSA switch: -517
here is the boot log for reference: debian_6.18-rc1master_1760779590.log (54.0 KB)
Using a base in tovalds master in merge window is no good idea for this,either use rc1 or the last stable.
04c7aacc is no commit in my repo (not found by github,so iām not sure why this is caused. But when you talk about sram you have to use dts and driver code from mainline and not simply overwrite with sdk files. The sram handling is different in mainline (dedicated sram node) and sdk (uses second reg property in eth node).
Currently i have some issues with mt7622 and non rss/lro related patches. 1 is after 6.18 (mmc probe order) and one is needed for mt7988,but breaks mt7622 network.
I was referring to 04c7aacc + 65f0e397 from Linux repository (btw, that is one of the reasons I do not use your BPI-Router-Linux repo: it is not a fork of the torvalds repo, but that is your choice, same as mine is to stick to the upstream version and not to a specific release or rc tag).
Here is a patch set for linux v6.18 tag: linux_6.18_7d0a66e4__mtk-openwrt-feeds_eb48282b_patches_+wed+gfp_dma32+rss+lro+nft_flow_offload~almost_stock_mt7996.tar.xz
The set of patches is tested with BPI-R4+BPI-BE14. I canāt tell much about mt7622, I do not own that hardware.
The hardware LRO in MT7988A is more like a marketing gimmick rather than a generally useful offload feature.
The limitations of hardware LRO in MT7988A (flagship SoC from MediaTek):
- limited to 2 IP addresses which you have to manually set up by hacking into so called āflow-tableā (flow redirector) through ethtool
- possible to extend to support 4 IP addresses by changing the code (which is not whatās code in whatsoever patches you could find from MTK feed)
- the IP addresses have to terminate on the router; nothing useful for forwarding traffic
- one application I can think of is BPI-R4 as a NAS server which is likely will benefit from hardware LRO but nothing else
So I would recommend focus the energy on upstreaming something else than hardware LRO.
The hardware LRO in MT7988A is very restrictive and limitedā¦
1 Like
If you have a patch for extending functionality iām open,but i have only 1 other device sending/receiving 10G traffic so i cannot test routing (which should already be mainline via ppe). RSS and LRO were the only way to get the 10G terminating traffic working. Yes it is limited to 4 adresses (i only used one).
@matt1606 yes my repo is (no more) a fork of torvalds,but created branches are always having it as base till stable release,then stable tree. The rc branch is mostly based on rc1ā¦also torvalds/linux. It does not need to be a fork. I do not like rolling release as it prevents testing against older versions for finding bugs
I donāt have a patch unfortunately. I spent 15 mins trying and gave up for the reasons I posted above. Apparently itās not a simple change of redefining a MACRO from 2 to 4.
RSS is crucial. LRO is not. I believe RSS is sufficient for hitting 10G for forwarding and terminating in the router.
regarding the MTK_MAX_LRO_IP_CNT const i got an answer from mtk:
The original design was likely due to the fact that we only have four DIP registers (LRO_RINGX_DIP_DWX), and these four DIP registers are shared among eth0/1/2. Thatās why the mtk_hwlro_add_ipaddr_idx function searches for an available DIP register. This design simply reserves two IP addresses for each netdev (hwlro_ip), but in reality, any IP address written to a DIP register will take effect.
and only with LRO i was able to reach the 10GBit/s on R4ā¦
regarding the first 1-2 iperf3 rounds are not reaching the 10GBit/sā¦third round does
HWLRO does require some startup time. The register LRO2_CTRL_DW2 is the minimum threshold to become a candidate for LRO aggregation.
1 Like
Here is a more complete patch set with lots of more applied than previously: linux_6.19-rc5_0f61b186__mtk-openwrt-feeds_42be1aec_patches.tar.xz
- for now I can say that it booted and networking is working =)
Iām astonished it went so fast this time, only two evenings spent on upstreaming the mtk-openwrt-feeds patches to linux v6.19-rc5, it is like MediaTek picked up my earlier patches, but never mentioned me as a contributor 
Guys, what is the present state of RSS/LRO? @frank-w are your kernels enabled with this by default?
Is you kernel 7.* ok for the non PRO version?
Kernel itself is for all boards. Lro/RSS is part of latest branches,but not yet tested on 7.x
For upstreaming some AI messages have to be answered and fixed. Got some informtion from mtk, but some issues are still unresolved (e.g. some corner cases not clean in current state).
1 Like
I have been running this series (via frank-wās BPI-Router-Linux 7.1-main, which carries it) on a BPI-R4 Pro (MT7988A) router and measured HW LRO behaviour rather than throughput. The results are reproducible, and I believe I can now describe the failure mechanism, not just the symptom. A throughput test cannot see any of this: software GRO delivers the same iperf numbers whether or not the LRO engine engages.
Setup: single-stream terminating TCP, iperf3 -R over a 10G SFP+ link, 4.3-4.7 Gbit/s, ~380 kpps, MSS 1448. One DIP rule armed via ethtool for the receiverās address, NETIF_F_LRO on. Software counters from the SDK debugfs (not part of this series) plus a small per-ring packet counter patch, verified against the hardware flush counters.
1) The flow is promoted and ring-bound, but almost never steered.
With the rule armed, the flow appears in the auto-learn table within seconds and stays there for entire runs. The table position is meaningful: the flow always occupies entry 1 and its superframes flush exclusively on ring 4; in one run a fourth flowās appearance at entry 4 coincided with ring 7 receiving its first packets ever, and when flows end the surviving entries compact toward entry 1. So on netsys v3 the low half of the table is the active set and entry n is bound to LRO ring 4+(n-1) - one flow per ring, which matches the original mechanism description in the 2016 mainline submission (āone flow is mapped to one RX ringā, selection āby throughput comparisonā only when candidates exceed rings; [email protected], Sep 2016). Note the v2/v3 debugfs dump prints āongingā/ācandidateā purely from the index; the v3 table entry read through LRO_ALT_DBG is eleven words of flow identity with no score or state field, so the dump labels are not evidence of anything - the ring binding above is.
Yet the bound ring receives between 0.002% and 0.05% of the flowās segments, and the rate is erratic: across five identical 60 s measurement windows, same register configuration, offered load constant at 379-385 kpps, the ingested-segment rate spanned 8.2 to 116.8 per second - a factor of 14. Nothing I could set from the driver moved it out of that band. Aggregation is flawless for the segments that do arrive: 54% of flushes fill to the SDL length cap, AGE timeout never fires, AGG timeout fires in ~1% of flushes.
2) The per-packet comparator is live, and correctly configured.
Clearing MTK_LRO_VLAN_EN and MTK_LRO_VLAN_VID_CMP_DEPTH at runtime takes ingestion to exactly zero - flush and aggregation counters do not advance at all over 45 s and 17.3 M packets - and restoring them brings it back. The traffic is DSA-tagged (tag_8021q VID plus a bridge VLAN), so the VLAN comparison stage is on the per-packet path and is matching. Sweeping CMP_DEPTH over 0ā¦3 has no reproducible effect (depth 1 was the highest rate in one run and the lowest in the replicate). Setting MTK_NON_LRO_MULTI_EN, which no vendor generation sets on this part, and clearing MTK_LRO_L4_CTRL_PSH_EN both land inside the spread of interleaved pristine controls in the same run.
So the flow is promoted, the ring is bound, the learned entry matches, the comparator is demonstrably evaluating the packets - and the packets still go to the RSS rings.
3) Every steered superframe costs one spurious retransmit.
The rest of the flow overtakes the held superframe on the RSS rings through separate NAPI contexts, the receiver sees a hole exactly one superframe wide, the sender fast-retransmits all of it, one DSACK. Across ten measurement windows in which the steering rate varied 24x by natural drift at fixed configuration, receiver-side DSACKs sent track flush events at r=0.927 (one DSACK per hole; the retransmit burst is coalesced by receive-side GRO into a single skb). Measured from the sender instead, excess retransmits match segments per superframe, which is the same relation counted per segment rather than per hole; that form held across seven runs spanning RSS on/off, BW threshold 100/3000/200000, a GLO_MEM mirror and a CRSN filter experiment. In absolute terms: 13-218 retransmits/GiB with the rule armed against a 0.4-1.9/GiB floor with it disarmed, with zero loss anywhere - frame engine drop counters and all IP/TCP error counters stay at zero.
The corollary is that engaging the engine harder is not a workaround. Any stimulus that raises ingestion raises the retransmit cost with it.
4) What I ruled out, each by a controlled A/B or register readback:
- RSS arbitration: with MTK_RSS_EN held clear the split persists and LRO engagement drops 5x.
- Interrupt moderation: the LRO rings read back EN=0.
- Timers: AGE never fires, AGG ~1%.
- BW threshold: both 100 and 200000 reduce engagement vs 3000.
- CRSN filter: clearing MTK_LRO_CRSN_BNW and LRO_CTRL_DW1 kills aggregation entirely, so the BNW+0xffffffff default is a whitelist admitting everything - and that run shows armed-with-zero-ingestion costs exactly the disarmed floor, attributing the entire retransmit elevation to packets entering the aggregator.
- Comparator bits: as in (2) above.
- Programming interface: GLO_MEM is gated on MTK_GLO_MEM_ACCESS, which only MT7987 carries in every SDK generation including 21.02; MT7988 is programmed through the direct registers this series uses, everywhere. I also verified the end state this series programs is register-identical to the 21.02 (5.4) production SDK driver for MT7988 - same ring blocks, DIP slots, MYIP_VLD, SDL, score delta, refresh timer, CRSN and VLAN bits (the sole delta, interrupt group assignment, is downstream of steering). So this is the behaviour of every vendor generation on MT7988, not a porting issue. As far as I can tell, no netsys v2/v3 part has ever had engagement validated; the 2016 MT7623 numbers are the only generation where a throughput test could have exposed non-engagement.
A note on method, since rate numbers are quoted above: identical configurations differ by up to 14x between adjacent windows on this hardware, so every comparison I report is against pristine controls interleaved in the same run, and the two firm results in (2) are structural (an exact zero, and a reversal on replication) rather than rate comparisons.
Since GRO on this hardware already reaches ~23 segments/skb while the SDL cap limits HW LRO to 8, the feature as it stands is strictly worse than not using it on MT7988: a third of the aggregation plus a large spurious-retransmit cost, and I would suggest not enabling it by default until the steering behaviour is understood.
Maybe someone knows from MediaTek: on netsys v3, what per-packet condition must hold for a promoted, ring-bound flowās segments to be steered to its LRO ring, given that the flow holds a ring binding, its learned table entry matches, and the VLAN comparator is demonstrably evaluating its packets? Is there per-entry or steering-stage state on MT7988 readable by any interface? The v1-era FE 0x300 window returns nothing on v3, the v3 ALT entry format carries no state, and the RX descriptor has no comparator-result flag on the miss path, so I have exhausted what is observable from the driver side.