Hi @frank-w is LRO/RSS etc upstreamed do you know ? and on which kernel version?
thank you
1 Like
Not yet,but recently i’ve send a RFC to netdev Mailinglist
https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/
But no comments yet. I have added these patches to my 6.18-rc branch so it will be included in my 6.18-main and my upcoming LTS branch.
3 Likes
Thank you for your post, raised speed from 4.5 gbit\s to 8.4 with this RSS option that you described
LRO is not started but anyway it sound much better
If someone is interested, here is a link to a Linux kernel patch set taken from a recent mtk-openwrt-feeds branch and adjusted for recent Linux branch: linux_6.19-pre-rc1_37bb2e72__mtk-openwrt-feeds_eb48282b_patches_+rss+lro+nft_flow_offload+64bit~almost_stock_mt7996.tar.xz
If I’ll have more time in the future, then maybe I’ll fork the Linux kernel git archive somewhere in public available repository with the patches applied.
- I keep the quilt format since it us used by mtk-openwrt-feeds, thus makes it easier to keep on track as well as cherry pick what is really needed.
- What is applied can be checked by opening the patches/series file which also has some comments; for now most of MediaTek provided mac80211 and mt76 patches are not applied, since even if they were modified to apply, they bring certain problems (e.g. recently since Linux kernel > 6.16 I’ve noticed that WiFi stops receiving traffic just after some time and/or amount of traffic served; I have a nearly full set of MediaTek patches applied to the Linux kernel v6.16-rc1 and working if someone is interested).
- What is tested is given after “+” signs in the archive name.
- MLO is not tested as I do not own a station type of device that is compatible with it.
- It uses @frank-w build scripts, yet I provide a trimmed kernel config file that should be just copied to the build directory and if necessary adjusted to some custom needs.
How to apply and build:
cd <path_to_linux_kernel>
git checkout <linux_commit_from_patch_archive_file>
tar -xJf <path_to_patch_archive_file>
quilt push -a
sed -ri 's/^(board=bpi-.*)/#\1/' build.conf
sed -ri 's/^#(board=bpi-r4)/\1/' build.conf
bash build.sh importconfig
cp patches/config ../build/.config
bash build.sh build
bash build.sh pack
Which kernel do you use as base? I got message about requesting for access,please check rights on your gdrive.
I try to send a v3 today. Noticed that the part i dropped in v2 is needed for mt7986 rss and i try to get lro on mt7986 working (current patch seems to break lro on mt7988)
Sorry for the permissions, should be fixed by now.
I use today’s Linux 6.19 (pre-rc1) commit 37bb2e72, I try to keep precise info in the header of patches/series file.
BTW I had to revert your patch about sram (04c7aacc) since it was committed, it was causing all the time:
mt7530-mmio 15020000.switch: Failed to register DSA switch: -517
here is the boot log for reference: debian_6.18-rc1master_1760779590.log (54.0 KB)
Using a base in tovalds master in merge window is no good idea for this,either use rc1 or the last stable.
04c7aacc is no commit in my repo (not found by github,so i’m not sure why this is caused. But when you talk about sram you have to use dts and driver code from mainline and not simply overwrite with sdk files. The sram handling is different in mainline (dedicated sram node) and sdk (uses second reg property in eth node).
Currently i have some issues with mt7622 and non rss/lro related patches. 1 is after 6.18 (mmc probe order) and one is needed for mt7988,but breaks mt7622 network.
I was referring to 04c7aacc + 65f0e397 from Linux repository (btw, that is one of the reasons I do not use your BPI-Router-Linux repo: it is not a fork of the torvalds repo, but that is your choice, same as mine is to stick to the upstream version and not to a specific release or rc tag).
Here is a patch set for linux v6.18 tag: linux_6.18_7d0a66e4__mtk-openwrt-feeds_eb48282b_patches_+wed+gfp_dma32+rss+lro+nft_flow_offload~almost_stock_mt7996.tar.xz
The set of patches is tested with BPI-R4+BPI-BE14. I can’t tell much about mt7622, I do not own that hardware.
The hardware LRO in MT7988A is more like a marketing gimmick rather than a generally useful offload feature.
The limitations of hardware LRO in MT7988A (flagship SoC from MediaTek):
- limited to 2 IP addresses which you have to manually set up by hacking into so called ‘flow-table’ (flow redirector) through ethtool
- possible to extend to support 4 IP addresses by changing the code (which is not what’s code in whatsoever patches you could find from MTK feed)
- the IP addresses have to terminate on the router; nothing useful for forwarding traffic
- one application I can think of is BPI-R4 as a NAS server which is likely will benefit from hardware LRO but nothing else
So I would recommend focus the energy on upstreaming something else than hardware LRO.
The hardware LRO in MT7988A is very restrictive and limited…
1 Like
If you have a patch for extending functionality i’m open,but i have only 1 other device sending/receiving 10G traffic so i cannot test routing (which should already be mainline via ppe). RSS and LRO were the only way to get the 10G terminating traffic working. Yes it is limited to 4 adresses (i only used one).
@matt1606 yes my repo is (no more) a fork of torvalds,but created branches are always having it as base till stable release,then stable tree. The rc branch is mostly based on rc1…also torvalds/linux. It does not need to be a fork. I do not like rolling release as it prevents testing against older versions for finding bugs
I don’t have a patch unfortunately. I spent 15 mins trying and gave up for the reasons I posted above. Apparently it’s not a simple change of redefining a MACRO from 2 to 4.
RSS is crucial. LRO is not. I believe RSS is sufficient for hitting 10G for forwarding and terminating in the router.
regarding the MTK_MAX_LRO_IP_CNT const i got an answer from mtk:
The original design was likely due to the fact that we only have four DIP registers (LRO_RINGX_DIP_DWX), and these four DIP registers are shared among eth0/1/2. That’s why the mtk_hwlro_add_ipaddr_idx function searches for an available DIP register. This design simply reserves two IP addresses for each netdev (hwlro_ip), but in reality, any IP address written to a DIP register will take effect.
and only with LRO i was able to reach the 10GBit/s on R4…
regarding the first 1-2 iperf3 rounds are not reaching the 10GBit/s…third round does
HWLRO does require some startup time. The register LRO2_CTRL_DW2 is the minimum threshold to become a candidate for LRO aggregation.
1 Like
Here is a more complete patch set with lots of more applied than previously: linux_6.19-rc5_0f61b186__mtk-openwrt-feeds_42be1aec_patches.tar.xz
- for now I can say that it booted and networking is working =)
I’m astonished it went so fast this time, only two evenings spent on upstreaming the mtk-openwrt-feeds patches to linux v6.19-rc5, it is like MediaTek picked up my earlier patches, but never mentioned me as a contributor 
Guys, what is the present state of RSS/LRO? @frank-w are your kernels enabled with this by default?
Is you kernel 7.* ok for the non PRO version?
Kernel itself is for all boards. Lro/RSS is part of latest branches,but not yet tested on 7.x
For upstreaming some AI messages have to be answered and fixed. Got some informtion from mtk, but some issues are still unresolved (e.g. some corner cases not clean in current state).
1 Like
HW LRO on MT7988A: the engine picks up only a tiny fraction of a learned flow, and every pickup causes a TCP retransmit
I have been running this series (via frank-w’s BPI-Router-Linux 7.1-main, which carries it) on a BPI-R4 Pro (MT7988A) router and measured HW LRO behaviour rather than throughput. This is not a race condition as far as I can tell. The problem is related to packet ordering, inside the hardware.
To reproduce: one TCP stream from a 10G host to the board, board receiving, about 4.5 Gbit/s and 380 000 segments per second. One LRO rule for the board’s own IP, programmed with ethtool -N.
What I see:
- The flow appears in
hw_lro_auto_tlbwithin seconds and stays there for the whole test. Total agginhw_lro_statsreaches only 0.002 – 0.05 % of the flow’s segments. The other 99.95 % arrive on the RSS rings.- With the rule programmed, the sender retransmits 13 – 218 segments per GiB. Without it, 0.4 – 1.9. The extra retransmits equal
Total agg. No error or drop counter moves.
What I believe happens: the engine holds up to 8 segments (the 12 KiB SDL) to build a superframe. While it holds them, the next segments of the same flow go straight to an RSS ring and reach TCP first. TCP sees a hole the size of one superframe and the sender retransmits it. When the superframe arrives, it is a duplicate. If the engine took every segment of the flow there would be no hole. It takes 0.02 %.
Question for MediaTek: after a flow is learned, should all of its segments go to the LRO ring? If yes, what makes the engine skip 99.95 % of them? If no, then TCP is reordered by design and HW LRO cannot be used for traffic terminating on the board.
To test it, on the board (eth0, address 192.168.1.1):
ethtool -K eth0 lro on
ethtool -N eth0 flow-type tcp4 dst-ip 192.168.1.1 loc 0
echo "5 1" > /proc/mtketh/hw_lro_auto_tlb
echo > /proc/mtketh/hw_lro_stats
On the peer: iperf3 -c 192.168.1.1 -t 60. It prints the sender’s retransmits and the bytes sent. Then on the board:
grep -A1 "^Total agg" /proc/mtketh/hw_lro_stats
Total agg is the number of segments the LRO rings took; bytes ÷ 1448 is the number the flow had. Repeat after ethtool -N eth0 delete 0: retransmits go back to almost zero.
Measured so far on a BPI-R4 Pro through the DSA conduit (tagged frames).
1 Like
You can test your hypotheses (rss breaks lro) by disabling the rss capability in code (MTK_RSS) to have LRO only.
dropping MTK_RSS here:
i believe there’s a bug with mtk_hwlro_stats_ebl - warning this is going to be a long read
mtk_hwlro_stats_ebl is declared at mtk_eth_dbg.c:42, written there, and read nowhere in the tree. And, hw_lro_stats_update() has no call site, mtk_poll_rx never invokes it. So on 7.2 (the current build i am using and checked the code - the sequence is echo “5 1” → sets a dead flag → Total agg reads 0 whatever the hardware does. Maybe it is fine on 7.1
regarding the statement related to RSS/LRO they lare separate in different registers if using Netsys V2+
here’s the patch for the counters issue
--- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
+++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
@@ -2552,6 +2552,12 @@
pktlen = RX_DMA_GET_PLEN0(trxd.rxd2);
+ if (eth->hwlro && mtk_hwlro_stats_ebl &&
+ ring->ring_no >= MTK_HW_LRO_RING(eth, 0)) {
+ hw_lro_stats_update(ring->ring_no, &trxd);
+ hw_lro_flush_stats_update(ring->ring_no, &trxd);
+ }
+
/* alloc new buffer */
if (ring->page_pool) {
struct page *page = virt_to_head_page(data);
@@ -3121,7 +3127,11 @@
if (!ring->data)
return -ENOMEM;
- if (mtk_page_pool_enabled(eth)) {
+ /* HW LRO rings advertise MTK_MAX_LRO_RX_LENGTH (~13.8K) in the
+ * descriptor's PLEN0 field, which no order-0 page pool buffer can
+ * satisfy. Keep those rings on the high-order frag allocator.
+ */
+ if (mtk_page_pool_enabled(eth) && rx_flag != MTK_RX_FLAGS_HWLRO) {
struct page_pool *pp;
pp = mtk_create_page_pool(eth, &ring->xdp_q, ring_no,
--- a/drivers/net/ethernet/mediatek/mtk_eth_dbg.c
+++ b/drivers/net/ethernet/mediatek/mtk_eth_dbg.c
@@ -1807,7 +1807,6 @@
char *p_delimiter = " \t";
long x = 0, y = 0;
u32 len = count;
- int ret;
if (len >= sizeof(buf)) {
pr_info("Input handling fail!\n");
@@ -1823,15 +1822,17 @@
p_token = strsep(&p_buf, p_delimiter);
if (!p_token)
x = 0;
- else
- ret = kstrtol(p_token, 10, &x);
+ else if (kstrtol(p_token, 10, &x))
+ return -EINVAL;
p_token = strsep(&p_buf, "\t\n ");
- if (p_token)
- ret = kstrtol(p_token, 10, &y);
+ if (p_token && kstrtol(p_token, 10, &y))
+ return -EINVAL;
- if (lro_dbg_func[x] && (ARRAY_SIZE(lro_dbg_func) > x))
- (*lro_dbg_func[x]) (y);
+ if (x < 0 || x >= (long)ARRAY_SIZE(lro_dbg_func) || !lro_dbg_func[x])
+ return -EINVAL;
+
+ (*lro_dbg_func[x])(y);
return count;
}
--- a/drivers/net/ethernet/mediatek/mtk_eth_dbg.h
+++ b/drivers/net/ethernet/mediatek/mtk_eth_dbg.h
@@ -329,6 +329,7 @@
int mtketh_debugfs_init(struct mtk_eth *eth);
void mtketh_debugfs_exit(struct mtk_eth *eth);
int mtk_do_priv_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd);
+extern u32 mtk_hwlro_stats_ebl;
void hw_lro_stats_update(u32 ring_no, struct mtk_rx_dma_v2 *rxd);
void hw_lro_flush_stats_update(u32 ring_no, struct mtk_rx_dma_v2 *rxd);
reviewing the code i found two more bugs
- HW LRO rings on MT7988 are backed by 4 KB pages but advertised as ~13.8 KB MT7988_CAPS includes MTK_HWLRO (mtk_eth_soc.h:1395), and mtk_page_pool_enabled() returns true for netsys v2+ (mtk_eth_soc.c:2178). In mtk_rx_alloc the HWLRO branch sets rx_data_len = MTK_MAX_LRO_RX_LENGTH = 12288 + 1536, giving ring->buf_size ≈ 13856 , then falls into the page-pool branch and allocates order-0 pages (mtk_eth_soc.c:3109-3132), while still programming rxd->rxd2 = RX_DMA_PREP_PLEN0(ring->buf_size) with 13856. The engine is told it has 13.8 KB of room in a 4 KB page. On the receive side, mtk_eth_soc.c:2585-2596 does build_skb(data, PAGE_SIZE) then skb_put(skb, pktlen) with pktlen from the descriptor.
Upstream never hits this because upstream only sets MTK_HWLRO on netsys v1 SoCs (MT7621/7622/7623/7629), where the page pool is off. Enabling it on MT7988 which comes from the restored downstream stack in 54df0014b9b2 makes the combination reachable for the first time. A full 12 KB superframe here is a DMA overrun past the page followed by skb_over_panic. The fact that you’re not crashing is more evidence that essentially nothing is being aggregated.
- Aggregated frames carry no gso_size. mtk_poll_rx never sets skb_shinfo(skb)->gso_size, so superframes are fine terminating locally but cannot be correctly forwarded or bridged. Not your case, i think, but it constrains where this can be enabled.
1 Like