The release issues seem to be related to strongswan running on the system. If strongswan is restarted after the reset the buffers seem to be released after a few minutes. Without restarting I have seen the buffers stuck for hours, so the ipsec stack must keep some of the buffers in use a lot longer than expected.
Still trying to figure out why the TX timeout happens in the first place. Seems to happen only in very specific circumstances (e.g. not for high load in general, maybe related to high load over a single TCP connection).
TL;DR: With the patch below your jumbo branch is running stable for a while now on my bpi R4, with high sustained traffic over all interfaces on a regular basis. iperf between jumbo enabled machines is 9-10gbps. I am still tracking down an issue with tx checksum offloading when traffic is routed from the lan sfp to the wan on the DSA switch, but that is not specific to the jumbo branch.
As mentioned in the earlier post I can tell the pool shutdown issue is caused by a strongswan setup with ipsec inferfaces that seems to keep buffers in use. My guess is it’s keeping a reference to the buffers from incoming traffic in a queue, blocking the release. Generating some traffic or shutting down the ipsec interface seems to release them after a while. So it seems this is just a bit unexpected behavior triggered by a low-use ipsec connection, but there may not be any actual memory issue.
The second issue with the branch was that with heavy traffic there were random eth resets (which then triggered the pool release issue). That is fixed by this patch:
commit 87ee0b7ff6fbe3a3fca3b4e0b7f0685bb281c5a9
Author: Frank Wunderlich <[email protected]>
Date: Wed Jan 8 17:46:18 2025 +0100
net: ethernet: mtk_eth_soc: increase MTK_RESV_BUF
Change CDM guard band RSV_BUF from 0x40 to 0x80, default 0x40 may
cause xDMA hang.
taken from mtk-sdk:
https://git01.mediatek.com/plugins/gitiles/openwrt/feeds/mtk-openwrt-feeds/+/872c417f7926279cc25c44a7b2886be434e38904%5E!
Fixes: 160d3a9b1929 ("net: ethernet: mtk_eth_soc: introduce MTK_NETSYS_V2 support")
Signed-off-by: Frank Wunderlich <[email protected]>
diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.h b/drivers/net/ethernet/mediatek/mtk_eth_soc.h
index 7719ee56d135..08bedf78ad64 100644
--- a/drivers/net/ethernet/mediatek/mtk_eth_soc.h
+++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.h
@@ -314,7 +314,7 @@
#define MTK_CHK_DDONE_EN BIT(28)
#define MTK_DMAD_WR_WDONE BIT(26)
#define MTK_WCOMP_EN BIT(24)
-#define MTK_RESV_BUF (0x40 << 16)
+#define MTK_RESV_BUF (0x80 << 16)
#define MTK_MUTLI_CNT (0x4 << 12)
#define MTK_LEAKY_BUCKET_EN BIT(11)
With these patches by R4 iperf3 is reaching 9-10gbps when using multiple connections for tranasfers between jumbo enabled machines. It’s 9gbps when receiving from a marvell based machine, otherwise ~10.
I am still tracking down one issue with tx checksum offloading that seems to only occur when routing between the sfp lan interface and the WAN interface on the internal switch. But that is not specific to this jumbo branch, that’s also in Frank’s 6.18 branch.
Disabling TX checksum offloading on eth0 is a workaround.
So for the moment I have no issues with the jumbo setup in the branch (modulo the buffer size patch), it’s working stable in a heavily used system.
I finally got the chance to test a bit more. There seems to be some issue with the branch that causes problems on large transfers routed between lan and wan. The lan MTU is 9000, the wan MTU is 1500.
The effect: The TCP stream has a 32-64 byte segment replaced with some other data. In some of the test cases I confirmed it’s a copy of 32-64 bytes from a later position in the same stream. This was tested by comparing the tcpdump from the outgoing interface on the BPI R4 (i.e., what the system thinks it’s sending) to the tcpdump on the receiving system (i.e., what actually was sent).
This happens with both IPv4 and IPv6 traffic.
This would suggest that the outbound segmentation offload is mixing up bytes from different packets?
What makes it hard to debug: It only is really visible in a specific setup:
It only happens when traffic is routed. The same traffic originating on the bpi r4 has no issues (e.g., sending the same traffic through a proxy on the bpi has no issues). I assume that has something to do with the buffers being passed around in the kernel vs. a new buffer being created when going through the proxy.
It only happens with a Nokia i-211m-l ONT connected to the WAN eth.
Putting a switch in between seems to prevent the issue (or at least trigger it a lot less). Not really sure why, maybe because the switch will drop some of the modified packets because of a bad eth checksum that is not affected in the same way the TCP checksum is? The ONT is known other issues related to checksums, so it may pass on a packet onto the fiber that it should block because of a bad checksum.
I can trace the modification of the TCP data to the ethernet packets leaving the BPI, so the ONT is not causing the garbled the data, but it seems to be passing on garbled data that other network equipment drops.
A BPI R4 with the same kernel connected to a cable modem does not have the same issue, again I assume because it handles the checksums differently (i.e., drops these packets).
It seems to mainly affect SSL traffic. That traffic then aborts with a “remote error: tls: bad record MAC”. The error message is obvious given some random bytes were changed. But why it does for example not impact large scp transfers is not obvious to me at all.
Some other observations:
It does not happen if all of the MTUs are 1500. But it does happen if one of the MTUs is larger, even if not directly affecting this traffic flow (e.g., the SFP wan even if unused) the issue appears. So it must be related to the max package size set for the mtk_eth_soc driver.
Disabling tsa and gso is also a workaround, which would make sense if this is an issue with the segmentation offload garbling bytes.
If I’ll find some time I’ll try to see if I can narrow the issue to a specific patch in that branch.
I modified the driver to use 7936 as max RX length instead if 9K. With that value everything works, no issue with the original ONT. As soon as I increase that to 8K the issues start. So wondering if there is an issue with the larger buffer sizes (either driver bug or an actual issue with the hardware offloading when the buffer size reaches 8K).
There is also an issue once the driver goes through the reset to change the max buffer size, I’ll see if the patches mentioned above may help with that.
Some more data:
If all MTUs are 2022 or less I see no issues. Once one MTU is larger (even if on an inactive port) the issue seems to appear. But as mentioned, it only affects packets that are forwarded between interfaces. Segmentation and checksum offloading seems to work fine if the traffic originates on the BPI.
Disabling only segmentation offload significantly reduces the problem, but does not fully avoid it. Only when checksum offloading is disabled does the system properly work with MTU sizes above 2022 bytes.
The issue usually happens 40M-100M into the transfer on a single TCP connection. It’s probabilistic, depending on the exact setup I see 5%-70% success rate for transferring 500M once the issue appears.
Once the issue appears even reducing the MTU size again does not fully fix the issue. Not sure if that is because the buffer size is never really reduced, or because the reset is not sufficient to fix the problem. But the probability of it appearing goes down once the MTU is lowered again.
When I look at TCP traces I see bursts of retransmits, and assuming the connection does not fully fail the transfer is recovering (e.g., I see one or two bursts if retransmits during a 500M transfer). So I am wondering if this is triggered by something rare that trips up the offload engines for a bit and then they recover.
In the TCP traces I see some packets that have a bad checksum (and these may or may not be dropped depending on the hardware involved), and I see some packets that have garbled contents (comparing the tcpdump outgoing on the router with a monitor between router an ONT).
I am not yet sure if this only happens on the ports behind the DSA switch. I am currently having trouble getting the ONT to properly sync with a copper module in the SFP WAN port, so some more indication the ONT model is at least part of the overall issue here. But it’s not that specific ONT instance (I swapped it with another ONT of the same model and saw the same issues). But as mentioned the ONT may not drop some of the packets it should, but still the BPI is sending some garbled packets which would indicate some driver issue to me.
Using another SFP ONT in the SFP WAN port has not yet show the TLS connection failure. But it may have the same underlying issue and just drop some of the packets, e.g. due to the checksum issues. But I also need to transfer more data to make sure that I have just not seen a less probable event yet.
I’m running official on the BPI-R4 Pro 8x now that it has been merged and running into this same issue with jumbo frames. Wondering if this will ever make it to main.
there are some recent patches in mtk-feed for jumbo frames, but they will not be taken into bpi-repo as this is stalled. i try to get the sdk compiled, but have not got it yet…but i have catched the patches into my kernel tree.
noticed that mtu can be set without manual down/up of the macs as this is done by kernel itself, but noticed some interface flaps which i have reported.
Nice that may be a good way to allow wider testing.
But i do not see the jumbo-frame patches (rss/lro and some smaller).thank you for porting rss/lro,i try to extract them into my repo if they work without mtk-feeds patches.
This patch makes it much more usable without putting macs down:
Btw. For rss on mt798[16] i had afair the rss dts patches in some tree…maybe accidently dropped in later rss branches…need to search for them to avoid your 760-23 patch
Mt7986:
Mt7981 was done by other user in mainline
looks like you have not added LRO yet, right? is RSS stable for you?
i wonder why you have with rss alone only 4.5 GBit/s…do you do bidir iperf-check? i had around 7.2GBit/s with RSS, but there was an iperf3 issue too where i had to use iperf2…not sure if it was RSS or LRO
btw. RSS/LRO is currently reworked by mtk due to the report from user mescheen (LRO does only catch a small amount of packets)
i’m not sure the corruption is caused by LRO or the mainline-code…but yes, there are patches for it from mescheen in my latest branches
i wonder why you have with rss alone only 4.5 GBit/s…[…] i had around 7.2GBit/s with RSS, but there was an iperf3 issue too where i had to use iperf2…not sure if it was RSS or LRO
test threaded=1 (r195) threaded=0 (r201)
MTU1500 ingress P1 5.25 G 5.30 G
MTU1500 ingress P4 6.66 G 5.27 G (2822 retr)
MTU1500 ingress P8 6.33 G 4.86 G (4098 retr)
MTU1500 egress P1 9.37 G 9.37 G
MTU1500 egress P4 9.35 G 9.40 G
MTU9000 ingress P4 9.88 G 9.16 G (350 retr)
MTU9000 egress P4 9.91 G 9.90 G
A useful way to separate the kernel limitation from the switch path is to test each interface independently: set the same MTU on both endpoints first, then on one LAN port at a time with ip link set dev <iface> mtu 9000, and verify with ip -d link plus a DF ping sized below the MTU. If the host interface accepts 9000 but a DSA LAN port returns ERANGE, that points to the switch/DSA driver path rather than cabling. It would also be worth recording whether the behavior differs between copper LAN and SFP, since that can narrow down where the MTU is being constrained.
A useful way to separate the kernel limitation from the switch path […] narrow down where the MTU is being constrained.
while 100% correct generally, in this specific case, the MTU was being constrained by missing support in the upstream mtk driver, which has been improved by porting downstream mtk & frank-ws changes
odroid is the 2.5 G cap: every odroid pair is ~2.35–2.47 G (odroid’s 2.5 G NIC on MxL lan2), and MTU 1500↔9000 moves it only ~5% — no CPU/RSS effect at 2.5 G.
banana↔changwang (10 G): ~9.7–9.8 at 9000 both ways; at 1500 ingress collapses (cw→banana 6.24 P4 / 4.75 P1) while egress stays 8.5–9.4. Same asymmetric shape as iperf3.
iperf2 ≈ iperf3 (cw→banana @9000 P4: 9.81 vs 9.88; @1500: 6.24 vs 6.66) — so neither the 1500 ingress wall nor frank’s ~7.2 is an iperf2-vs-3 artifact.
