[BPI-R4] any plans for jumbo frame support?

Banana Pi Router design BPI-R4/BPI-R4 Pro(MT7988)
63

I applied your patches from the 6.18-jumbo-refactor branch on top of Frank’s 6.18-jumbo branch (which I think is what you also used).

It seems this only happens when there is some use of the interface. In my case the interface was down, but I still had the vlans set up. When I remove all vlans, then down eth1 I changed the mtu without any issues. If I only down eth1 it still happened.

64

I never tested it with vlans before. Thanks for the feedback. I’ll check what I can do. Can you please provide the dmesg messages/ other outputs?

Thanks!

65

I double checked the mtk_eth_soc code against your branch and noticed there were a few differences. When I use exactly the code from your branch the system seems to work without issues up to now. It seems that the code differences caused the driver to release and reallocate the pools more often (i.e., on mtu change and link loss). I am not sure something else would cause a release/realloc of the pool, and if so it that would still cause issues. I’ll monitor for a while and report back if I see other issues.

Apologies for the confusion, somehow I missed the code differences the first time.

Throughput between various machines is fantastic, reaching 9-10 gbit consistently depending on the combination of machines (it seems a bit slower receiving from some Marvell based systems). With the previous 2022 bytes mtu limit it was never really exceeding 8.5, mostly around 7.8.

-Wilfried

1 Like
66

Got response from MTK

You previously mentioned the issue of being unable to automatically reallocate the RX buffer when eth->rx_buf_len changes. We have improved this through the following patches for your reference. a7ee029fdce5c712c9e48b83eb2a48c6503c975d - openwrt/feeds/mtk-openwrt-feeds - Gitiles 54f68b94df5da1d91de4f8b3fd4a24dbf75d6dfa - openwrt/feeds/mtk-openwrt-feeds - Gitiles

1 Like
67

Actually this now only happens on a reset (the Destroy/Allocated messages for debugging):

2026-06-05T06:35:39.552270-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet eth0: NETDEV WATCHDOG: CPU: 2: transmit queue 3 timed out 5010 ms
2026-06-05T06:35:39.552312-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet eth0: transmit timed out
2026-06-05T06:35:39.759780-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet eth0: Link is Down
2026-06-05T06:35:39.759819-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet en-lan0: Link is Down
2026-06-05T06:35:40.059809-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Destroy page pool 15 (0000000082aa6b33)
2026-06-05T06:35:40.059853-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Destroy page pool 14 (000000004557608c)
2026-06-05T06:35:40.379802-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Destroy page pool 16 (00000000404f2379)
2026-06-05T06:35:40.379845-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Destroy page pool 17 (000000002f1635d9)
2026-06-05T06:35:40.379851-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Destroy page pool 18 (0000000023df6579)
2026-06-05T06:35:40.379883-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Destroy page pool 19 (00000000c12eca25)
2026-06-05T06:35:40.379890-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Destroy page pool 20 (000000005d0ffbfd)
2026-06-05T06:35:40.389809-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Destroy page pool 21 (00000000a7486c15)
2026-06-05T06:35:40.389850-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Destroy page pool 22 (0000000020a7a126)
2026-06-05T06:35:40.589773-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: warm reset failed
2026-06-05T06:35:40.609791-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Allocated page pool 23 (000000007c757762)
2026-06-05T06:35:40.609832-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Allocated page pool 24 (00000000093c7742)
2026-06-05T06:35:40.609860-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Allocated page pool 25 (00000000d33ca2cd)
2026-06-05T06:35:40.727359-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Allocated page pool 26 (000000004d66a525)
2026-06-05T06:35:40.727399-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Allocated page pool 27 (0000000035b9ba1d)
2026-06-05T06:35:40.727406-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Allocated page pool 28 (0000000090f06102)
2026-06-05T06:35:40.727411-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Allocated page pool 29 (000000000230ede4)
2026-06-05T06:35:40.727417-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Allocated page pool 30 (0000000039297c2a)
2026-06-05T06:35:40.727422-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet: Allocated page pool 31 (0000000092220bd4)
2026-06-05T06:35:40.727427-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet eth0: configuring for fixed/internal link mode
2026-06-05T06:35:40.727433-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet eth0: mtk_open: set max-mtu of mac #0 to 9190 (9K+XGMII)
2026-06-05T06:35:40.727438-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet eth0: Link is Up - 10Gbps/Full - flow control rx/tx
2026-06-05T06:35:40.727453-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet en-lan0: configuring for inband/10gbase-r link mode
2026-06-05T06:35:40.859799-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet en-lan0: mtk_open: set max-mtu of mac #1 to 9190 (9K+XGMII)
2026-06-05T06:35:40.859833-04:00 nostromo kernel: mtk_soc_eth 15100000.ethernet en-lan0: Link is Up - 10Gbps/Full - flow control off
2026-06-05T06:36:40.409773-04:00 nostromo kernel: page_pool_release_retry() stalled pool shutdown: id 15, 4 inflight 60 sec
2026-06-05T06:36:40.729790-04:00 nostromo kernel: page_pool_release_retry() stalled pool shutdown: id 20, 4 inflight 60 sec
2026-06-05T06:36:40.729828-04:00 nostromo kernel: page_pool_release_retry() stalled pool shutdown: id 21, 7 inflight 60 sec
2026-06-05T06:36:40.729835-04:00 nostromo kernel: page_pool_release_retry() stalled pool shutdown: id 22, 1 inflight 60 sec
2026-06-05T06:37:40.729786-04:00 nostromo kernel: page_pool_release_retry() stalled pool shutdown: id 15, 4 inflight 120 sec
2026-06-05T06:37:41.049777-04:00 nostromo kernel: page_pool_release_retry() stalled pool shutdown: id 20, 4 inflight 120 sec
2026-06-05T06:37:41.049813-04:00 nostromo kernel: page_pool_release_retry() stalled pool shutdown: id 21, 7 inflight 120 sec
2026-06-05T06:37:41.049820-04:00 nostromo kernel: page_pool_release_retry() stalled pool shutdown: id 22, 1 inflight 120 sec

It’s going through a free/alloc cycle when the reset happens, and I assume some pending packets during that reset keep the buffers marked as “in flight”.

The reset happens without the jumbo patches, they started when I moved to 6.18 kernels, so that they are happening is not related to jumbo packets. And it’s happening on an interface that has no jumbo packets enabled (but the reset affects all interfaces). The page_pool_release issue only happens with the jumbo patches, but that seems reasonable given they did change some of the page alloc mechanisms if I read them correctly.

The reset is triggered early in a heavy lan->wan transfer, it seems the eth0 connection to the internal switch gets funky and stops all outgoing packets while still receiving. Still trying to figure that one out.

I’ll have a look at the patches mentioned, see if they fix the page_pool_release issue.

68

Have you applied the the new patches on top of my ported patches or on top of @DeadMeat’s changes? Maybe it is not compatible…

The patches should fix the problem that you cannot just set mtu at any state of the interface only bring up mac first then down,then set mtu and bring up again

69

Will try that tonight.

70

The release issues seem to be related to strongswan running on the system. If strongswan is restarted after the reset the buffers seem to be released after a few minutes. Without restarting I have seen the buffers stuck for hours, so the ipsec stack must keep some of the buffers in use a lot longer than expected.

Still trying to figure out why the TX timeout happens in the first place. Seems to happen only in very specific circumstances (e.g. not for high load in general, maybe related to high load over a single TCP connection).

71

I knew the issue seemed familiar, the branch is missing

net: ethernet: mtk_eth_soc: increase MTK_RESV_BUF

With that no more resets on the test transfers.

72

Hey @wteiken, how is your progress? Any assistance required?

73

TL;DR: With the patch below your jumbo branch is running stable for a while now on my bpi R4, with high sustained traffic over all interfaces on a regular basis. iperf between jumbo enabled machines is 9-10gbps. I am still tracking down an issue with tx checksum offloading when traffic is routed from the lan sfp to the wan on the DSA switch, but that is not specific to the jumbo branch.

As mentioned in the earlier post I can tell the pool shutdown issue is caused by a strongswan setup with ipsec inferfaces that seems to keep buffers in use. My guess is it’s keeping a reference to the buffers from incoming traffic in a queue, blocking the release. Generating some traffic or shutting down the ipsec interface seems to release them after a while. So it seems this is just a bit unexpected behavior triggered by a low-use ipsec connection, but there may not be any actual memory issue.

The second issue with the branch was that with heavy traffic there were random eth resets (which then triggered the pool release issue). That is fixed by this patch:

commit 87ee0b7ff6fbe3a3fca3b4e0b7f0685bb281c5a9
Author: Frank Wunderlich <[email protected]>
Date:   Wed Jan 8 17:46:18 2025 +0100

    net: ethernet: mtk_eth_soc: increase MTK_RESV_BUF
    
    Change CDM guard band RSV_BUF from 0x40 to 0x80, default 0x40 may
    cause xDMA hang.
    
    taken from mtk-sdk:
    https://git01.mediatek.com/plugins/gitiles/openwrt/feeds/mtk-openwrt-feeds/+/872c417f7926279cc25c44a7b2886be434e38904%5E!
    
    Fixes: 160d3a9b1929 ("net: ethernet: mtk_eth_soc: introduce MTK_NETSYS_V2 support")
    Signed-off-by: Frank Wunderlich <[email protected]>

diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.h b/drivers/net/ethernet/mediatek/mtk_eth_soc.h
index 7719ee56d135..08bedf78ad64 100644
--- a/drivers/net/ethernet/mediatek/mtk_eth_soc.h
+++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.h
@@ -314,7 +314,7 @@
 #define MTK_CHK_DDONE_EN	BIT(28)
 #define MTK_DMAD_WR_WDONE	BIT(26)
 #define MTK_WCOMP_EN		BIT(24)
-#define MTK_RESV_BUF		(0x40 << 16)
+#define MTK_RESV_BUF		(0x80 << 16)
 #define MTK_MUTLI_CNT		(0x4 << 12)
 #define MTK_LEAKY_BUCKET_EN	BIT(11)

With these patches by R4 iperf3 is reaching 9-10gbps when using multiple connections for tranasfers between jumbo enabled machines. It’s 9gbps when receiving from a marvell based machine, otherwise ~10.

I am still tracking down one issue with tx checksum offloading that seems to only occur when routing between the sfp lan interface and the WAN interface on the internal switch. But that is not specific to this jumbo branch, that’s also in Frank’s 6.18 branch. Disabling TX checksum offloading on eth0 is a workaround.

So for the moment I have no issues with the jumbo setup in the branch (modulo the buffer size patch), it’s working stable in a heavily used system.

Thanks for checking in!

-Wilfried

74

I finally got the chance to test a bit more. There seems to be some issue with the branch that causes problems on large transfers routed between lan and wan. The lan MTU is 9000, the wan MTU is 1500.

The effect: The TCP stream has a 32-64 byte segment replaced with some other data. In some of the test cases I confirmed it’s a copy of 32-64 bytes from a later position in the same stream. This was tested by comparing the tcpdump from the outgoing interface on the BPI R4 (i.e., what the system thinks it’s sending) to the tcpdump on the receiving system (i.e., what actually was sent).

This happens with both IPv4 and IPv6 traffic.

This would suggest that the outbound segmentation offload is mixing up bytes from different packets?

What makes it hard to debug: It only is really visible in a specific setup:

  • It only happens when traffic is routed. The same traffic originating on the bpi r4 has no issues (e.g., sending the same traffic through a proxy on the bpi has no issues). I assume that has something to do with the buffers being passed around in the kernel vs. a new buffer being created when going through the proxy.
  • It only happens with a Nokia i-211m-l ONT connected to the WAN eth.
    • Putting a switch in between seems to prevent the issue (or at least trigger it a lot less). Not really sure why, maybe because the switch will drop some of the modified packets because of a bad eth checksum that is not affected in the same way the TCP checksum is? The ONT is known other issues related to checksums, so it may pass on a packet onto the fiber that it should block because of a bad checksum.
    • I can trace the modification of the TCP data to the ethernet packets leaving the BPI, so the ONT is not causing the garbled the data, but it seems to be passing on garbled data that other network equipment drops.
    • A BPI R4 with the same kernel connected to a cable modem does not have the same issue, again I assume because it handles the checksums differently (i.e., drops these packets).
  • It seems to mainly affect SSL traffic. That traffic then aborts with a “remote error: tls: bad record MAC”. The error message is obvious given some random bytes were changed. But why it does for example not impact large scp transfers is not obvious to me at all.

Some other observations:

  • It does not happen if all of the MTUs are 1500. But it does happen if one of the MTUs is larger, even if not directly affecting this traffic flow (e.g., the SFP wan even if unused) the issue appears. So it must be related to the max package size set for the mtk_eth_soc driver.
  • Disabling tsa and gso is also a workaround, which would make sense if this is an issue with the segmentation offload garbling bytes.

If I’ll find some time I’ll try to see if I can narrow the issue to a specific patch in that branch.

75

Update:

  • I modified the driver to use 7936 as max RX length instead if 9K. With that value everything works, no issue with the original ONT. As soon as I increase that to 8K the issues start. So wondering if there is an issue with the larger buffer sizes (either driver bug or an actual issue with the hardware offloading when the buffer size reaches 8K).

  • There is also an issue once the driver goes through the reset to change the max buffer size, I’ll see if the patches mentioned above may help with that.

Some more data:

  • If all MTUs are 2022 or less I see no issues. Once one MTU is larger (even if on an inactive port) the issue seems to appear. But as mentioned, it only affects packets that are forwarded between interfaces. Segmentation and checksum offloading seems to work fine if the traffic originates on the BPI.
  • Disabling only segmentation offload significantly reduces the problem, but does not fully avoid it. Only when checksum offloading is disabled does the system properly work with MTU sizes above 2022 bytes.
  • The issue usually happens 40M-100M into the transfer on a single TCP connection. It’s probabilistic, depending on the exact setup I see 5%-70% success rate for transferring 500M once the issue appears.
  • Once the issue appears even reducing the MTU size again does not fully fix the issue. Not sure if that is because the buffer size is never really reduced, or because the reset is not sufficient to fix the problem. But the probability of it appearing goes down once the MTU is lowered again.
  • When I look at TCP traces I see bursts of retransmits, and assuming the connection does not fully fail the transfer is recovering (e.g., I see one or two bursts if retransmits during a 500M transfer). So I am wondering if this is triggered by something rare that trips up the offload engines for a bit and then they recover.
  • In the TCP traces I see some packets that have a bad checksum (and these may or may not be dropped depending on the hardware involved), and I see some packets that have garbled contents (comparing the tcpdump outgoing on the router with a monitor between router an ONT).
  • I am not yet sure if this only happens on the ports behind the DSA switch. I am currently having trouble getting the ONT to properly sync with a copper module in the SFP WAN port, so some more indication the ONT model is at least part of the overall issue here. But it’s not that specific ONT instance (I swapped it with another ONT of the same model and saw the same issues). But as mentioned the ONT may not drop some of the packets it should, but still the BPI is sending some garbled packets which would indicate some driver issue to me.
  • Using another SFP ONT in the SFP WAN port has not yet show the TLS connection failure. But it may have the same underlying issue and just drop some of the packets, e.g. due to the checksum issues. But I also need to transfer more data to make sure that I have just not seen a less probable event yet.
1 Like