Flowtable corrupts large TCP packets on BananaPi BPI-R4-PRO-8X

Dears,

Hardware: BananaPi BPI-R4-PRO-8X SoC: MediaTek MT7988A (Filogic 880), 4x ARM Cortex-A73 OpenWrt: 24.10-SNAPSHOT, kernel 6.6.93, target mediatek/filogic Symptoms

  • SSH connections to remote hosts fail consistently via router, work fine with direct ISP connection:
> ssh [email protected]
Connection reset by 1.1.1.1 port 22
  • Websites load slowly, stall, or partially load
  • Short HTTP requests work fine; large/long-lived TCP connections break

Root cause

The board ships with /etc/flowtable.conf, a board-specific nftables config that enables hardware-accelerated packet forwarding via the MT7988A flowtable engine:

table inet filter {
flowtable f {
hook ingress priority filter + 1;
devices = { eth0,eth1,eth2,lan0,lan3,mxl_lan0,mxl_lan1,mxl_lan2,mxl_lan3,mxl_lan5 };
flags offload;
counter;
}
chain forward {
type filter hook forward priority filter; policy accept;
meta l4proto { tcp, udp } flow add @f;
}
}

The flowtable silently truncates large TCP segments in transit. Verified via tcpdump:

# Client sent 1672-byte SSH key exchange packet:
client → serverseq 23:1695 length 1672[P.]

# Server acknowledged only 1471 bytes — 224 bytes missing:
server → clientack 1471length 1120[P.]

# Client sent next chunk (unaware of gap):
client → serverseq 1695:2903 length 1208[P.]

# Server received out-of-sequence data → RST:
server → clientRST

224 bytes (bytes 1471–1694) never reached the server despite MTU being 1500 on all hops (tracepath confirmed pmtu 
1500 end-to-end).

Both hardware and software offload modes are affected. Removing only flags offload (software flowtable) does not fix the truncation — ruling out the hardware DMA engine specifically and pointing to a bug in the flowtable fastpath itself.

Fix

Remove /etc/flowtable.conf and optionally flush the runtime table without rebooting:

# Permanent fix (survives reboot):
mv /etc/flowtable.conf /etc/flowtable.conf.bak

# Apply immediately without reboot:
nft delete table inet filter 2>/dev/null

Impact of the fix

Slightly reduced forwarding throughput — every packet goes through the full netfilter stack instead of the fastpath. On MT7988A (4x Cortex-A73) this is negligible for 1Gbps routing. Only noticeable if simultaneously saturating multiple 2.5G/10G ports.

2 Likes

This is because perhaps your “fix” has disabled HW offload.

I haven’t looked closely at nor used 24.x. I didn’t recall seeing “/etc/flowtable.conf” in master-snapshot before 25.x fork and as well as in 25.x. Perhaps a 24.x thing? I would expect firewall/offload is configured in “/etc/config/firewall”

For R4 (non Pro), HW offload works well in 25.x release.

1672 bytes look too much for mtu,wonder why there is an ack.

24.x from bpi may use the older tagging protocol and different vlan stack. So can you test with lower mtu (e.g. 1490) on the dsa user port?

Exclude port 22 from this rule?

What is your MTU size? I do not think ppe will support much more then 1500.

HIya, I was facing the same issue and I was wondering why I couldn’t establish a connection from behind the router on my laptop/desktop/mobile using SSH:

$ ssh -T [email protected]
Connection reset by 20.26.156.215 port 22

Your fix seems to have fixed it for me:

# Permanent fix (survives reboot):
mv /etc/flowtable.conf /etc/flowtable.conf.bak

# Apply immediately without reboot:
nft delete table inet filter 2>/dev/null

I wonder how this was missed during the development stage or even testing?

I am using the official Banana Pi provided image as listed on https://docs.banana-pi.org/en/BPI-R4_Pro/BananaPi_BPI-R4_Pro#_r4pro_4e_openwrt.

How would you do that? Do you mind sharing the commands to run? Thanks a lot.

You can check MTU first with “ip a”. It should be not more than 1500 byte.

Setting can be done when interface is down with sonething like this

ip link set dev interface mtu 1500 up

Hiya, Frank! I am on the Banana Pi BPI-R4 Pro (4E) if that makes a difference.

So nothing is above what you recommended except the loopback interface:

$ ip -c a s | rg -i mtu               
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-wan state UP group default qlen 1000
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
5: lan0@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN group default qlen 1000
6: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN group default qlen 1000
7: lan3@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN group default qlen 1000
8: mxl_lan0@eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN group default qlen 1000
9: mxl_lan1@eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN group default qlen 1000
10: mxl_lan2@eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN group default qlen 1000
11: mxl_lan3@eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN group default qlen 1000
12: mxl_lan5@eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP group default qlen 1000
13: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1000
14: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
15: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
16: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
17: br-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
34: phy0.1-ap0: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
35: phy0.2-ap0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

The seems to work if and only if, HW offloading is not set to HW in the Firewall section of the UI. If it is even with the below file:

table inet filter {
	flowtable f {
		hook ingress priority filter + 1;
		devices = { eth0,eth1,eth2,lan0,lan2,lan3,mxl_lan0,mxl_lan1,mxl_lan2,mxl_lan3,mxl_lan5 };
		flags offload;
		counter;
	}

	chain forward {
		type filter hook forward priority filter; policy accept;
		tcp dport 22 return;
		tcp sport 22 return;
		meta l4proto { tcp, udp } flow add @f;
	}
}

It will fail.

I just wanted to make sure it is no mtu related issue like the trace above looks like.

Should not make a difference (8x vs.4e) except it is r4pro where lan-ports are from mxl switch and i guess a bug here with ppe and like described above just delete the mapping to the flowtable (not complete inet chain).

Just this line has to be dropped:

meta l4proto { tcp, udp } flow add @f;

Then it should work (of course for complete cleanup the flowtable should be removed too to be not an unsed chain).

I tried (suggestion of Claude) the below as well:

for i in lan0 lan2 lan3 mxl_lan0 mxl_lan1 mxl_lan2 mxl_lan3 mxl_lan5; do
  ip link set dev $i down
  ip link set dev $i mtu 1490
  ip link set dev $i up
done

It made no difference.

Anyway: The biggest problem I am having is compiling OpenWrt. I simply do not know which commands to run to produce an image that can be flashed to a SD card. I would also like to enable kernel-level debugging so if anything turns up I can step through the code but all the available versions I can find do not compile with debugging information available. I really wish someone could help me.

For basic steps:

https://wiki.fw-web.de/doku.php?id=en:bpi-r4:start#openwrt

Just use the right board profile

For kernel options use “make kernel_menuconfig”

You could try removing the mxl_lanX ports from this line, if the mxl switch is the problem.

Also

And

So eth0 and eth2 should not be in the devices line I believe.

Newer kernel versions and also openwrt (r4pro PR) do not use the mxl_prefix anymore. And using flow offload only on the internal port imho does not make much sense…but this can be used to track issue down with ssh…the lan5 port should not break.

What I meant was that this line:

devices = { eth0,eth1,eth2,lan0,lan3,mxl_lan0,mxl_lan1,mxl_lan2,mxl_lan3,mxl_lan5 };

Should be:

devices = { eth1,lan0,lan3,mxl_lan0,mxl_lan1,mxl_lan2,mxl_lan3,mxl_lan5 };

And when skipping the mxl ports, it should be:

devices = { eth1,lan0,lan3 };

The fact that eth0 and eth2 are somewhere within this path, it is all handled in underlying switchdev (dsa) drivers. They should not be placed in this devices line.

I created an issue on issue on their official repository: https://github.com/BPI-SINOVOIP/BPI-R4PRO-4E-OPENWRT-V24.10.0-Master-Devel/issues/1. Hopefully (fingers crossed) it will get looked at and fixed.

CCing @sinovoip.