[BPI-R4] any plans for jumbo frame support?

[jpsollie (Janpieter Sollie)]

As a high-performance (10g) routing device, it seems normal the bpi-r4 would support at least 4k ethernet frames. Unfortunately, current settings seem disappointing:

root@OpenWrt:~# ip -d link list | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 allmulti 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 524280 tso_max_segs 65535 gro_max_size 65536
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1504 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 76:da:d6:19:d3:bf brd ff:ff:ff:ff:ff:ff promiscuity 5 allmulti 4 minmtu 68 maxmtu 2026 numtxqueues 16 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 parentbus platform parentdev 15100000.ethernet
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-lan state UP mode DEFAULT group default qlen 1000
    link/ether 9e:8a:e4:c4:60:0b brd ff:ff:ff:ff:ff:ff promiscuity 1 allmulti 1 minmtu 68 maxmtu 2026
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br-lan state DOWN mode DEFAULT group default qlen 1000                                link/ether c2:20:34:c4:c2:0b brd ff:ff:ff:ff:ff:ff promiscuity 1 allmulti 1 minmtu 68 maxmtu 2026                                                   
5: wan@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN mode DEFAULT group default qlen 1000             link/ether 76:da:d6:19:d3:bf brd ff:ff:ff:ff:ff:ff promiscuity 1 allmulti 1 minmtu 68 maxmtu 15338
6: lan1@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 76:da:d6:19:d3:bf brd ff:ff:ff:ff:ff:ff promiscuity 1 allmulti 1 minmtu 68 maxmtu 15338
7: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN mode DEFAULT group default qlen 1000            link/ether 76:da:d6:19:d3:bf brd ff:ff:ff:ff:ff:ff promiscuity 1 allmulti 1 minmtu 68 maxmtu 15338
8: lan3@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 76:da:d6:19:d3:bf brd ff:ff:ff:ff:ff:ff promiscuity 1 allmulti 1 minmtu 68 maxmtu 15338                                                  
9: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000                                                          link/ether 88:c9:b3:b0:b9:8d brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 68 maxmtu 9194 numtxqueues 1 numrxqueues 1 gso_max_size 64000 gso_max_segs 64 tso_max_size 64000 tso_max_segs 64 gro_max_size 65536 parentbus pci parentdev 0001:03:00.0                                                 
10: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 88:c9:b3:b0:b9:8e brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 68 maxmtu 9194 numtxqueues 1 numrxqueues 1 gso_max_size 64000 gso_max_segs 64 tso_max_size 64000 tso_max_segs 64 gro_max_size 65536 parentbus pci parentdev 0001:04:00.0
11: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 76:da:d6:19:d3:bf brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 68 maxmtu 65535

Eth3 + 4 are pci rtl8125 devices, so not a part of the default config, but the maxmtu of the normal devices seems disappointing … Is this a feature that can be added to the driver later on? Or is it final?

[lowjoel]

@frank-w do you think that Diff - 5f4624aa155af460eaaee5f3b740fb879b033a07^! - openwrt/feeds/mtk-openwrt-feeds - Gitiles would be able to resolve this and if we can get this upstreamed?

An earlier attempt in https://github.com/openwrt/openwrt/pull/17121 seems to have stalled.

[frank-w]

I can add it to my repo,but problem is mostly testing.

[Rup]

Hello, I am not able to test openwrt build, but as I am using bpi-r4 as a sfp+ nas with bookworm, I can compile a new kernel and test it, whenever it will be available.

[kvic]

fyi.

The last time I ported the 9K patch to OpenWrt master branch (kernel 6.12). I got these runtime errors:

mt7530-mmio 15020000.switch: nonfatal error -34 setting MTU to 1500 on port 0
mt7530-mmio 15020000.switch: nonfatal error -34 setting MTU to 1500 on port 1
mt7530-mmio 15020000.switch: nonfatal error -34 setting MTU to 1500 on port 2
mt7530-mmio 15020000.switch: nonfatal error -34 setting MTU to 1500 on port 3

That was a few weeks ago. I skipped the patch for now.

[frank-w]

Thanks,i see there is a 9k patch for 6.12,but still only for eth not the switch

https://git01.mediatek.com/plugins/gitiles/openwrt/feeds/mtk-openwrt-feeds/+/HEAD/master/files/target/linux/mediatek/patches-6.12/999-eth-31-mtk_eth_soc-add-9k-jumbo-frame-support.patch

edit: seems like the patch needs a previous patch adding MTK_XMAC_RX_CFG2 and MTK_XMAC_MAX_RX_MASK, not yet found this patch

Found it

https://git01.mediatek.com/plugins/gitiles/openwrt/feeds/mtk-openwrt-feeds/+/HEAD/master/files/target/linux/mediatek/patches-6.12/999-eth-30-mtk_eth_soc-change-default-rx-buffer-length.patch

[kvic]

The 9k patch for 6.12 was what I ported.

Just checked again in my build env, the “rx-buffer-length” patch was included in the build that spit the runtime errors in my previous post.

I didn’t look into the detail of the errors. I skipped the 9K patch because jumbo frame is not an important feature for my SOHO environment.

But would love to figure out (if I can be of any help) on the runtime errors.

[lowjoel]

I would be glad to test – I have the board and I can figure my way around the C code but I don’t have the board spec sheets and such. Might be a bit slow because I can only do this on weekends though.

[frank-w]

uploaded to 6.18-jumbo branch, but got same error

[    2.312211] mt7530-mmio 15020000.switch: nonfatal error -34 setting MTU to 1500 on port 0                                          
[    2.313805] mt7530-mmio 15020000.switch: nonfatal error -34 setting MTU to 1500 on port 1                                          
[    2.315270] mt7530-mmio 15020000.switch: nonfatal error -34 setting MTU to 1500 on port 2                                          
[    2.316697] mt7530-mmio 15020000.switch: nonfatal error -34 setting MTU to 1500 on port 3

not nice, but seems not breaking (traffic over switch works on quick test)

$ LANG=C errno 34
ERANGE 34 Numerical result out of range

so it looks like there is a max mtu setting in dsa driver too, makes sense, but i have not found any patch changing this in sdk

but also noticed this for eth0 too

[    2.388794] eth0: mtu greater than device maximum
[    2.388799] mtk_soc_eth 15100000.ethernet eth0: error -22 setting MTU to 1504 to include DSA overhead

but after the mt7530-mmio mesages, maybe because the port6 cannot use the 9k frames

[frank-w]

got information from mtk that mt7988 switch supports at least 9K

From a hardware perspective, for the MT7988 internal switch to support jumbo frames, it only requires the global MAC control register to be configured correctly.

GMACCR 0x30e0 (switch reg r 0x30e0) [1:0] MAX_RX_PKT_LEN: Must be set to 3, which means enabling MAX_RX_JUMBO. [5:2] MAX_RX_JUMBO: 0 and 1 are reserved; 2 means 2K bytes, 3 means 3K bytes, … up to 15 for 15K bytes.

The MT7988 internal switch definitely supports jumbo frames. This is likely a limitation related to the framework (netdev max mtu), and we will check further.

but had no time yet to look deeper into the code, maybe i find some time on wednesday

#define MT7530_GMACCR			0x30e0

is defined in drivers/net/dsa/mt7530.h and mt7530.c

looks like there is already

#define  MAX_RX_JUMBO(x)		((x) << 2)
#define  MAX_RX_JUMBO_MASK		GENMASK(5, 2)
#define  MAX_RX_PKT_LEN_MASK		GENMASK(1, 0)
#define  MAX_RX_PKT_LEN_1522		0x0
#define  MAX_RX_PKT_LEN_1536		0x1
#define  MAX_RX_PKT_LEN_1552		0x2
#define  MAX_RX_PKT_LEN_JUMBO		0x3

static int
mt7530_port_change_mtu(struct dsa_switch *ds, int port, int new_mtu)
{
	struct mt7530_priv *priv = ds->priv;
	int length;
	u32 val;

	/* When a new MTU is set, DSA always set the CPU port's MTU to the
	 * largest MTU of the user ports. Because the switch only has a global
	 * RX length register, only allowing CPU port here is enough.
	 */
	if (!dsa_is_cpu_port(ds, port))
		return 0;

	mt7530_mutex_lock(priv);

	val = mt7530_mii_read(priv, MT7530_GMACCR);
	val &= ~MAX_RX_PKT_LEN_MASK;

	/* RX length also includes Ethernet header, MTK tag, and FCS length */
	length = new_mtu + ETH_HLEN + MTK_HDR_LEN + ETH_FCS_LEN;
	if (length <= 1522) {
		val |= MAX_RX_PKT_LEN_1522;
	} else if (length <= 1536) {
		val |= MAX_RX_PKT_LEN_1536;
	} else if (length <= 1552) {
		val |= MAX_RX_PKT_LEN_1552;
	} else {
		val &= ~MAX_RX_JUMBO_MASK;
		val |= MAX_RX_JUMBO(DIV_ROUND_UP(length, 1024));
		val |= MAX_RX_PKT_LEN_JUMBO;
	}

	mt7530_mii_write(priv, MT7530_GMACCR, val);

	mt7530_mutex_unlock(priv);

	return 0;
}

Looks correct so far and i guess we need some debug here for checking the framework limitation.

as far as i see the error message comes from dsa framework

net/dsa/user.c:2835:		dev_warn(ds->dev, "nonfatal error %d setting MTU to %d on port %d\n",

so as mt7530_port_change_mtu (ops->port_change_mtu) looks correct we should look if there is a limitation in dsa framework somehow.

looks like the MTU is fixed there

	ret = dsa_user_change_mtu(user_dev, ETH_DATA_LEN);
	if (ret && ret != -EOPNOTSUPP)
		dev_warn(ds->dev, "nonfatal error %d setting MTU to %d on port %d\n",
			 ret, ETH_DATA_LEN, port->index);

include/uapi/linux/if_ether.h:36:#define ETH_DATA_LEN	1500		/* Max. octets in payload	 */

or this is only the base setup (initialize as default in dsa_user_create), as the callback from mt7530 driver is not yet called here. but then maybe 9K frames work later…but wonder why 1500 shows this errors here as this should be still supported. later this function (dsa_user_change_mtu) is caleld again in dsa_user_change_conduit (“Update the MTU of the new CPU port through cross-chip notifiers”).

maybe we can just ignore these non-fatal messages?? can you please test if jumbo-frames working?

[kvic]

@frank-w

maybe we can just ignore these non-fatal messages?? can you please test if jumbo-frames working?

I believe it won’t work as 9K jumbo frame. The nonfatal errors basically tell users that the ports on the switch is set to 1500.

The error seems to be caused by this arithmetic’s inside dsa_user_change_mtu():

  overhead = dsa_tag_protocol_overhead(cpu_dp->tag_ops);
  mtu_limit = min_t(int, conduit->max_mtu, dev->max_mtu + overhead);
  old_conduit_mtu = conduit->mtu;
  new_conduit_mtu = largest_mtu + overhead;
  if (new_conduit_mtu > mtu_limit)
    return -ERANGE;

Worth checking if conduit->max_mtu is set to 9K. Seems there is missing link between DSA and mtk_eth_soc driver.

I tried forcing the following inside mtk_eth_soc.c:

eth->netdev[mac->id]->max_mtu = MTK_MAX_RX_LENGTH_9K - MTK_RX_ETH_HLEN;

but doesn’t help.

And I can’t afford more time to dig further. HTH

[kvic]

The other good news is that MTK released on Monday an additional patch which enables 9K adjustable at runtime (instead of compile time as discussed above).

The patch is here: https://git01.mediatek.com/plugins/gitiles/openwrt/feeds/mtk-openwrt-feeds/+/3ca030585a302745d1551c44b7993d8abc5f9d2b

Even though the previous “nonfatal errors” still exist on startup. Now users can adjust it. from command line, and apparently it works for me:

root@BPI-R4:~# ifconfig eth0 mtu 9190 up
root@BPI-R4:~# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr <redacted>
          inet6 addr: fe80::aae3:eeff:fec6:ca20/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9190  Metric:1
          RX packets:755 errors:0 dropped:0 overruns:0 frame:0
          TX packets:567 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:284879 (278.2 KiB)  TX bytes:215650 (210.5 KiB)
          Interrupt:109

9190 is maximum I can go without any further modification to the existing code.

I don’t have actual devices to test but there is no reason now that it doesn’t work as 9k jumbo frame.

[frank-w]

I do not expect the non-fatal errors are not gone because the dsa core initializes the port (imho also user-port) with fixed value of 1500 and switch driver is not touched by this patch (but anyway thanks for it!). But even if the mac is configured with higher mtu,lower mtu should be allowed on switch side…wonder why this fails.

@kvic

I don’t have actual devices to test but there is no reason now that it doesn’t work as 9k jumbo frame.

Thats also my problem i’m not sure if one or another device i have is capable to 9K frames

[ericwoud (Eric W.)]

@frank-w

Thats also my problem i’m not sure if one or another device i have is capable to 9K frames

Is 9k frames on eth1/2 possible? Put eth1 in a separate net-namespace and connect it to lan0 (in default netns) on the same device with a short patch cable.

[frank-w]

Only if patch works, i need some external device which supports 9k frames.

[ericwoud (Eric W.)]

You could check mtk_eth_soc with a connection between eth1 and eth2? Then dsa between eth1 and lan0.

[ericwoud (Eric W.)]

Do you have a 1000M or 2500M usb rtl adapter? Rtl8153 and rtl8156 both should support mtu 9000.

Bus 002 Device 023: ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
Bus 002 Device 024: ID 0bda:8156 Realtek Semiconductor Corp. USB 10/100/1G/2.5G LAN

21: enu1u1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:e0:4c:68:08:09 brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 68 maxmtu 9194 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 16354 gso_max_segs 65535 tso_max_size 16354 tso_max_segs 65535 gro_max_size 65536 gso_ipv4_max_size 16354 gro_ipv4_max_size 65536 parentbus usb parentdev 2-1.1:1.0 
    altname enx00e04c680809
22: enu1u2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 1c:bf:ce:4a:80:6c brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 68 maxmtu 16362 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 16354 gso_max_segs 65535 tso_max_size 16354 tso_max_segs 65535 gro_max_size 65536 gso_ipv4_max_size 16354 gro_ipv4_max_size 65536 parentbus usb parentdev 2-1.2:1.0 

So setup the usb adapter in a separate namespace in a script. You can test the entire setup in 1 script.

I do the same for testing hardware flow offloading nowadays, just using 1 bananapi board, instead of 3, only running 1 test-script.

Here is an example, I was playing around with mtu’s (not offloading here). It is a good start with setting up a test-connection inside namespaces between 2 local interfaces. It is just a rough test-script for personal use, not really tidied up.

#!/bin/bash

o="$1"

[[ "$o" == "qq" || "$o" == "ad" ]] && mtu="mtu 1504" || mtu="mtu 1500"

intf1="lan1"
intf2="enu1u2c2"

cleanup() {
  kill -9 $(pidof iperf3)
  ip netns delete ns1
  ip netns delete ns2
  echo
}

trap cleanup EXIT

ip netns add ns1
ip link set $intf1 netns ns1

ip -net ns1 link set $intf1 up

ip -net ns1 link set $intf1 down
ip -net ns1 link set dev $intf1 mtu 1500
ip -net ns1 link set dev $intf1 nomaster         2> /dev/null
ip -net ns1 route del 192.168.22.0/24 dev $intf1 2> /dev/null
ip -net ns1 route del 192.168.22.0/24 dev $intf1.10  2> /dev/null
ip -net ns1 route del 192.168.22.0/24 dev $intf1.10.20  2> /dev/null
ip -net ns1 addr del 192.168.22.1/24 dev $intf1  2> /dev/null
ip -net ns1 link del ${intf1}.10                 2> /dev/null
ip -net ns1 link del ${intf1}.20                 2> /dev/null
ip -net ns1 link del ${intf1}.10.20              2> /dev/null
ip -net ns1 link set $intf1 up

orig1=$intf1
proto=""

if [[ "$o" == "ad" ]]; then
  proto="protocol 802.1ad"
  o="qq"
fi
if [[ "$o" == "qq" ]]; then
  ip -net ns1 link add link $intf1 name ${intf1}.10 type vlan $proto id 10
  intf1="${intf1}.10"
  ip -net ns1 link set $intf1 up mtu 1500
  o="q"
fi
if [[ "$o" == "q" ]]; then
  ip -net ns1 link add link $intf1 name ${intf1}.20 type vlan id 20
  intf1="${intf1}.20"
  ip -net ns1 link set $intf1 up mtu 1500
fi

  ip -net ns1 link set $intf1 up
  ip -net ns1 link set dev $orig1 $mtu
  ip -net ns1 addr add 192.168.22.1/24 broadcast 192.168.22.255 dev $intf1
  ip -net ns1 route add 192.168.22.0/24 dev $intf1


o="$1"

ip netns add ns2
ip link set $intf2 netns ns2

ip -net ns2 link set $intf2 up

ip -net ns2 link set $intf2 down
ip -net ns2 link set dev $intf2 mtu 1500
ip -net ns2 addr del 192.168.22.2/24 dev $intf2  2> /dev/null
ip -net ns2 addr del 192.168.1.8/24 dev $intf2   2> /dev/null
ip -net ns2 route del default                   2> /dev/null
ip -net ns2 link del ${intf2}.10                 2> /dev/null
ip -net ns2 link del ${intf2}.20                 2> /dev/null
ip -net ns2 link del ${intf2}.10.20              2> /dev/null
ip -net ns2 link set $intf2 up

orig2=$intf2
proto=""

if [[ "$o" == "ad" ]]; then
  proto="protocol 802.1ad"
  o="qq"
fi
if [[ "$o" == "qq" ]]; then
  ip -net ns2 link add link $intf2 name ${intf2}.10 type vlan $proto id 10
  intf2="${intf2}.10"
  ip -net ns2 link set $intf2 up mtu 1500
  o="q"
fi
if [[ "$o" == "q" ]]; then
  ip -net ns2 link add link $intf2 name ${intf2}.20 type vlan id 20
  intf2="${intf2}.20"
  ip -net ns2 link set $intf2 up mtu 1500
fi

  ip -net ns2 link set $intf2 up
  ip -net ns2 link set dev $orig2 $mtu
  ip -net ns2 addr add 192.168.22.2/24 broadcast 192.168.22.255 dev $intf2
  ip -net ns2 route add default via 192.168.22.1 dev $intf2



while ! ip -net ns1 link show dev $orig1 up 2>/dev/null | grep -q "state UP"
do sleep 0.2; done
while ! ip -net ns2 link show dev $orig2 up 2>/dev/null | grep -q "state UP"
do sleep 0.2; done

  ip -net ns1 a show dev $orig1
  ip -net ns1 a show dev $intf1
  ip -net ns1 r

  ip -net ns2 a show dev $orig2
  ip -net ns2 a show dev $intf2
  ip -net ns2 r

echo GO!

ip  netns exec ns2 ping -l 10 -c 3 192.168.22.1

ip  netns exec ns1 iperf3 -s &
ip  netns exec ns2 iperf3 -c 192.168.22.1 --bidir -t 2

When runnig this script, make sure the interfaces are not under control of systemd-networkd or NetworkManager or alike.

Don’t forget to put some copper between the 2 interfaces.

[frank-w]

thanks first will try my 2p5g usb network adapter

r8152 2-2:1.0 enx00e04c68001b: renamed from eth0
Bus 002 Device 002: ID 0bda:8156 Realtek Semiconductor Corp. USB 10/100/1G/2.5G LAN

$ sudo ip link set dev enx00e04c68001b mtu 9000
[sudo] Passwort für frank: 
frank@frank-u24:/media/data_ext/git/kernel/BPI-R2-4.14 (6.18-jumbo) [1M39U]
$ sudo ip link show enx00e04c68001b
4: enx00e04c68001b: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9000 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:e0:4c:68:00:1b brd ff:ff:ff:ff:ff:ff

looks like it supports it

have applied the patches to my 6.18-jumbo branch (needed to solve 2 conflicts).

but r4 DSA-Ports seem to no support this

# ip link set dev lan3 mtu 9000                                                                                                                                           
RTNETLINK answers: Numerical result out of range

have to test with sfp…

but looks same for MAC

root@bpi-r4-v11:~
# ip link set dev eth2 mtu 9000                                                                                                                                           
Error: mtu greater than device maximum.
root@bpi-r4-v11:~
# uname -a
Linux bpi-r4-v11 6.18.0-rc1-bpi-r4-jumbo #2 SMP Sat Nov 22 15:12:16 CET 2025 aarch64 GNU/Linux

[kvic]

@frank-w

but r4 DSA-Ports seem to no support this

# ip link set dev lan3 mtu 9000                                                                                                                                           
RTNETLINK answers: Numerical result out of range

If you had included the patch (released by MTK on Monday) that I posted above, you would not have run into this issue.

root@BPI-R4:~# ifconfig lan2
lan2      Link encap:Ethernet  HWaddr <redacted>
          UP BROADCAST MULTICAST  MTU:9000  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

[frank-w]

I have included it,see my 6.18-jumbo branch

Commits · frank-w/BPI-Router-Linux

Which codebase do you use? Maybe you have additional patches?