Bananapi r4 10G sfp+ not reaching full speed

rajasjoshi · January 6, 2025, 3:13pm

Hi All,

I have 2 banana pi r4s both running latest openwrt mainline build. Both are connected to each other using the 10G SFP+ module from the official store (link). One board acts as a client and other as server.

The sfp+ module is getting detected as well. But when I run iperf3 tests, it can only reach 5Gbps speed. Even in the ethtool, it shows max 5000baseT/Full. Also, hw forward offloading option doesnt work with the firewall.

ethtool output:

root@OpenWrt:~# ethtool  sfp-lan
Settings for sfp-lan:
        Supported ports: [  ]
        Supported link modes:   100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                2500baseT/Full
                                5000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                2500baseT/Full
                                5000baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  100baseT/Full
                                             1000baseT/Full
                                             10000baseT/Full
                                             2500baseT/Full
                                             5000baseT/Full
        Link partner advertised pause frame use: Symmetric Receive-only
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 11
        Transceiver: external
        MDI-X: Unknown
        Current message level: 0x000000ff (255)
                               drv probe link timer ifdown ifup rx_err tx_err
        Link detected: yes

dmesg logs:

root@OpenWrt:~# dmesg | grep sfp
[    3.947083] mtk_soc_eth 15100000.ethernet sfp-lan: renamed from eth1
[    3.962654] mtk_soc_eth 15100000.ethernet sfp-wan: renamed from eth2
[   11.499684] sfp sfp1: Host maximum power 2.0W
[   11.504580] sfp sfp2: Host maximum power 2.0W
[   11.826513] sfp sfp2: module OEM              SFP-10G-T        rev 1    sn 2409090010       dc 240909
[   16.521303] mtk_soc_eth 15100000.ethernet sfp-lan: configuring for inband/10gbase-r link mode
[   16.553200] br-lan: port 4(sfp-lan) entered blocking state
[   16.558719] br-lan: port 4(sfp-lan) entered disabled state
[   16.564263] mtk_soc_eth 15100000.ethernet sfp-lan: entered allmulticast mode
[   16.571523] mtk_soc_eth 15100000.ethernet sfp-lan: entered promiscuous mode
[   16.638991] mtk_soc_eth 15100000.ethernet sfp-wan: configuring for inband/10gbase-r link mode
[   16.671495] br-wan: port 2(sfp-wan) entered blocking state
[   16.677052] br-wan: port 2(sfp-wan) entered disabled state
[   16.682601] mtk_soc_eth 15100000.ethernet sfp-wan: entered allmulticast mode
[   16.690254] mtk_soc_eth 15100000.ethernet sfp-wan: entered promiscuous mode
[   43.351331] mtk_soc_eth 15100000.ethernet sfp-lan: PHY [i2c:sfp2:11] driver [Aquantia AQR113C] (irq=POLL)
[   80.280325] mtk_soc_eth 15100000.ethernet sfp-lan: left allmulticast mode
[   80.287126] mtk_soc_eth 15100000.ethernet sfp-lan: left promiscuous mode
[   80.293859] br-lan: port 4(sfp-lan) entered disabled state
[   80.681761] mtk_soc_eth 15100000.ethernet sfp-wan: left allmulticast mode
[   80.688569] mtk_soc_eth 15100000.ethernet sfp-wan: left promiscuous mode
[   80.695299] br-wan: port 2(sfp-wan) entered disabled state
[   82.493217] mtk_soc_eth 15100000.ethernet sfp-lan: configuring for inband/10gbase-r link mode
[   82.524478] br-lan: port 4(sfp-lan) entered blocking state
[   82.529962] br-lan: port 4(sfp-lan) entered disabled state
[   82.535455] mtk_soc_eth 15100000.ethernet sfp-lan: entered allmulticast mode
[   82.542586] mtk_soc_eth 15100000.ethernet sfp-lan: entered promiscuous mode
[   82.617943] mtk_soc_eth 15100000.ethernet sfp-wan: configuring for inband/10gbase-r link mode
[   82.649209] br-wan: port 2(sfp-wan) entered blocking state
[   82.654692] br-wan: port 2(sfp-wan) entered disabled state
[   82.660191] mtk_soc_eth 15100000.ethernet sfp-wan: entered allmulticast mode
[   82.667353] mtk_soc_eth 15100000.ethernet sfp-wan: entered promiscuous mode
[  109.388312] mtk_soc_eth 15100000.ethernet sfp-lan: PHY [i2c:sfp2:11] driver [Aquantia AQR113C] (irq=POLL)
[  115.693211] mtk_soc_eth 15100000.ethernet sfp-lan: Link is Up - 10Gbps/Full - flow control rx/tx
[  115.693239] br-lan: port 4(sfp-lan) entered blocking state
[  115.707473] br-lan: port 4(sfp-lan) entered forwarding state

And iperf3 output:

root@OpenWrt:~# iperf3 -c 192.168.1.1 -b10G
Connecting to host 192.168.1.1, port 5201
[  5] local 192.168.1.104 port 39032 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   576 MBytes  4.83 Gbits/sec    0    209 KBytes
[  5]   1.00-2.00   sec   569 MBytes  4.77 Gbits/sec    0    216 KBytes
[  5]   2.00-3.00   sec   564 MBytes  4.73 Gbits/sec    0    331 KBytes
[  5]   3.00-4.00   sec   562 MBytes  4.71 Gbits/sec    0    320 KBytes
[  5]   4.00-5.00   sec   561 MBytes  4.70 Gbits/sec    0    314 KBytes
[  5]   5.00-6.00   sec   483 MBytes  4.05 Gbits/sec    0    211 KBytes
[  5]   6.00-7.00   sec   564 MBytes  4.73 Gbits/sec    0    195 KBytes
[  5]   7.00-8.00   sec   564 MBytes  4.73 Gbits/sec    0    233 KBytes
[  5]   8.00-9.00   sec   563 MBytes  4.72 Gbits/sec    0    209 KBytes
[  5]   9.00-10.00  sec   563 MBytes  4.72 Gbits/sec    0    198 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.44 GBytes  4.67 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  5.44 GBytes  4.67 Gbits/sec                  receiver

iperf Done.

Could you all please let me know what could be the issue? Also, let me know if additional information is required.

Thanks, Rajas

frank-w · January 6, 2025, 3:56pm

more than 4.7GB needs RSS and LRO where we try the last year to get it working. Currently i write with mtk, some of the openwrt devs and some others about RSS as pre-requisite to lro. but this is still not working. The maximum was 5.2Gbit/s with rps disabled, the 4 pdma irqs to different cpus and tx irq to cpu2.

rajasjoshi · January 6, 2025, 4:09pm

Hi @frank-w,

Thanks for the quick reply. Is there any image or build available where I could get it working? I just need the 10Gbps sfp connection to work at full speeds with iperf3. I dont need the wifi7 changes.

Thanks, Rajas

frank-w · January 6, 2025, 4:12pm

no, as we do not have it yet working…only the old SDK (kernel 5.4) and bpi image based on it may work here. No upstream kernel supports it yet (also not with dev patches).

i have a branch 6.13-rsslro in my linux-repo where i discuss with these people, but it does not work yet.

but in case you are a dev you can compare code with downstream openwrt to find the missing part

rajasjoshi · January 6, 2025, 4:15pm

@frank-w,

Thanks for the information. I am fine with a older kernel, so I will try with the SDK. Meanwhile I will check the repo as well. Thanks for the support!!

Regards, Rajas

kusmi · January 6, 2025, 4:51pm

I also think, if you run iperf3 on the BananaPi itself, all traffic goes over CPU anyways? As the CPU “generates” that traffic and the other side consumes the traffic, so I doubt you might ever reach 10G with iperf3 running locally - or am I wrong?

frank-w · January 6, 2025, 6:16pm

basicly yes, but with current patches we can spread the calculation over all 4 cpu-cores and i generate traffic on PC (minisforum MS-01) so R4 only needs to receive…currently with 5.55 GBit/s (~50% idle on all cpu cores)

M10 · January 6, 2025, 11:33pm

Any chance to run dpdk and vpp? Debian or openwrt?

Average_Elm_Enjoyer · January 14, 2025, 2:52pm

There are no DPDK drivers available for any of the Mediatek’s SoCs: DPDK | Supported Hardware