BPI-R4: Not getting full 10GBps speed on WAN

Hello, new user here, completely new to OpenWRT and BPI. Hoping to get some help and hope you don’t mind noob questions.

I’m trying to troubleshoot a strange issue.

I have my desktop linked to the BPI-R4 set up as an OpenWRT router via a 10GBps SFP+ DAC. The R4 is connected to my ISP’s ONT via a Cat8 ethernet cable with an SFP adapter into the SFP-WAN port.

When I ssh into the R4 and do speedtest, I get around 4089.55 Mbit/s Down and 2080.85 Mbit/s Up.

When I perform the same speed test on my desktop PC, I get 2439.02 Mbit/s Down, and 2073.75 Mbit/s Up.

If I run iperf between the two machines, I get

[ ID] Interval Transfer Bandwidth [ 1] 0.0000-10.0000 sec 11.0 GBytes 9.41 Gbits/sec [ 1] 0.0000-10.0060 sec 11.0 GBytes 9.40 Gbits/sec

If I directly connect my desktop to the ONT and perform the same speed measurement, I get around the same 4 GBps speed.

What could be causing the slowdown? I know that at least from one unit to another I am at least getting pretty close to the 10GBps top speed, so it’s not an issue with my hardware. Is there anything I could change in the configurations?

Also, my ISP advertises 10 GBps speed, but the R4 and my desktop’s 10G NIC both get only up to a max of 4Mbit/s speed. Is that to be expected?

Are you sure your ISP allows you to download at 10gbe speed? Even if they sell you at that speed, a speedtest may not be able to get that speed from the speedtest servers. Aside from that, if the NIC is not accelerated, there may be a performance issue. Try to run top on the R4 while you’re executing the speedtest and check if the CPU load rises above 1 … if so, you need to enable packet steering

Current mainline (and openwrt) does not support a feature called RSS (and lro is not complete) for aggregating rx packages. Tx should be 10G,but maybe because of ack packages need to be handled. Also for traffic between both sfp you have to enable flow ofload,else all traffic goes over cpu.

Hi jpsollie - Thanks for the suggestion. I’ll check with my ISP again about it, though last time I talked to them they said that they had checked remotely and the ONT was capable of delivering 10G. How would I enable packet steering?

Frank - thanks! I have read about other people trying flow offload, so I tried this myself. When I went into the settings in LuCI and checked “Software Offload” and “Hardware Offload”, speed actually dropped to a fraction of the normal speed. I was getting around 0.22 MBps down.

Update your openwrt to the latest snapshot.

Thanks! Upgrading to the latest snapshot and then checking “Software Offload” > “Hardware Offload” in Network>Firewall> General Settings>Routing/NAT Offsetting did improve things a bit!

Speedtest from my desktop pc. Now it’s getting similar speeds as when I do speedtest from the BPI-R4!

EDIT: Iperf test

i use speedtestpp (its in the openwrt packages) and get about 4.5gbit up/down.

root@OpenWrt:~# speedtest 
SpeedTest++ version 1.20.3
Speedtest.net command line interface
Info: https://github.com/oskarirauta/speedtestcpp
Author: Francesco Laurita <[email protected]>
Co-authored-by: Oskari Rauta <[email protected]>

IP: xxx ( Init7 ) Location: [xx, xx]
Finding fastest server...
10 Servers online
Server #34312: Vaduz speed.plus.li.prod.hosts.ooklaserver.net:8080 by plus.li (21 km from you): 5 ms
Ping: 5 ms.
Jitter: 0 ms.

server selected profile detected: profile selected from server configuration

Download: 3785.11 Mbit/s                                    
Upload: 4289.39 Mbit/s

I take it this is init7?

Definitely does 10G with them (iperf3 running on the r4):

Connecting to host speedtest.init7.net, port 5201
[  5]   0.00-10.00  sec  10.9 GBytes  9.33 Gbits/sec  1213             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.33 Gbits/sec                  receiver

Your issue is the speedtest counter part. Most have only 10G to begin with so will never fill your pipe. If you really want speedtest, read https://www.init7.net/de/support/faq/speedtest/

Sorry, I guess I am too dumb to use blockquote properly (?) so let me do this https://pastebin.com/4UyJYQkR

how reliable is this command, when no upload is tested? is it also possible to use speedtestpp with that server?

for anyone else interested: iperf3 -c speedtest.init7.net

If you want simultaneously, try -bidir

thanks. the results where a bit surprising tbh… not sure how to interpret them:

root@OpenWrt:~# iperf3 -c speedtest.init7.net
Connecting to host speedtest.init7.net, port 5201
[  5] local xxx port 45314 connected to port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.09 GBytes  9.33 Gbits/sec    0   3.09 MBytes       
[  5]   1.00-2.00   sec  1.10 GBytes  9.41 Gbits/sec    0   3.09 MBytes       
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec    0   3.09 MBytes       
[  5]   3.00-4.00   sec  1.09 GBytes  9.39 Gbits/sec   59   2.24 MBytes       
[  5]   4.00-5.00   sec  1.09 GBytes  9.39 Gbits/sec    7   1.89 MBytes       
[  5]   5.00-6.00   sec  1.10 GBytes  9.41 Gbits/sec    0   2.31 MBytes       
[  5]   6.00-7.00   sec  1.09 GBytes  9.37 Gbits/sec    0   2.59 MBytes       
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.67 MBytes       
[  5]   8.00-9.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.71 MBytes       
[  5]   9.00-10.00  sec  1.09 GBytes  9.40 Gbits/sec    0   2.75 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.40 Gbits/sec   66             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.40 Gbits/sec                  receiver

iperf Done.

that looks good!

root@OpenWrt:~# iperf3 --bidir -c speedtest.init7.net 
Connecting to host speedtest.init7.net, port 5201
[  5] local xxx port 40858 connected to port 5201
[  7] local xxx port 40862 connected to port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][TX-C]   0.00-1.00   sec   512 MBytes  4.29 Gbits/sec  1333   1.69 MBytes       
[  7][RX-C]   0.00-1.00   sec   430 MBytes  3.61 Gbits/sec                  
[  5][TX-C]   1.00-2.00   sec   560 MBytes  4.70 Gbits/sec    0   1.92 MBytes       
[  7][RX-C]   1.00-2.00   sec   426 MBytes  3.57 Gbits/sec                  
[  5][TX-C]   2.00-3.00   sec   565 MBytes  4.74 Gbits/sec    0   2.12 MBytes       
[  7][RX-C]   2.00-3.00   sec   426 MBytes  3.58 Gbits/sec                  
[  5][TX-C]   3.00-4.00   sec   611 MBytes  5.13 Gbits/sec    0   2.34 MBytes       
[  7][RX-C]   3.00-4.00   sec   412 MBytes  3.46 Gbits/sec                  
[  5][TX-C]   4.00-5.00   sec   351 MBytes  2.94 Gbits/sec  586   1010 KBytes       
[  7][RX-C]   4.00-5.00   sec   461 MBytes  3.87 Gbits/sec                  
[  5][TX-C]   5.00-6.00   sec   388 MBytes  3.26 Gbits/sec    0   1.25 MBytes       
[  7][RX-C]   5.00-6.00   sec   456 MBytes  3.83 Gbits/sec                  
[  5][TX-C]   6.00-7.00   sec   473 MBytes  3.97 Gbits/sec    0   1.51 MBytes       
[  7][RX-C]   6.00-7.00   sec   445 MBytes  3.73 Gbits/sec                  
[  5][TX-C]   7.00-8.00   sec   518 MBytes  4.34 Gbits/sec    0   1.73 MBytes       
[  7][RX-C]   7.00-8.00   sec   435 MBytes  3.65 Gbits/sec                  
[  5][TX-C]   8.00-9.00   sec   684 MBytes  5.74 Gbits/sec    0   2.00 MBytes       
[  7][RX-C]   8.00-9.00   sec   340 MBytes  2.85 Gbits/sec                  
[  5][TX-C]   9.00-10.00  sec   587 MBytes  4.92 Gbits/sec    0   2.20 MBytes       
[  7][RX-C]   9.00-10.00  sec   420 MBytes  3.52 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-10.00  sec  5.13 GBytes  4.40 Gbits/sec  1919             sender
[  5][TX-C]   0.00-10.00  sec  5.13 GBytes  4.40 Gbits/sec                  receiver
[  7][RX-C]   0.00-10.00  sec  4.16 GBytes  3.57 Gbits/sec  131984             sender
[  7][RX-C]   0.00-10.00  sec  4.15 GBytes  3.57 Gbits/sec                  receiver

iperf Done.

not so good anymore?

root@OpenWrt:~# iperf3 -c --bidir speedtest.init7.net 
Segmentation fault

upsi :smiley:

i am not sure how reliable those results are with that behaviour. any other idea how to check that?

That looks weird indeed. Will dig a bit in the evening…

I think the segfault is a recent thing, never saw this in my tests two week ago.

First of all it’s important to know which version of OpenWrt you are using. I can only help you with official OpenWrt from openwrt.org, for the R4 you can download the SD card image here.

Second of all, at this point you can only obtain full 10G routing performance if you enable hardware flow offloading.

If you are using the R4 as (NAT-) router, doing this should fix the speed issues:

uci set firewall.@defaults[0].flow_offloading='1'
uci set firewall.@defaults[0].flow_offloading_hw='1'
uci commit firewall

If you are having a bridge between the ISPs ONT and the interface your client machine is connected to you will in addition to the above also need to install bridger:

opkg update
opkg install bridger

To obtain full speed in both directions simultaneously you may need to build from source and include @Dale’s patch for utilizing more than one PPE. I will integrate it with OpenWrt once it gets accepted upstream, hopefully in a couple of days.

1 Like

OT but bridger has always been a mystery to me. From a layman’s pov what does it do when is it needed? the term bridge is confusing to me at least. does it apply in a lan-bridge, vlan-bridge in network config? or when the bpi-r4 acts as a bridge(and not a NAT router) between ONT and the client machine?

Bridger is a Linux user space daemon which includes and eBPF tc classifier
program that offloads the bridge network datapath.

Bridger creates connection tracking flows for the connections which happen within a Linux bridge between different interfaces. In this way, those flows can be added to PPE and get offloaded in hardware instead of having to travel through the CPU.

From the cases you describe, the “lan bridge” and the “bpi-r4 acts as a bridge” are the exact same. LAN/WAN terminology doesn’t matter when all interfaces involved are bridged.

bridger is not needed for hardware-offloading bridged traffic flowing between different ports of a DSA switch, such as the 4x 1GE RJ-45 ports of the R4.

However, the SFP cages are not part of that DSA switch, but are considered separate network interfaces. Hence, the only way to offload traffic between those is via the PPE, and that works based on flows which need to be tracked.

In both cases, VLANs can differ and tagging/untagging would always be taken care of in hardware, as both, the DSA switch and the PPE support doing that.

1 Like

Thank you for the crystal clear explanation. bridger is not needed in my use case. as both sfp+ ports and dsa switch (eth0, eth1, eth2) are not bridged. as there is no major traffic originating or terminating on the bpi-r4 in my use case. i expect when multiple ppe comes online it will have ticked all my checkbox.

again, a huge thank you.

Would you need bridger when using hw-nat between let’s say eth1 and lanbr0 (lan1 + lan2 + lan3 + lan4 bridged)?

yes. if you have a bridge consisting of eth1 + ports on the dsa switch. no if eth1 is an independent routable interface and not part of the bridge consisting of lan1+lan2+lan3+lan4

this just gave me an idea that may simplify the management of the interfaces and vlans as I manually define vlans base on ports and not through the bridge interface.

with bridger, i can actually create a big bridge consisting of eth1+ 4x1Gbe then use luci to create and manage vlans… interesting…

Hmz, I thought you would need to add connection tracking on the bridged ports to be able to do hardware offloaded nat with bridged interfaces.

Anyway, I feel a new autumn project coming along… See if it woiks using nftables for connection tracking the bridged interfaces. No time for it now.