[BPI-R4 | Solved] Bad 10gbit sfp performance?

oli · May 7, 2024, 10:49am

Hi, i just got my 10gbit connection but somehow i only get about 1gbit (up+down). So i am not sure if this is a provider problem or something in the r4/openwrt.

root@OpenWrt:~# ethtool eth2
Settings for eth2:
	Supported ports: [ FIBRE ]
	Supported link modes:   10000baseLR/Full 
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10000baseLR/Full 
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 10000Mb/s
	Duplex: Full
	Port: FIBRE
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: on
	Current message level: 0x000000ff (255)
			       drv probe link timer ifdown ifup rx_err tx_err
	Link detected: yes

I already rebooted, running openwrt snapshot from maybe a week ago. To me, it looks ok on my side (imo), so probably an ISP problem? Anyone an idea how to figure out more?

edit: it cant be lan3 as this is a 1gbit port i changed it now to the first sfp+ rj45 port, but still only 1gbit. so i checked eth1 (thats the port)

root@OpenWrt:/tmp# ethtool eth1
Settings for eth1:
	Supported ports: [ FIBRE ]
	Supported link modes:   1000baseX/Full 
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  1000baseX/Full 
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Link partner advertised link modes:  1000baseX/Full 
	Link partner advertised pause frame use: Symmetric Receive-only
	Link partner advertised auto-negotiation: Yes
	Link partner advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Port: FIBRE
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: on
	Current message level: 0x000000ff (255)
			       drv probe link timer ifdown ifup rx_err tx_err
	Link detected: yes

looks to me there is really a problem with this auto-negotiation thingen?

root@OpenWrt:/tmp# ethtool -A eth1 rx off tx off autoneg off 
Cannot get device pause settings: Not supported
root@OpenWrt:/tmp# ethtool -s eth1 speed 10000
Cannot advertise speed 10000

is this sfp-rj45-adapter crap?

edit 2:

root@OpenWrt:/tmp# ./speedtest-netperf.sh -H netperf-eu.bufferbloat.net
2024-05-07 11:12:03 Starting speedtest for 60 seconds per transfer session.
Measure speed to netperf-eu.bufferbloat.net (IPv4) while pinging gstatic.com.
Download and upload sessions are sequential, each with 5 simultaneous streams.
.............................................................
 Download: 2264.54 Mbps
  Latency: [in msec, 61 pings, 0.00% packet loss]
      Min:   2.254
    10pct:   2.310
   Median:   2.466
      Avg:   2.471
    90pct:   2.614
      Max:   2.709
 CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 58 samples]
     cpu0:  55.5 +/-  5.3  @ 1500 MHz
     cpu1:  13.7 +/-  3.2  @ 1500 MHz
     cpu2:  21.1 +/-  6.8  @ 1500 MHz
     cpu3:  13.7 +/-  3.5  @ 1500 MHz
 Overhead: [in % used of total CPU available]
  netperf:   8.6
.............................................................
   Upload: 3731.88 Mbps
  Latency: [in msec, 61 pings, 0.00% packet loss]
      Min:   2.213
    10pct:   2.239
   Median:   2.343
      Avg:   2.377
    90pct:   2.525
      Max:   2.680
 CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 58 samples]
     cpu0:  16.0 +/-  4.1  @ 1500 MHz
     cpu1:   8.5 +/-  2.7  @ 1500 MHz
     cpu2:  17.4 +/-  1.4  @ 1500 MHz
     cpu3:   6.4 +/-  2.6  @ 1500 MHz
 Overhead: [in % used of total CPU available]
  netperf:   5.0

So it seems like this sfp-rj45-adapter is crap i guess? Or is this a driver problem?

frank-w · May 7, 2024, 11:57am

From where to where do you test? Switchports (existing rj45 - wan,lan0-lan2) are only 1g.

10g are only the sfp (eth1 and eth2). Eth2 shows 10g,eth1 only 1g linkspeed.

Please not that we have not yet 10G working,only ~4.5g due to missing rss/lro support in mainline code

oli · May 7, 2024, 12:32pm

i test from: my workstation → eth1 (rj45 sfp+) → speedtest.net

eth1 is a sfp+rj45-10gbit-adapter, so i assume there is some problem with the driver

frank-w · May 7, 2024, 1:18pm

Please try without forwarding,step by step,so if your provider is connected on eth2 (wan sfp) then make speedtest over this only first.

oli · May 7, 2024, 5:08pm

thats what i did, and reached about 2.5gbit down and 3.5gbit upload, see above the speedtest-netperf.sh

frank-w · May 7, 2024, 5:13pm

Ok,sorry,so this test was eth2,right? As eth1 is only supporting 1000Mbit/s (1G) speed acording to ethtool…are you sure is should be a 10G adapter?

What does ethtool -m eth1 say and maybe dmesg?

oli · May 7, 2024, 5:33pm

i will answer in a couple days, i am now at another location for the next 3-4 days. it should be 100% a 10g rj45-sfp-adapter

jcdutton · May 7, 2024, 7:44pm

I think one of the SFP is only 2.5Gbps to data throughput. The other one I have tested at 5Gbps so trying in the other socket might be worth a try. Some extra features like LRO and RSS are needed to reach higher than that.

frank-w · May 8, 2024, 5:05pm

For me it sounds like the 3.5g/2.5g are ok (maximum from ISP) and only the other sfp is the problem which only supports 1g mode based on ethtool output

oli · May 8, 2024, 7:25pm

close, yes. Max from ISP is 10/10, but its totally fine for me to have only 2 or 3gbit as long as no hw acceleration is possible. The other SFP is indeed a problem. It is this one: https://www.aliexpress.us/item/3256805471213618.html

frank-w · May 8, 2024, 8:17pm

Without rss/lro you should get ~4.5gbit/s if your isp supports it…

oli · May 8, 2024, 10:20pm

maybe speedtest.net does not provide more

i will test again when i am back home

oli · May 10, 2024, 2:40pm

measured again:

root@OpenWrt:/# speedtest-netperf.sh -H netperf-eu.bufferbloat.net
2024-05-10 14:37:24 Starting speedtest for 60 seconds per transfer session.
Measure speed to netperf-eu.bufferbloat.net (IPv4) while pinging gstatic.com.
Download and upload sessions are sequential, each with 5 simultaneous streams.
.............................................................
 Download: 2372.16 Mbps
  Latency: [in msec, 61 pings, 0.00% packet loss]
      Min:   2.140
    10pct:   2.258
   Median:   2.446
      Avg:   2.435
    90pct:   2.562
      Max:   2.767
 CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 58 samples]
     cpu0:  56.2 +/-  3.2  @ 1500 MHz
     cpu1:  13.8 +/-  3.2  @ 1500 MHz
     cpu2:  20.5 +/-  3.1  @ 1500 MHz
     cpu3:  13.1 +/-  3.2  @ 1500 MHz
 Overhead: [in % used of total CPU available]
  netperf:   9.1
.............................................................
   Upload: 3914.68 Mbps
  Latency: [in msec, 61 pings, 0.00% packet loss]
      Min:   2.131
    10pct:   2.191
   Median:   2.278
      Avg:   2.328
    90pct:   2.475
      Max:   3.155
 CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 58 samples]
     cpu0:  16.4 +/-  4.4  @ 1500 MHz
     cpu1:   7.8 +/-  2.7  @ 1500 MHz
     cpu2:  17.5 +/-  0.9  @ 1500 MHz
     cpu3:   6.5 +/-  3.0  @ 1500 MHz
 Overhead: [in % used of total CPU available]
  netperf:   5.5

root@OpenWrt:~# ethtool -m eth1
Offset		Values
------		------
0x0000:		03 04 07 00 00 00 01 00 00 00 01 01 0d 00 00 00 
0x0010:		08 03 00 1e 4f 45 4d 20 20 20 20 20 20 20 20 20 
0x0020:		20 20 20 20 00 00 1b 21 53 46 50 2b 31 30 47 2d 
0x0030:		54 2d 49 20 20 20 20 20 41 20 20 20 03 52 00 4d 
0x0040:		00 1a 00 00 57 45 50 4f 51 54 42 5a 31 30 30 31 
0x0050:		20 20 20 20 32 33 30 39 31 35 20 20 68 f0 05 a9 
0x0060:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0070:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0080:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0090:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x00a0:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x00b0:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x00c0:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x00d0:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x00e0:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x00f0:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0100:		5f 00 d3 00 55 00 d8 00 8c a0 75 30 88 b8 79 18 
0x0110:		1d 4c 01 f4 19 64 03 e8 4d f0 06 30 3d e8 06 f2 
0x0120:		2b d4 00 c7 27 10 00 df 00 00 00 00 00 00 00 00 
0x0130:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0140:		00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 
0x0150:		01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 23 
0x0160:		2d 40 83 bd 0b b8 13 90 18 a0 00 00 00 00 00 00 
0x0170:		00 00 00 00 00 00 00 00 00 00 00 ff ff ff ff 00 
0x0180:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x0190:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x01a0:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x01b0:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x01c0:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x01d0:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x01e0:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
0x01f0:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

dmesg log at:

The dmesg log is interesting. It seems like as if it tries to use 10gbit but falls back to 1gbit.

phentx · May 11, 2024, 9:34am

Okay so i’ve checked this with support, and they say vendor image has no problem.

frank-w · May 11, 2024, 9:44am

which support (bpi/sfp vendor)? yes, we have not yet rss/lro in mainline-code…it is still in progress…we still porting some other parts.

phentx · May 11, 2024, 9:49am

BPI support, by vendor image i mean the one from MTK.

oli · May 11, 2024, 9:49am

it seems to me that this is the wrong thread you are posting to?

phentx · May 11, 2024, 10:02am

No I don’t think so.

oli · May 11, 2024, 10:14am

Ok, can you then please tell me what your posting

has to do with my problem with a bad sfp+ performance with that 10gbit rj45 adapter? Because i dont get it. Or did you check my problem with the BPI support?

phentx · May 11, 2024, 10:39am

Again, vendor image from MTK has no problem reaching 10gbps on sfp+ port. This is what they said.

OpenWRT on the other hand is stuck with ~4.5gbps.