BPI-R2 slow ethernet speed

Banana Pi Router design
41

@bluelightning32 - thank you very much for the extensive test results. Unusual behavior! nothing obvious springs to mind, the relative performance is definitely indicative of something, it narrows down the possibilities for sure. It ‘feels’ like a bus contention issue or some kind of buffer-overflow problem as you suggest - but it’s nothing that I’ve personally encountered before with networking gear

42

Glad see the damn breaking on this issue. I had to resort to other hardware to keep my project going but would circle back to the R2 if someone needs a solution tested here. bluelightning32’s results appear consistent with with problems/experiments I did.

43

Any updates on this?

I’m observing a similar problem, but much worse than @bluelightning32. When downloading through the Masquerading of the BPI R2 I only get a few hundreds of kbit/s. This makes the R2 not usable as a router for me right now.

When downloading to the Pi itself or uploading through it the speed is ok, though. I can also confirm, that there is a difference between multiple kinds of data. With if=/dev/zero it is fast, but with if=/dev/urandom it is slow. Also just downloading files through it (e.g. debian dvd iso) is around the same few hundreds of kbit/s.

I’ve tried with the official (https://github.com/BPI-SINOVOIP/BPI-R2-bsp) and @frank-w s (https://github.com/frank-w/BPI-R2-4.14) kernel and also with hwnat(http://www.fw-web.de/dokuwiki/doku.php?id=en:bpi-r2:hwnat) on.

I will be doing some more detailed tests later this week and will post the output as well as my configuration here.

44

disable eth0 tso with ethtool and see if it helps the issue as following commands

ethtool -K eth0 tso off

45

nope, that does not help

46

Does not seem to help me either. I think it made it worse for me, tried to undo it by calling ethtool -K eth0 tso on, but that didn’t seem to help, so maybe it got worse because of something else.

I also finally got to doing some measurements. My setup is:

Server <–> [WAN] BananaPi [LAN] <–> Laptop

Currently the LAN is bridged with another lan on the Pi and hwnat is on. But I tried earlier without these and it didn’t seem to help. If I have time I will run the measurements again with a fresh bare minimum setup.

I ran three different tests:

  1. dd if=/dev/zero | pv | netcat 9999 and on the other side netcat -lp 9999 | pv | dd of=/dev/null
  2. dd if=/dev/urandom | pv | netcat 9999 and on the other side the same as above
  3. Hosting a debian netinst image with apache2 and wget to download it.

So here are the numbers:

Server <—> Pi

netcat zero

Server -> Pi

Roughly 60MB/s -> 480MBit/s

Pi -> Server

Roughly 65MB/s -> 520MBit/s

netcat urandom

Server -> Pi

Roughly 7MB/s -> 56MBit/s

Pi -> Server

Roughly 9MB/s -> 72MBit/s

wegt debian iso

Server -> Pi

Roughly 6MB/s -> 48MBit/s

Pi -> Server

Roughly 109MB/s -> 872MBit/s

Pi <—> Laptop

netcat zero

Pi -> Laptop

Roughly 0.6MB/s -> 4.8MBit/s

Laptop -> Pi

Roughly 80MB/s -> 640MBit/s

netcat urandom

Pi -> Laptop

Roughly 0.0001MB/s -> 0.0008MBit/s

Laptop -> Pi

Roughly 60MB/s -> 480MBit/s

wegt debian iso

Pi -> Laptop

Roughly 0.1MB/s -> 0.8MBit/s Getting slower a lot after a few seconds

Laptop -> Pi

Roughly 15MB/s -> 120MBit/s

Server <–Pi–> Laptop

netcat zero

Server -> Laptop

Roughly 12MB/s -> 96MBit/s

Laptop -> Server

Roughly 55MB/s -> 440MBit/s

netcat urandom

Server -> Laptop

To slow to measure

Laptop -> Server

Roughly 55MB/s -> 440MBit/s

wegt debian iso

Server -> Laptop

To slow to measure

Then I investigated the to slow to measure netcat a bit. I tried setting the blocksize of dd to 100 bytes. Then around 3000 to 5000 bytes went through before it stopped. When setting the blocksize to 10 bytes around 20000 went through.

Keep in mind, something may be wrong with my settings as it was not that bad before(at least some kbit/s instead of to slow to measure). As I said i will try with a fresh image once I have time.

47

I did some similar speed tests cu the BPI-R2 device and on the wan/ppp0 interface a get at most 250Mbps. The tests were done using latest kernel 4.19.27 and 4.14.104 (Frank github repo) with HW NAT enabled for 4.14 but no difference in speed. I’m using Debian 10 Buster.

I have a 1Gbps Internet connection.

With my old gigabit TP-link 1043nd v2 router running DD-WRT with HW NAT enabled I get over 900Mbps on the speed tests. I’ll try and put the latest OpenWRT image 18.06 and check if I get better speeds.

Just saw in the log some crashes and some timeouts related to the network interfaces.

    Mar 09 00:36:17 bpi-r2 kernel: WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:320 dev_watchdog+0x27c/0x280
Mar 09 00:36:17 bpi-r2 kernel: NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
Mar 09 00:36:17 bpi-r2 kernel: Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp aes_arm_bs crypto_simd cryptd tun des_generic ip6table_nat nf_conntr
Mar 09 00:36:17 bpi-r2 kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: G           O    4.14.104-bpi-r2-main #1
Mar 09 00:36:17 bpi-r2 kernel: Hardware name: Mediatek Cortex-A7 (Device Tree)
Mar 09 00:36:17 bpi-r2 kernel: [<c0113040>] (unwind_backtrace) from [<c010d2ec>] (show_stack+0x20/0x24)
Mar 09 00:36:17 bpi-r2 kernel: [<c010d2ec>] (show_stack) from [<c0aac344>] (dump_stack+0x90/0xa4)
Mar 09 00:36:17 bpi-r2 kernel: [<c0aac344>] (dump_stack) from [<c0125d34>] (__warn+0xf8/0x110)
Mar 09 00:36:17 bpi-r2 kernel: [<c0125d34>] (__warn) from [<c0125d94>] (warn_slowpath_fmt+0x48/0x50)
Mar 09 00:36:17 bpi-r2 kernel: [<c0125d94>] (warn_slowpath_fmt) from [<c08deca8>] (dev_watchdog+0x27c/0x280)
Mar 09 00:36:17 bpi-r2 kernel: [<c08deca8>] (dev_watchdog) from [<c019b194>] (call_timer_fn+0x50/0x198)
Mar 09 00:36:17 bpi-r2 kernel: [<c019b194>] (call_timer_fn) from [<c019b3c8>] (expire_timers+0xec/0x14c)
Mar 09 00:36:17 bpi-r2 kernel: [<c019b3c8>] (expire_timers) from [<c019b4d0>] (run_timer_softirq+0xa8/0x1c0)
Mar 09 00:36:17 bpi-r2 kernel: [<c019b4d0>] (run_timer_softirq) from [<c01015dc>] (__do_softirq+0xec/0x370)
Mar 09 00:36:17 bpi-r2 kernel: [<c01015dc>] (__do_softirq) from [<c012c800>] (irq_exit+0xe4/0x14c)
Mar 09 00:36:17 bpi-r2 kernel: [<c012c800>] (irq_exit) from [<c0180e34>] (__handle_domain_irq+0x70/0xc4)
Mar 09 00:36:17 bpi-r2 kernel: [<c0180e34>] (__handle_domain_irq) from [<c01014a8>] (gic_handle_irq+0x5c/0xa0)
Mar 09 00:36:17 bpi-r2 kernel: [<c01014a8>] (gic_handle_irq) from [<c010df8c>] (__irq_svc+0x6c/0x90)
Mar 09 00:36:17 bpi-r2 kernel: Exception stack(0xde947f38 to 0xde947f80)
Mar 09 00:36:17 bpi-r2 kernel: 7f20:                                                       00000000 00061e34
Mar 09 00:36:17 bpi-r2 kernel: 7f40: 1e51e000 c011f720 de946000 c1103ccc c1103c6c c11bc91f c0d2bd68 00000001
Mar 09 00:36:17 bpi-r2 kernel: 7f60: 00000000 de947f94 de947f98 de947f88 c0109818 c010981c 600e0013 ffffffff
Mar 09 00:36:17 bpi-r2 kernel: [<c010df8c>] (__irq_svc) from [<c010981c>] (arch_cpu_idle+0x48/0x4c)
Mar 09 00:36:17 bpi-r2 kernel: [<c010981c>] (arch_cpu_idle) from [<c0ac77f8>] (default_idle_call+0x30/0x3c)
Mar 09 00:36:17 bpi-r2 kernel: [<c0ac77f8>] (default_idle_call) from [<c016d808>] (do_idle+0xdc/0x14c)
Mar 09 00:36:17 bpi-r2 kernel: [<c016d808>] (do_idle) from [<c016db14>] (cpu_startup_entry+0x28/0x2c)
Mar 09 00:36:17 bpi-r2 kernel: [<c016db14>] (cpu_startup_entry) from [<c011092c>] (secondary_start_kernel+0x170/0x194)
Mar 09 00:36:17 bpi-r2 kernel: [<c011092c>] (secondary_start_kernel) from [<801018ec>] (0x801018ec)
Mar 09 00:36:17 bpi-r2 kernel: ---[ end trace 593bb1a22e52195d ]---
Mar 09 00:36:17 bpi-r2 kernel: mtk_soc_eth 1b100000.ethernet eth0: transmit timed out
Mar 09 00:36:17 bpi-r2 kernel: mtk_soc_eth 1b100000.ethernet eth1: transmit timed out
48

Is this crash in debian/openwrt and which kernel 4.14 or 4.19? Same behaviour with the other kernel/system?

49

This crash is from Debian running kernel 4.14.104. I will try and see if I can replicate same crash with kernel 4.19.27.

50

i know this timeouts from porting 2nd gmac to 4.19. i realized that i only need changes in dsa and dts so only this is changed for 2nd gmac in 4.19. But i can’t reproduce the timeouts in 4.14 after i knew it…

you can try to revert a293bdb3d53aac4216fb379f84aa25aa65aaa3ef (at least changes in drivers/net/ethernet/mediatek/mtk_eth_soc.c/h BQL/DMA/CDMP) in 4.14-main and look if this problem is gone.

The main-problem with decreased ethernet-Speed (i also can’t reproduce…i have now 2 lan+wan connected and no problem) was already in 4.19 without my dsa-Patches but maybe you can fix the watchdog-warning for transmit-timeout

51

Today I did test with the 5.0.0 kernel from the 5.0-phy branch.

The wan interface is missing, so I am using eth1 as wan. eth1 is not comming up automatically, BUT:
I do get my 400Mbit/s now :slight_smile:

On the iPhone I do get 180Mbit/s

All devices connected directly to the BPi-r2.

52

5.0-phy is currently in testing (and only working till yesterday)…2nd gmac is outside dsa-driver,so no wan…also lan-ports named from 1 till 4 (instead of counting from 0 to 3) but this should be easy to fix in dts (i have not done that to don’t interfere rene’s patches).

but it’s nice having more testers…can anybody (who have currently problema) test thos codebase if there are problems on lan-ports (multiple connections)?

53

I did not recognize that the lan port numbering has changed, I am currently not using the 4th port, thank you for the hint.

Do you have any idea why it works much better for me with this kernel? Is there anything I could test to help you?

Maybe interesting part of bootlog:

[   12.856354] mt7530 mdio-bus:01: Linked as a consumer to regulator.6
[   12.862840] mt7530 mdio-bus:01: Linked as a consumer to regulator.17
[   12.869210] mt7530 mdio-bus:01: GPIO lookup for consumer reset
[   12.875100] mt7530 mdio-bus:01: using device tree for GPIO lookup
[   12.881263] of_get_named_gpiod_flags: parsed 'reset-gpios' property of node '/ethernet@1b100000/mdio-bus/switch@1    [0]' - status (0)
[   12.893054] gpio gpiochip0: Persistence not supported for GPIO 33
[   12.901554] Setup P5 HWTRAP: 0x1115e9f, phy-mode: PHY P0
[   12.907739] mt7530_port_disable: P0
[   12.912671] mt7530_port_disable: P1
[   12.916852] mt7530_port_disable: P2
[   12.917249] A link change request failed with some changes committed already. Interface br0 may have been left with an inconsistent configuration, please check.
[   12.921073] mt7530_port_disable: P3
[   12.938876] mt7530_port_disable: P4
[   12.939546] mtk_validate: rgmii
[   12.944317] mt7530_port_disable: P5
[   12.948039] mtk_soc_eth 1b100000.ethernet eth1: PHY     [mdio-bus:00] driver     [Generic PHY]
[   12.948764] mtk_soc_eth 1b100000.ethernet eth1: configuring for phy/rgmii link mode
[   12.953611] mt7530_cpu_port_enable: P6
[   12.960791] mtk_mac_config_hw: GMAC1: mode rgmii
[   12.960800] mtk_mac_config: GMAC1: M0: (rgmii) mcr:1056300 pause=10
[   12.960979] A link change request failed with some changes committed already. Interface eth1 may have been left with an inconsistent configuration, please check.
[   12.970403] mdio_bus dsa-0.0: GPIO lookup for consumer reset
[   12.973102] mtk_mac_config: GMAC1: M0: (rgmii) mcr:105e300 pause=0
[   12.977663] mdio_bus dsa-0.0: using lookup tables for GPIO lookup
[   12.984578] mtk_mac_config: GMAC1: M0: (rgmii) mcr:105e300 pause=0
[   12.995197] mtk_soc_eth 1b100000.ethernet eth0: configuring for fixed/trgmii link mode
[   12.995266] mtk_mac_config_hw: GMAC0: mode trgmii
[   12.995274] mtk_mac_config: GMAC0: M1: (trgmii) mcr:1056300 pause=12
[   12.995444] A link change request failed with some changes committed already. Interface eth0 may have been left with an inconsistent configuration, please check.
[   12.995490] mtk_mac_config: GMAC0: M1: (trgmii) mcr:1056300 pause=12
[   12.995495] mtk_mac_link_up: GM0
[   12.995518] mtk_soc_eth 1b100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[   12.995535] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   12.998464] mdio_bus dsa-0.0: No GPIO consumer reset found
[   13.115769] libphy: dsa slave smi: probed
[   13.119855] mt7530_get_phy_flags: P1
[   13.123762] mt7530 mdio-bus:01 lan1 (uninitialized): PHY     [dsa-0.0:01] driver     [Generic PHY]
[   13.132873] mt7530_get_phy_flags: P2
[   13.136738] mt7530 mdio-bus:01 lan2 (uninitialized): PHY     [dsa-0.0:02] driver     [Generic PHY]
[   13.145945] mt7530_get_phy_flags: P3
[   13.150160] mt7530 mdio-bus:01 lan3 (uninitialized): PHY     [dsa-0.0:03] driver     [Generic PHY]
[   13.159364] mt7530_get_phy_flags: P4
[   13.164279] mt7530 mdio-bus:01 lan4 (uninitialized): PHY     [dsa-0.0:04] driver     [Generic PHY]
[   13.174326] of_get_named_gpiod_flags: can't parse 'link-gpios' property of node '/ethernet@1b100000/mdio-bus/switch@1/ports/port@6/fixed-link    [0]'
[   13.205043] of_get_named_gpiod_flags: can't parse 'reset-gpios' property of node '/ethernet@1b100000/mdio-bus/switch@1/ports/port@6    [0]'
[   13.217423] mt7530 mdio-bus:01: mt7530_adjust_link: P6 phy-mode trgmii
[   13.228076] DSA: tree 0 setup
[   13.291159] br0: port 1(lan1) entered blocking state
[   13.296196] br0: port 1(lan1) entered disabled state
[   13.301463] device lan1 entered promiscuous mode
[   13.309197] device eth0 entered promiscuous mode
[   13.313920] mt7530_port_enable: P1
[   13.317810] mt7530 mdio-bus:01 lan1: configuring for phy/gmii link mode
[   13.324613] mt7530_phylink_mac_config P1, mode: 0, gmii, mcr=56300 pause=10
[   13.331839] mt7530 mdio-bus:01: Unsupported interface: 0
[   13.337946] mt7530_phylink_mac_config P1, mode: 0, gmii, mcr=5e300 pause=0
[   13.346625] br0: port 2(lan2) entered blocking state
[   13.351578] br0: port 2(lan2) entered disabled state
[   13.357181] device lan2 entered promiscuous mode
[   13.365054] br0: port 3(lan3) entered blocking state
[   13.370009] br0: port 3(lan3) entered disabled state
[   13.375658] device lan3 entered promiscuous mode
[   13.453278] mt7530_port_enable: P2
[   13.457210] mt7530 mdio-bus:01 lan2: configuring for phy/gmii link mode
[   13.463961] mt7530_phylink_mac_config P2, mode: 0, gmii, mcr=56300 pause=10
[   13.471119] mt7530 mdio-bus:01: Unsupported interface: 0
[   13.477216] mt7530_phylink_mac_config P2, mode: 0, gmii, mcr=5e300 pause=0
[   13.484660] mt7530_port_enable: P3
[   13.488553] mt7530 mdio-bus:01 lan3: configuring for phy/gmii link mode
[   13.495317] mt7530_phylink_mac_config P3, mode: 0, gmii, mcr=56300 pause=10
[   13.502546] mt7530 mdio-bus:01: Unsupported interface: 0
[   13.508544] mt7530_phylink_mac_config P3, mode: 0, gmii, mcr=5e300 pause=0
[   15.612157] mt7530_phylink_mac_config P3, mode: 0, gmii, mcr=5e307 pause=0
[   15.619030] mt7530 mdio-bus:01 lan3: Link is Up - 100Mbps/Full - flow control off
[   15.626554] IPv6: ADDRCONF(NETDEV_CHANGE): lan3: link becomes ready
[   15.633023] br0: port 3(lan3) entered blocking state
[   15.637973] br0: port 3(lan3) entered forwarding state
[   15.643852] IPv6: ADDRCONF(NETDEV_CHANGE): br0: link becomes ready
[   16.093523] mtk_mac_config: GMAC1: M0: (rgmii) mcr:105e33b pause=e
[   16.099701] mtk_soc_eth 1b100000.ethernet eth1: Link is Up - 1Gbps/Full - flow control rx/tx
[   16.108174] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   16.492552] mt7530_phylink_mac_config P1, mode: 0, gmii, mcr=5e33b pause=f
[   16.499419] mt7530 mdio-bus:01 lan1: Link is Up - 1Gbps/Full - flow control rx/tx
[   16.521775] IPv6: ADDRCONF(NETDEV_CHANGE): lan1: link becomes ready
[   16.528208] br0: port 1(lan1) entered blocking state
[   16.533205] br0: port 1(lan1) entered forwarding state
[   16.572045] mt7530_phylink_mac_config P2, mode: 0, gmii, mcr=5e30b pause=0
[   16.578923] mt7530 mdio-bus:01 lan2: Link is Up - 1Gbps/Full - flow control off
[   16.586261] IPv6: ADDRCONF(NETDEV_CHANGE): lan2: link becomes ready
[   16.592788] br0: port 2(lan2) entered blocking state
[   16.598886] br0: port 2(lan2) entered forwarding state
54

i do not know the changes in detail, but there are much changes in ethernet-driver so maybe we dropped some bad code…currently we try to cleanup code

there are also some debug-messages present, your error in bootlog may came from lan-port-numbering (lan0 does not exist atm)

55

After 1 week of testing I did not recognize any issues. Also dmesg is not showing any issues.

(I did only test the network, HDMI is not working and I did not test the build in WiFi. Docker works fine)

56

Hdmi in 5.0 only support xserver (no fbcon) in separate branch only (5.0-hdmi).internal wifi-driver only works in 5.0-wifi…

Is the speed-issue fixed in 5.0-phy?

57

From my point of view it is fixed. I do get 400 Mbit/s when downloading

  • large files with pacman (archlinux package manager)
  • large files with Firefox
  • games from gogs (simple download via Firefox)
  • games from steam (parallel downloads with steam client)

Excellent quality when streaming from

:slight_smile:

Thank you very much for your hard work!

It seems nobody else did test this branch. Maybe you could create a release to make it easier for people to test without the need to compile themselves.

58

it’s not my work…i only tested patches from rene van dorst and give some feedback

I will compile this branch and upload to my gdrive for testing

btw. i have added 5.0-phylink where lan-ports start on 0 (-3), have currently no idea how to get wan instead of eth1 in code…maybe by udev-rule

59

Are the changes from this branch https://github.com/vDorst/linux-1/tree/v5.0.1-mt762x-phylink-hnat integrated into your 5.0-phylink branch ? I’m planning also to do some tests with these changes.

60

the patches are from gmac2-fix-branch…hnat is not included

have uploaded phylink-branch to my gdrive:

https://drive.google.com/open?id=17MoFc3vIuGHDEV5SsCmGegJDr009IXls

bpi-r2_5.0.0_-phylink.tar.gz