BPI-R2 slow ethernet speed

That is interesting. Here is the summary of results so far

pause enable on switch and gmac -> I can consistently can trigger the timeout, somewhere between 15min and 8hrs of load testing

No pause enabled -> No Timeout

Pause enabled on switch only -> No Timeout

Pause enabled on GMAC only -> No Timeout so far… looking good but needs to run longer to be sure

ipref - Nopause -> Need to test

ipref - pause on switch -> ~860 mbps

ipref - pause on gmac -> ~820 mpbs

I am testing with a cable directly from a macbook to the r2. Now the ethernet on the macbook is connected via the USB bus; so I don’t know how good it really is. I’ll need to cable up to my network and test beetween the macbook and a PC to see if that isn’t the ~800mbs bottleneck. It could be.

Other notes. I don’t have the wifi enabled at all. I am still using a Slackware ARM user space a little after the 14.2 release. I really don’t think user space stuff figures into this much but its worth noting. I am compiling the kernel on the device, which is still running what is at this point an older gcc 7.3.0.

Maybe a new gcc builds code that runs a little quicker for the interrupt handlers? (speculating wildly)

Can I ask what you use to load test? My timeout trigger is running samba on the pie, sharing out a sata-sdd directory without about 40GB of files in it, mounting it on the client and doing cat * > /dev/null.

on my main-router (which is now running 7 days) i make normal traffic…on my test-r2 i use iperf3 for testing (r2 server or client connected over switch to my laptop). no samba or similar installed…only “router-tools” on main device and some additional things in lxc containers (vnc-server,web-server+mysql)

I was running samba in a contain, passing the lan1 port in as PHY, but I installed the package on the “host” and shut down all the containers just to take more testing variables out.

Well this is disapointing. I thought maybe I’d found a solid workaround with the pause frames; but appareently not. Alhough I did not get the kernel oops I did trigger the broken frames issues on anothere abuse test.

Does it work without the pause (both removed) or not? How is speed with iperf3 (both directions)?

I really though it was working with pause both removed. I had not been able to produce a crash so I left it running in a loop over the weekend; same job just keep cat’ing a smbshare full of files to dev/null.

While there was no oops, and no “eth0: timed out” in the log it did start producing broken packets. Ping would still work but not TCP. So the answer is with no pause enabled, it seems more stable but can still break.

So I guess while pause frames might create the conditons that often trigger the failure, they are not causal.

Could you try if 5.9-main is also affected? It has additional patch for supporting mtu-setting. Maybe packets are stripped off due to additional dsa-tag if they grow to max. Ping packets are not big,but tcp packets can go upto max-mtu and may be damaged if adding additional tag which results in packets larger than mtu-max.

I upgraded to the 5.9-main branch in your repo. So far same behaviors. The abuse test ran for hours followed by the system only being able to send short frames until a restart. Nothing in dmesg output, no opps, just not working.

I have not tried lowering the MTU yet. I started the test again with /proc/sys/net/core/dev_weight doubled from 64 to 128, in case its an interrupt handling thing. Assuming that does not work (I dont expect it will) is there an MTU value you’d suggest, I know you have spent a lot of time studying the hardware, any specific transmission size related corner case you are trying to avoid with that suggestion?

I see the default MTU on the eth0 MAC is 1504… while on the ports its 1500. The extra four bytes on the MAC are to handle the port tags and still allow normal 1500 size ethernet frames? What MTUs are you interested in my trying to lower first the MAC or the ports both? My unscientific instinct is maybe set the MAC to 1500 and the ports to either 1496 or handful of bytes fewer.

Not set MTU manually,i just wanted that driver is capable to handle setting mtu-change request (maybe from dsa core). Mhm then i guess it is not caused by MTU setting (is fixed in 5.4). Then i’m still digging in the dark…also because i do not see your errors on my machines. Is all correctly soldered?

If its a hardware issue its intrinsic to the design. I have two r2 units right now, both show the same behavior.

It hard to hit the errors, which is why its so hard to even know if a fix is actually a fix. Sometimes I have to keep it running an smb copy for 8+ hours before it breaks. So I am not suprised you don’t see the issues. Its not just samba either you can do something like btrfs send through netpipes to backup the attached disk and that may or may not succeed, I had the other unit die on me after sending 50GB once.

I’ve reverted my R2 to original kernel 4.4.70-BPI-R2-Kernel (from https://github.com/BPI-SINOVOIP/BPI-R2-bsp.git) And results are good - 900Mbps DL, 300Mbps UL. No problems on TCP traffic HTTPS download above 500Mbps. So it doesn’t look like HW issue.

In the future a plan to test SINOVOIP’s kernels 4.14. and 5.4


not being able to use the swtich ports individually is sorta of a deal breaker for me. At that point I’d have to put a swtich under it to break out vlans, and I might as well use raspbery pi machines with better vendor hardware support (sorry sinovoip but if not for Frank’s efforts your product would deliver on basically nothing it advertised.)

Sinovoips 4.14 and 5.4 are basicly mainline or parts taken from my repo…so i guess you will have same issues. 4.4 is too old,no port-separation and too much modified which is not documented…i had massive problems merging updates into 4.4 tree due to changed files

For now - it works. I don’t need port separation, however lack of the ‘swconfig’ is not good, but VLANS works as expected. I was thinking about those lanX interfaces - I don’t know how they are made in the kernel, but I see the MTU impact here. I prefer not to do anything special, the eth0/eth1 is enough for me, I can live with VLAN on all ports.


So with the device weight set higher. I got the ethernet timeout message again (5.9.0-main).

[Mon Oct 19 04:31:43 2020] ------------[ cut here ]------------
[Mon Oct 19 04:31:43 2020] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:443 dev_watchdog+0x300/0x304
[Mon Oct 19 04:31:43 2020] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
[Mon Oct 19 04:31:43 2020] Modules linked in: ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_conntrack xt_nat xt_REDIRECT veth iptable_mangle iptable_filter iptable_nat nf_nat fuse snd_usb_audio snd_hwdep snd_usbmidi_lib snd_rawmidi snd_seq_device ir_rc5_decoder rc_hauppauge au8522_dig tuner au8522_decoder au8522_common au0828 lima tveeprom gpu_sched spi_mt65xx videobuf2_vmalloc pwm_mediatek nvmem_mtk_efuse mtk_pmic_keys mt6577_auxadc mtk_thermal
[Mon Oct 19 04:31:43 2020] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.9.0-bpi-r2 #1
[Mon Oct 19 04:31:43 2020] Hardware name: Mediatek Cortex-A7 (Device Tree)
[Mon Oct 19 04:31:43 2020] Backtrace: 
[Mon Oct 19 04:31:43 2020] [<c010ce14>] (dump_backtrace) from [<c010d144>] (show_stack+0x20/0x24)
[Mon Oct 19 04:31:43 2020]  r7:00000009 r6:600f0113 r5:00000000 r4:c177f998
[Mon Oct 19 04:31:43 2020] [<c010d124>] (show_stack) from [<c06bc5a4>] (dump_stack+0xd0/0xe4)
[Mon Oct 19 04:31:43 2020] [<c06bc4d4>] (dump_stack) from [<c0125a7c>] (__warn+0xe8/0x100)
[Mon Oct 19 04:31:43 2020]  r7:00000009 r6:c1485e38 r5:00000000 r4:c1701ce8
[Mon Oct 19 04:31:43 2020] [<c0125994>] (__warn) from [<c0125b08>] (warn_slowpath_fmt+0x74/0xa0)
[Mon Oct 19 04:31:43 2020]  r9:df596500 r8:00000009 r7:c0c421d4 r6:000001bb r5:c1485e38 r4:c1485dfc
[Mon Oct 19 04:31:43 2020] [<c0125a98>] (warn_slowpath_fmt) from [<c0c421d4>] (dev_watchdog+0x300/0x304)
[Mon Oct 19 04:31:43 2020]  r8:ffffffff r7:c1703d00 r6:dd938000 r5:dd9382a8 r4:00000000
[Mon Oct 19 04:31:43 2020] [<c0c41ed4>] (dev_watchdog) from [<c01ac1dc>] (call_timer_fn+0x50/0x1a0)
[Mon Oct 19 04:31:43 2020]  r8:000a7cc8 r7:c0c41ed4 r6:dd9382a8 r5:00000100 r4:c17f635c
[Mon Oct 19 04:31:43 2020] [<c01ac18c>] (call_timer_fn) from [<c01ad620>] (run_timer_softirq+0x560/0x650)
[Mon Oct 19 04:31:43 2020]  r9:df596500 r8:00000122 r7:000a7cc8 r6:00000000 r5:00000000 r4:dd9382a8
[Mon Oct 19 04:31:43 2020] [<c01ad0c0>] (run_timer_softirq) from [<c0101508>] (__do_softirq+0x158/0x3d0)
[Mon Oct 19 04:31:43 2020]  r10:ffffe000 r9:00000082 r8:00000100 r7:c17f5e84 r6:00000001 r5:00000002
[Mon Oct 19 04:31:43 2020]  r4:c1703084
[Mon Oct 19 04:31:43 2020] [<c01013b0>] (__do_softirq) from [<c012d63c>] (irq_exit+0xd8/0xe4)
[Mon Oct 19 04:31:43 2020]  r10:c135f218 r9:e1003000 r8:de829000 r7:00000001 r6:00000000 r5:00000000
[Mon Oct 19 04:31:43 2020]  r4:ffffe000
[Mon Oct 19 04:31:43 2020] [<c012d564>] (irq_exit) from [<c018ab24>] (__handle_domain_irq+0x70/0xc4)
[Mon Oct 19 04:31:43 2020]  r5:00000000 r4:c16a420c
[Mon Oct 19 04:31:43 2020] [<c018aab4>] (__handle_domain_irq) from [<c010136c>] (gic_handle_irq+0x5c/0xa0)
[Mon Oct 19 04:31:43 2020]  r9:e1003000 r8:c1701ec8 r7:e1002000 r6:e100200c r5:c177fa58 r4:c1706834
[Mon Oct 19 04:31:43 2020] [<c0101310>] (gic_handle_irq) from [<c0100b0c>] (__irq_svc+0x6c/0x90)
[Mon Oct 19 04:31:43 2020] Exception stack(0xc1701ec8 to 0xc1701f10)
[Mon Oct 19 04:31:43 2020] 1ec0:                   00000000 062fa878 df59d8f4 c011e600 c17f67bc c1705f28
[Mon Oct 19 04:31:43 2020] 1ee0: c1705f70 00000001 00000001 c17f5628 c135f218 c1701f24 c1701f28 c1701f18
[Mon Oct 19 04:31:43 2020] 1f00: c0109720 c0109724 600f0013 ffffffff
[Mon Oct 19 04:31:43 2020]  r9:c1700000 r8:00000001 r7:c1701efc r6:ffffffff r5:600f0013 r4:c0109724
[Mon Oct 19 04:31:43 2020] [<c01096dc>] (arch_cpu_idle) from [<c0f30f80>] (default_idle_call+0x48/0x12c)
[Mon Oct 19 04:31:43 2020] [<c0f30f38>] (default_idle_call) from [<c015bc84>] (do_idle+0xf0/0x14c)
[Mon Oct 19 04:31:43 2020]  r7:00000001 r6:c1705f70 r5:c1705f28 r4:ffffe000
[Mon Oct 19 04:31:43 2020] [<c015bb94>] (do_idle) from [<c015bf90>] (cpu_startup_entry+0x28/0x30)
[Mon Oct 19 04:31:43 2020]  r10:e07fcd00 r9:c16568bc r8:00000001 r7:00000000 r6:c16568bc r5:c183b000
[Mon Oct 19 04:31:43 2020]  r4:000000d6 r3:c169a2a0
[Mon Oct 19 04:31:43 2020] [<c015bf68>] (cpu_startup_entry) from [<c0f2a728>] (rest_init+0xbc/0xc4)
[Mon Oct 19 04:31:43 2020] [<c0f2a66c>] (rest_init) from [<c1600a9c>] (arch_call_rest_init+0x18/0x1c)
[Mon Oct 19 04:31:43 2020]  r5:c183b000 r4:c1705f00
[Mon Oct 19 04:31:43 2020] [<c1600a84>] (arch_call_rest_init) from [<c1600ef0>] (start_kernel+0x3d8/0x3fc)
[Mon Oct 19 04:31:43 2020] [<c1600b18>] (start_kernel) from [<00000000>] (0x0)
[Mon Oct 19 04:31:43 2020] ---[ end trace 5e289ae5c8833b6b ]---
[Mon Oct 19 04:31:43 2020] mtk_soc_eth 1b100000.ethernet eth0: transmit timed out
[Mon Oct 19 04:31:43 2020] mtk_soc_eth 1b100000.ethernet eth0: Link is Down
[Mon Oct 19 04:31:43 2020] mtk_soc_eth 1b100000.ethernet eth0: configuring for fixed/trgmii link mode
[Mon Oct 19 04:31:43 2020] mtk_soc_eth 1b100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[Mon Oct 19 04:41:34 2020] zDumpCache.sh (6672): drop_caches: 3

How did you set this?

You can echo a value to /proc/sys/net/core/dev_weight

This should increase the budget for network intrupt t handling if I understand correctly.

does this mean, that interrupts are called more (after network-buffer filled less)?

my understanding is the parameter changes the amount of the time the system will spend running the interrupt handler draining packets from the queue. A higher dev_weight should me fewer interrupts that block longer. On a typical server machine this usually causes more latency but gets you a little more thruput.