Thanks for your test,can you please try with branch 5.1-main to know which problems are present in mainline (no net changes except wifi). Maybe revert wifi-changes there because this touches watchdog if you get same problems
Could anyone using 4.14 try reverting this patch:
And see if problem persists?
Thanks to @LeXa2 for finding this
Yep, as @frank-w already reported here I had managed to bisect my problem down to the patch 0027-net-next-mediatek-fix-DQL-support.patch. As soon as I remove/revert it the network stall under high load is gone. As a stress-test I had used iperf and iperf3 executed in client mode on R2 with test duration set to 6000 seconds. This testing was finished moments ago and everything went fine. So, having this problem sorted out I’m about to try to measure some performance figures of R2 in line with the performance testing scheme I’ve been describing earlier.
1 Like
So, performance benchmarks.
Initial setup
- Box A: x86-64 linux-based NAS, CPU AMD A4-5300 @3.4Ghz, MB Gigabyte GA-F2A88XM-HD3, built-in ethernet based on RTL8168e connected to local gigabit ethernet infrastructure.
- Box B: x86-64 ultrabook running debian, CPU Intel Core i5-4210U, USB3.0 gigabit ethernet card based on ax88179 chip.
- Box R2: Banana Pi R2
Test 0: “Getting the baseline without R2”
B connected to the local network infrastructure (i.e. R2 is not involved in this testing).
iperf3 B->A: ~920Mbps
iperf3 A->B: ~910Mbps
iperf A<->B (full duplex): ~795Mbps A->B, ~875Mbps B->A
Test 1: “Performance of R2 switch chip”
B connected to the lan3 port of R2. Port lan1 of R2 connected to the local network infrastructure.
R2 DSA interfaces lan1…4 bridged together into interface br-lan.
iperf3 B->A: ~920Mbps
iperf3 A->B: ~910Mbps
iperf A<->B (full duplex): ~798Mbps A->B, ~889Mbps B->A
Analysis: R2 switching speed for this usecase seem to be good enough and do not cause any slowdown compared to baseline numbers.
Test 2: “Is wan port any different to lanX?”
B connected to the wan port of R2. Port lan1 of R2 connected to the local network infrastructure.
R2 DSA interfaces lan1…4 and wan all bridged together into interface br-lan.
iperf3 B->A: ~920Mbps
iperf3 A->B: ~909Mbps
iperf A<->B (full duplex): ~798Mbps A->B, ~895Mbps B->A
Analysis: when bridged with other DSA ports wan port seems to behave exactly the same way as lanX ports do. Switching speed for this usecase seem to be good enough and do not cause any slowdown compared to baseline numbers.
Test 3: “R2 CPU routing performance”
B connected to the wan port of R2. Port lan1 of R2 connected to the local network infrastructure.
R2 DSA interfaces lan1…4 bridged together into interface br-lan. R2 interface wan assigned its own subnet, box B assigned IP address from this subnet. Routing rule added on A specifying R2 ip assigned to br-lan as gateway for wan’s subnet. Routing rule added on B specifying R2 ip assigned to wan as gateway for br-lan’s subnet. Iptables flushed on R2 and all polices are set to ACCEPT.
iperf3 B->A: ~840Mbps
iperf3 B->A (data from /dev/urandom): ~832Mbps
iperf3 A->B: ~780Mbps
iperf3 A->B (data from /dev/urandom): ~785Mbps
iperf A<->B (full duplex): ~694Mbps A->B, ~793Mbps B->A
Analysis: R2 raw CPU forwarding performance seems to be OK. No NAT or conntracking though for this testcase though. Entropy of the forwarded data seems to have no influence on throughtput.
Test 4: “R2 CPU NAT performance”
B connected to the wan port of R2. Port lan1 of R2 connected to the local network infrastructure.
R2 DSA interfaces lan1…4 bridged together into interface br-lan. R2 interface wan assigned its own subnet, box B assigned IP address from this subnet. Routing rule added on A specifying R2 ip assigned to br-lan as gateway for wan’s subnet. Routing rule added on B specifying R2 ip assigned to wan as gateway for br-lan’s subnet. Iptables on R2 configured with default OpenWRT fw3 to perform NAT for connections from br-lan to wan. No in-kernel mtk hwnat.
iperf3 B->A: ~751Mbps
iperf3 B->A (data from /dev/urandom): ~750Mbps
iperf3 A->B: ~615Mbps
iperf3 A->B (data from /dev/urandom): ~612Mbps
Analysis: R2 CPU-based NAT forwarding performance seems to be OK. Entropy of the forwarded data seems to have slight influence on throughtput but the scale of this influence is really low.
Conclusions
R2 seems to perform well enough to be used as a SOHO router. It’d be useful to perform additional tests involving two pairs of cross-communicating hosts to determine if switch chip in R2 is good enough to allow non-blocking commutation of two simultaneous full-duplex gigabit streams. Hardware NAT performance was not tested due to the 4.14 kernel used was not patched to include it.
1 Like
Which kernel/system have you used?
Could you try create >1gbit traffic (multiple iperf-processes)?
existing gmac should do trgmii (2.5gbit/s) but in my tests i get only 1gbit/s (lan+lan to local and wan+lan to local with second gmac active). I read (pointed by rene) in technical document that framengine is wired only with 1gbit/s which afaik makes trgmii impossible.
Kernel is basically the same as one will get compiling BananaPI BPI-R2 Openwrt18.06 Demo Image and Source Code Release 2019-03-06 image except for:
-
Applied DTS-fix patches:
0065-dts-fix-bpi-r2-memory-to-2GB.patch,0066-dts-mt7623-add-pcie.patchand0067-dts-bpi-r2-fix-second-gmac.patch. -
Removed patch:
0027-net-next-mediatek-fix-DQL-support.patch. -
Kernel config adjusted a bit:
diff --git a/target/linux/mediatek/mt7623/config-4.14 b/target/linux/mediatek/mt7623/config-4.14
index a38f02f2..b13675c6 100644
--- a/target/linux/mediatek/mt7623/config-4.14
+++ b/target/linux/mediatek/mt7623/config-4.14
@@ -1,4 +1,12 @@
# CONFIG_AIO is not set
+CONFIG_AHCI_MTK=y
+CONFIG_ATA=y
+CONFIG_SATA_AHCI=y
+CONFIG_SATA_AHCI_PLATFORM=y
+CONFIG_SCSI=y
+CONFIG_BLK_DEV_SD=y
+CONFIG_RTC_DRV_MT7622=y
+CONFIG_WATCHDOG_SYSFS=y
CONFIG_ALIGNMENT_TRAP=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_ARCH_HAS_DEBUG_VIRTUAL=y
@@ -33,7 +41,9 @@ CONFIG_ARM=y
CONFIG_ARM_APPENDED_DTB=y
CONFIG_ARM_ARCH_TIMER=y
CONFIG_ARM_ARCH_TIMER_EVTSTREAM=y
-# CONFIG_ARM_ATAG_DTB_COMPAT is not set
+CONFIG_ARM_ATAG_DTB_COMPAT=y
+CONFIG_ARM_ATAG_DTB_COMPAT_CMDLINE_FROM_BOOTLOADER=y
+# CONFIG_ARM_ATAG_DTB_COMPAT_CMDLINE_EXTEND is not set
CONFIG_ARM_CPU_SUSPEND=y
# CONFIG_ARM_CPU_TOPOLOGY is not set
CONFIG_ARM_GIC=y
@@ -490,3 +500,5 @@ CONFIG_ZBOOT_ROM_BSS=0
CONFIG_ZBOOT_ROM_TEXT=0
CONFIG_ZLIB_DEFLATE=y
CONFIG_ZLIB_INFLATE=y
+CONFIG_EXT4_FS=y
+CONFIG_VFAT_FS=y`
Not this time as (1) I had finally put this particular R2 under workload it was initially bought for and (2) I haven’t got enough traffic generating boxes on hand right now to do it. Will try to do it in future as I had ordered two more R2s and they are already on a half the way through from China to my place. Will have to borrow my girlfriend’s laptop to use as a “box C” in tests 
.
I’m not sure if R2 is able to perform fast enough to see the real benefit from trgmii even if all the HW wiring is in place for 2.5Gbps. Here are some interesting figures illustrating the max possible kernel network stack bandwidth with the kernel I use now on R2:
Opened two terminals connected to the R2. Then, in first terminal executed:
root@lx2bpir2:/# /etc/init.d/firewall stop ... lots of output stripped... root@lx2bpir2:/# iperf -s -i1 -p 5001 ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------Then in second terminal:
root@lx2bpir2:/etc/config# iperf -c 127.0.0.1 -i1 -t5 -p5001 ------------------------------------------------------------ Client connecting to 127.0.0.1, TCP port 5001 TCP window size: 2.50 MByte (default) ------------------------------------------------------------ [ 3] local 127.0.0.1 port 35816 connected with 127.0.0.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 316 MBytes 2.65 Gbits/sec [ 3] 1.0- 2.0 sec 315 MBytes 2.64 Gbits/sec [ 3] 2.0- 3.0 sec 317 MBytes 2.66 Gbits/sec [ 3] 3.0- 4.0 sec 318 MBytes 2.66 Gbits/sec [ 3] 4.0- 5.0 sec 318 MBytes 2.67 Gbits/sec [ 3] 0.0- 5.0 sec 1.55 GBytes 2.66 Gbits/sec
What is seen here is a fact that pure localhost TCP communication through in-kernel network layer maxed out at around 2.5Gbps - and this is with iptables cleared out and without diving deeper into kernel skb and netdev related stuff. Enabling a default OpenWRT firewall slows things down to ~2.2Gbps - this is the cost of having “-A INPUT -i lo -j ACCEPT” and “-A OUTPUT -o lo -j ACCEPT” at the top of resp. iptables chains. So I won’t be having a lot of hopes for seeing the benefit of trgmii connection between MACs and PHYs in R2. And for 2.5Gbps to be possible it is not enough to have trgmii link in GMAC as 2.5Gbps should be also active on the switch ASIC side for these MACs. Are you sure that this is the case?
1 Like
I’m not sure because rene says me that on available patches trgmii-mode is not initialized the right way and patch for second gmac brings another wrong setting: phy-mode have to be the same for mac0 and mac1…else there are damaged packets…both trgmii is not possible because only 1 gmac is really trgmii (the other only rgmii) and both rgmii leaves unused bandwidth of 1.5gbit/s (if trgmii is possible)
Well, to know for sure we need to somehow fetch and read R2 HW interconnection schematics and datasheets on all participating chips. My experience from digging through this kind of docs tells me that it won’t be an easy ride even if we’d get the mentioned docs into our hands 
.
From official specs on Mediatek site (quote) “MT7623N also implements 10/100/1000 Ethernet RGMII and TRGMII interfaces”. It leaves us with 3.5Gbps max theoretical bandwidth in case everything is fine with wiring and speed modes are supported on a switch ASIC side (which I expect they are). I’ve got doubts R2 is able to handle so much traffic when routing using CPU. It should be pretty easy to test though with four boxes. Test setup should be like this: lan0 and lan1 bridged together into br-lan, lan2 and lan3 bridged together into br-lan-1, br-lan and br-lan-1 set to use separate subnets, ip_forwarding enabled on R2, firewall cleared. Communication should be like lan0->lan2 and lan1->lan4 simultaneously. If aggregate bandwidth would end up any higher than 1Gbps then (a) trgmii is working and (b) I was underestimating R2. It’s only a matter of finding enough boxes and a moment to perform this test now.
Where do you get the dts-fix-patches (they are not in upstream)?
Created them myself using your kernel 4.14 as a base. There’s nothing new in there compared to what you already know and use.
have reverted the DQL-Patch in my 4.14-main, anyone who have Problems should test if this fixes the issue
release built by travis:
strange that 4.19/5.x is also reported to have the slowdown, where this Patch was not applied
btw. the ethernet-fixes-patch on 4.9 also reverts the following previous patch:
0029-net-next-ethernet-mediatek-add-CDM-able-to-recognize.patch
0054-net-ethernet-mediatek-fixed-deadlock-captured-by-loc.patch (spinlock-change)
and modifies handling of this (partially revert the patch) (restore old mtk_rx_alloc+mtk_w32-calls and add new mtk_rx_alloc_qdma,mtk_rx_clean_qdma):
0037-net-next-mediatek-bring-up-QDMA-RX-ring-0.patch (mtk_rx_alloc)
and this:
0051-net-mediatek-increase-tx_timeout.patch (watchdog_timeo 5=>30=>15)
and this (remove complete CDM-block):
0029-net-next-ethernet-mediatek-add-CDM-able-to-recognize.patch
0043-net-next-mediatek-enable-special-tag-indication-for-.patch
also removes new block introduced by
0042-net-next-mediatek-honour-special-tag-bit-inside-RX-D.patch (find out which mac the packet comes from)
and changes to new variant (not the old code)
Maybe we/i can use base 4.9 (https://github.com/frank-w/BPI-R2-4.14/tree/4.9-main_new) to split+squash patches together and see which code is not reverted and need to be ported
interesting are also the settings for cpumask_set_cpu and irq_set_affinity_hint which may be increasing speed in 4.9 but not in other kernel-versions
Well, it is important here to keep efforts structured. I mean, there’s a mediatek linux kernel team out there somewhere working on getting new features into the mainline. So I’d say that this work on tidying up patches and determining what to port and what to leave behind should be coordinated with those chaps.
It is also important to line up the goals for porting. I’d say that as end-users we’re interested in having properly working (1) mainline kernel and (2) all LTS kernels. From what I know 4.9 from your repo seems to be more or less working, at least when talking about DSA/networking stuff. At the same time 4.14 and 4.19 seem to need more love to be stable enough for R2 to be used as everyday router. Thus I’d outline priorities as pushing as hard as possible to get updated to use phylink layer driver into the mainline (as having it in stable mainline will result in getting it into the next LTS down the road) and as a second level prio we should be pushing to make network stable in 4.19 and 4.14.
You are workin on phylink too?
I’m not working on it in a typical meaning of the term “to work”. I’m following the work that is happening around porting mtk_eth_soc to phylink (both on mediatek mailing list and here on this forum, and in your kernel repo too) and from time to time try to build a kernel with new version of phylink patches and test it.
P.S. Received two new R2 boards today, arrived surprisingly fast - took only 8 days in transit.
Have not seen phylink patches in mtk mailinglist…maybe netdev (i’m not watching this)?
It is probably me being wrong here - I was under impression that I’ve seen approx the same phylink patches as you’ve got in your git repo somewhere else and as the only other place I’ve been visiting regarding mtk_etc_soc are mediatek kernel development mailing lists I thought that origin of the phylink patches is there. If that’s not the case than it’s my error/imagination, sorry for that.
Imho only rene’s repo (https://github.com/vDorst/linux-1) has same Patches…
Have you done some further tests? Any problems with 4.14/4.19?
I’ve got three R2 boards and I’m working with a number of your Kernel versions Frank - I’ll do some testing with the discussed setup this weekend - sorry I keep on disappearing! I know we need all hands on deck but I’ve had health issues - all resolved now
I was recently digging through OpenWRT kernel patches for BPiR2 and stumbled upon this one:
Interesting lines are these:
+ if (!dsa_is_cpu_port(ds, 5)) {
+ val |= MHWTRAP_P5_DIS;
+ val |= MHWTRAP_P5_MAC_SEL;
+ val |= MHWTRAP_P5_RGMII_MODE;
+ }
What seems to be done here is the initialization of the port 5 of the switch to use RGMII mode no matter what is set in the DTS, but only in case this port was marked as a cpu port in DTS.
Right…port5 (eth1=wan) is rgmi only, i don’t know why it is only working if set to trgmii. Port6=eth0=lan supports trgmii. Imho trgmii was not setup the right way so it causes issues if adding the second gmac with another mode.