Hi all!
At the beginning of this post, I would like to thank everybody in this forum and especially @frank-w for all the extraordinary work in order to improve the software support of R64 and R2.
Initially I wanted to ask for help, since the routing performance of my BPI-R64 was quite bad, but recently I found that this is a thermal issue (CPU cooling is definitely needed) and therefore I would like to only share my test results and discuss some possible optimization here.
I general I would like to use the BPI-R64 as router/firewall in my home network. The following things are important for me:
- high network throughput “LAN to LAN” and decent throughput “LAN to WAN”
- iptables support
In order to verify my requirements, I did some network performance tests with iperf3 and scp. Since I’m only interested in routing scenarios, I connected 3 fairly good machines to WAN, LAN0 and LAN1 in order to test the performance (MASQUERADE/NAT is not needed).
I used one of the latest kernels from @frank-w and the user land from the official Ubuntu 18.04. (with the kernel from the official BPI release 2020-05-08, the performance was definitely worse)
build.conf (excerpt)
...
#board=bpi-r2
board=bpi-r64
#mainline uboot for r64 needs 64bit uImage
#uimagearch=arm64
Kernel
[root@bpi-r64 ~]# uname -a
Linux bpi-r64 5.4.77-bpi-r64-main #2 SMP PREEMPT Sat Nov 14 19:35:18 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
Network config is quite simple:
auto eth0
iface eth0 inet manual
auto eth1
iface eth1 inet manual
auto lan0
iface lan0 inet static
address 10.2.0.2
netmask 255.255.255.252
post-up echo 1 > /sys/class/net/lan0/queues/rx-0/rps_cpus
auto lan1
iface lan1 inet static
address 10.2.0.6
netmask 255.255.255.252
post-up echo 1 > /sys/class/net/lan1/queues/rx-0/rps_cpus
results LAN to LAN - before I found the thermal issue
date; iperf3 -n 5G -i 2 -c 10.2.0.5
So 15 Nov 2020 15:49:44 CET
Connecting to host 10.2.0.5, port 5201
[ 5] local 10.2.0.1 port 54348 connected to 10.2.0.5 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-2.00 sec 200 MBytes 837 Mbits/sec 0 1.04 MBytes
[ 5] 2.00-4.00 sec 180 MBytes 755 Mbits/sec 0 1.04 MBytes
[ 5] 4.00-6.00 sec 174 MBytes 729 Mbits/sec 0 1.04 MBytes
[ 5] 6.00-8.00 sec 151 MBytes 634 Mbits/sec 0 1.04 MBytes
[ 5] 8.00-10.00 sec 121 MBytes 509 Mbits/sec 0 1.04 MBytes
[ 5] 10.00-12.00 sec 101 MBytes 425 Mbits/sec 0 1.04 MBytes
[ 5] 12.00-14.00 sec 76.2 MBytes 320 Mbits/sec 0 1.04 MBytes
[ 5] 14.00-16.00 sec 75.0 MBytes 315 Mbits/sec 0 1.04 MBytes
[ 5] 16.00-18.00 sec 56.2 MBytes 236 Mbits/sec 0 1.04 MBytes
[ 5] 18.00-20.00 sec 18.8 MBytes 78.6 Mbits/sec 0 1.04 MBytes
[ 5] 20.00-22.00 sec 47.5 MBytes 199 Mbits/sec 0 1.04 MBytes
[ 5] 22.00-24.00 sec 33.8 MBytes 142 Mbits/sec 0 1.04 MBytes
[ 5] 24.00-26.00 sec 45.0 MBytes 189 Mbits/sec 0 1.04 MBytes
[ 5] 26.00-28.00 sec 35.0 MBytes 147 Mbits/sec 0 1.04 MBytes
The whole time, I thought it is not a thermal issue, since the CPU temperature is not really increasing during the test, but now I’m quite sure the this is not measuring the CPU temperature.
[root@bpi-r64 ~]# date; while true; do cat /sys/class/thermal/thermal_zone0/temp; sleep 2; done;
Sun Nov 15 15:49:43 CET 2020
46300
49500
49500
49700
49400
48700
49000
48300
47900
47600
47100
47500
46800
47400
And the issue can be easily found by checking the actual CPU frequency:
[root@bpi-r64 ~]# while true; do cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq; sleep 2; done;
1262500
1262500
1137500
1025000
1025000
812500
600000
437500
437500
30000
30000
437500
437500
437500
437500
437500
As you can see, the CPU frequency sometimes drops to even 30MHz.
results LAN to LAN - with professional cooling device (my fingers)
date; iperf3 -n 5G -i 2 -c 10.2.0.5
So 15 Nov 2020 15:58:49 CET
Connecting to host 10.2.0.5, port 5201
[ 5] local 10.2.0.1 port 54362 connected to 10.2.0.5 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-2.00 sec 200 MBytes 838 Mbits/sec 0 1.04 MBytes
[ 5] 2.00-4.00 sec 208 MBytes 870 Mbits/sec 0 1.04 MBytes
[ 5] 4.00-6.00 sec 206 MBytes 865 Mbits/sec 0 1.04 MBytes
[ 5] 6.00-8.00 sec 205 MBytes 860 Mbits/sec 0 1.04 MBytes
[ 5] 8.00-10.00 sec 211 MBytes 886 Mbits/sec 0 1.04 MBytes
[ 5] 10.00-12.00 sec 191 MBytes 802 Mbits/sec 0 1.04 MBytes
[ 5] 12.00-14.00 sec 206 MBytes 865 Mbits/sec 0 1.04 MBytes
[ 5] 14.00-16.00 sec 212 MBytes 891 Mbits/sec 0 1.04 MBytes
[ 5] 16.00-18.00 sec 212 MBytes 891 Mbits/sec 0 1.04 MBytes
[ 5] 18.00-20.00 sec 212 MBytes 891 Mbits/sec 0 1.04 MBytes
[ 5] 20.00-22.00 sec 212 MBytes 891 Mbits/sec 0 1.04 MBytes
[ 5] 22.00-24.00 sec 212 MBytes 891 Mbits/sec 0 1.04 MBytes
[ 5] 24.00-26.00 sec 211 MBytes 886 Mbits/sec 0 1.04 MBytes
[ 5] 26.00-28.00 sec 204 MBytes 855 Mbits/sec 0 1.04 MBytes
[ 5] 28.00-30.00 sec 204 MBytes 855 Mbits/sec 0 1.04 MBytes
[ 5] 30.00-32.00 sec 195 MBytes 818 Mbits/sec 0 1.04 MBytes
[ 5] 32.00-34.00 sec 196 MBytes 823 Mbits/sec 0 1.04 MBytes
[ 5] 34.00-36.00 sec 205 MBytes 860 Mbits/sec 0 1.04 MBytes
[ 5] 36.00-38.00 sec 206 MBytes 865 Mbits/sec 0 1.04 MBytes
[ 5] 38.00-40.00 sec 198 MBytes 828 Mbits/sec 0 1.04 MBytes
[ 5] 40.00-42.00 sec 210 MBytes 881 Mbits/sec 0 1.04 MBytes
[ 5] 42.00-44.00 sec 210 MBytes 881 Mbits/sec 0 1.04 MBytes
[ 5] 44.00-46.00 sec 212 MBytes 891 Mbits/sec 0 1.04 MBytes
[ 5] 46.00-48.00 sec 212 MBytes 891 Mbits/sec 0 1.04 MBytes
[ 5] 48.00-49.57 sec 168 MBytes 892 Mbits/sec 0 1.04 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-49.57 sec 5.00 GBytes 867 Mbits/sec 0 sender
[ 5] 0.00-49.57 sec 5.00 GBytes 866 Mbits/sec receiver
iperf Done.
So all-in-all I’m quite happy with the performance and I will do further test with iptables and maybe NAT next week.
I also used the following optimization from bpi-r2-improved-network-performance-by-rps since it increased the performance a bit, even with the thermal issue (i.e. when I didn’t cool the CPU).
echo 1 > /sys/class/net/lan0/queues/rx-0/rps_cpus
echo 1 > /sys/class/net/lan1/queues/rx-0/rps_cpus
Further optimizations
I have seen that the second GMAC (eth1) is available, but seems not to be used (WAN and LAN0-4 interfaces use eth0). Did anybody try this for R64 already (I have seen work for R2 only). Is this because of my /etc/network/interfaces config or does it need to be changes in e.g. mt7622-bananapi-bpi-r64.dts?
BR Michael