Adding second gmac to 4.14


(Frank W.) #12

@moore can you please confirm that you also have no traffic on cpu_port0 and working on a fix?


(Frank W.) #13

@garywang can you take a look at this problem?


(gary) #14

Hi Frank

I’m not sure why the cpu_port0(eth0) doesn’t work, but I think we can check the statisitics of switch port to make sure if traffic isn’t sent out over lan0. If the traffic is not sent out from lan0, that means the problem is caused by mtk eth driver.

And I think we have to ensure the switch tag is enabled on switch side, and each package made by cpu must have the field of switch tag.

Also Ryder said the patch of hardware NAT is ready for reviewing, so can you please wait that version release.


(Frank W.) #15

I have currently the gmac-kernel not runnimg because i’m testing some other things in main.

I also assume that the switch-tag is not set for eth0/cpu-port0. How can i see this in code (wgere should it be set)?

for the most people 2nd cpu-port and hdmi is more important than hnat

Ryder said that patches ae for 4.16…i had tried to port wifi-driver to 4.16, but hang at timer-functions where api has changed


(gary) #16

Hi Frank

I just guess this problem is related to switch-tag, but we have to check the statistics of all switch ports, and debug the code to see what problem happens.


(Frank W.) #17

ok, i try to get the statistics for lan0 on my second R2 with “ethertool -i lan0” while ping my router over it (must revert my routing back to cpu_port0 first)

Linux bpi-r2 4.14.19-bpi-r2-gmac #132 SMP Mon Feb 26 16:23:26 CET 2018 armv7l
[16:42] root@bpi-r2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group defau0
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qle0
    link/ether c6:86:c6:05:22:e7 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qle0
    link/ether ba:6b:99:cd:5e:20 brd ff:ff:ff:ff:ff:ff
4: wan@eth1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group 0
    link/ether ba:6b:99:cd:5e:20 brd ff:ff:ff:ff:ff:ff
5: lan0@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group0
    link/ether c6:86:c6:05:22:e7 brd ff:ff:ff:ff:ff:ff
6: lan1@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group0
    link/ether c6:86:c6:05:22:e7 brd ff:ff:ff:ff:ff:ff
7: lan2@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group0
    link/ether c6:86:c6:05:22:e7 brd ff:ff:ff:ff:ff:ff
8: lan3@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group0
    link/ether c6:86:c6:05:22:e7 brd ff:ff:ff:ff:ff:ff
[16:42] root@bpi-r2:~# ethtool -i lan0
driver: dsa
version: 
firmware-version: N/A
bus-info: platform
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
[16:42] root@bpi-r2:~# ethtool -i wan
driver: dsa
version: 
firmware-version: N/A
bus-info: platform
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

statistics:

[16:43] root@bpi-r2:~# ethtool -S eth0
NIC statistics:
     tx_bytes: 0
     tx_packets: 0
     tx_skip: 0
     tx_collisions: 0
     rx_bytes: 0
     rx_packets: 0
     rx_overflow: 0
     rx_fcs_errors: 0
     rx_short_errors: 0
     rx_long_errors: 0
     rx_checksum_errors: 0
     rx_flow_control_packets: 0

[16:44] root@bpi-r2:~# ethtool -S eth1
NIC statistics:
     tx_bytes: 0
     tx_packets: 0
     tx_skip: 0
     tx_collisions: 0
     rx_bytes: 0
     rx_packets: 0
     rx_overflow: 0
     rx_fcs_errors: 0
     rx_short_errors: 0
     rx_long_errors: 0
     rx_checksum_errors: 0
     rx_flow_control_packets: 0
     p05_TxDrop: 0
     p05_TxCrcErr: 0
     p05_TxUnicast: 0
     p05_TxMulticast: 0
     p05_TxBroadcast: 0
     p05_TxCollision: 0
     p05_TxSingleCollision: 0
     p05_TxMultipleCollision: 0
     p05_TxDeferred: 0
     p05_TxLateCollision: 0
     p05_TxExcessiveCollistion: 0
     p05_TxPause: 0
     p05_TxPktSz64: 0
     p05_TxPktSz65To127: 0
     p05_TxPktSz128To255: 0
     p05_TxPktSz256To511: 0
     p05_TxPktSz512To1023: 0
     p05_Tx1024ToMax: 0
     p05_TxBytes: 0
     p05_RxDrop: 0
     p05_RxFiltering: 0
     p05_RxMulticast: 0
     p05_RxBroadcast: 0
     p05_RxAlignErr: 0
     p05_RxCrcErr: 0
     p05_RxUnderSizeErr: 0
     p05_RxFragErr: 0
     p05_RxOverSzErr: 0
     p05_RxJabberErr: 0
     p05_RxPause: 0
     p05_RxPktSz64: 0
     p05_RxPktSz65To127: 0
     p05_RxPktSz128To255: 0
     p05_RxPktSz256To511: 0
     p05_RxPktSz512To1023: 0
     p05_RxPktSz1024ToMax: 0
     p05_RxBytes: 0
     p05_RxCtrlDrop: 0
     p05_RxIngressDrop: 0
     p05_RxArlDrop: 0

as you see statistics for eth1 (cpu-port1) is more detailed than eth0 (cpu-port0). in eth0 there are p05-sets (port 5 of mt7530??), these entries are missing on eth0 (maybe port 6)

now to lan0:

[16:50] root@bpi-r2:~# ifup lan0
[16:50] root@bpi-r2:~# ifconfig lan0
lan0      Link encap:Ethernet  HWaddr 08:00:00:00:00:00  
          inet addr:192.168.0.10  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::c486:c6ff:fe05:22e7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:276 errors:0 dropped:4 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:45834 (44.7 KiB)  TX bytes:826 (826.0 B)

[16:50] root@bpi-r2:~# ping 192.168.0.5
PING 192.168.0.5 (192.168.0.5) 56(84) bytes of data.
^C
--- 192.168.0.5 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6235ms

[16:50] root@bpi-r2:~# ethtool -S lan0
NIC statistics:
     tx_packets: 36
     tx_bytes: 3216
     rx_packets: 356
     rx_bytes: 54746
     TxDrop: 0
     TxCrcErr: 0
     TxUnicast: 28
     TxMulticast: 14
     TxBroadcast: 1
     TxCollision: 0
     TxSingleCollision: 0
     TxMultipleCollision: 0
     TxDeferred: 0
     TxLateCollision: 0
     TxExcessiveCollistion: 0
     TxPause: 0
     TxPktSz64: 2
     TxPktSz65To127: 37
     TxPktSz128To255: 4
     TxPktSz256To511: 0
     TxPktSz512To1023: 0
     Tx1024ToMax: 0
     TxBytes: 3957
     RxDrop: 0
     RxFiltering: 0
     RxMulticast: 229
     RxBroadcast: 125
     RxAlignErr: 0
     RxCrcErr: 0
     RxUnderSizeErr: 0
     RxFragErr: 0
     RxOverSzErr: 0
     RxJabberErr: 0
     RxPause: 0
     RxPktSz64: 174
     RxPktSz65To127: 38
     RxPktSz128To255: 72
     RxPktSz256To511: 45
     RxPktSz512To1023: 27
     RxPktSz1024ToMax: 0
     RxBytes: 61154
     RxCtrlDrop: 0
     RxIngressDrop: 0
     RxArlDrop: 0

as i see there are RX and TX bytes…so lan-port itself works, but the internal routing (is assume mt7530 => SOC) does not work

after the pings eth0 has traffic in booth directions but not the same amount

[17:09] root@bpi-r2:~# ethtool -S eth0
NIC statistics:
     tx_bytes: 9258
     tx_packets: 98
     tx_skip: 0
     tx_collisions: 0
     rx_bytes: 307700
     rx_packets: 2036
     rx_overflow: 0
     rx_fcs_errors: 0
     rx_short_errors: 0
     rx_long_errors: 0
     rx_checksum_errors: 0
     rx_flow_control_packets: 0

where does ethtool measure that traffic on eth0 (on soc-side or switch-side)?

some thinkings: soc->switch works and the backway not (packets from switch are not accepted by soc)…ok…did some extra-tests

ping again my router (192.168.0.5):

[root@bananapi ~]# tcpdump -nni eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:13:34.104680 IP truncated-ip - 6 bytes missing! 192.168.0.10 > 192.168.0.5: ICMP echo request, id 815, seq 1, length 64
17:13:35.150095 IP truncated-ip - 6 bytes missing! 192.168.0.10 > 192.168.0.5: ICMP echo request, id 815, seq 2, length 64
17:13:36.190053 IP truncated-ip - 6 bytes missing! 192.168.0.10 > 192.168.0.5: ICMP echo request, id 815, seq 3, length 64
17:13:37.230044 IP truncated-ip - 6 bytes missing! 192.168.0.10 > 192.168.0.5: ICMP echo request, id 815, seq 4, length 64
17:13:38.270038 IP truncated-ip - 6 bytes missing! 192.168.0.10 > 192.168.0.5: ICMP echo request, id 815, seq 5, length 64
17:13:39.310056 IP truncated-ip - 6 bytes missing! 192.168.0.10 > 192.168.0.5: ICMP echo request, id 815, seq 6, length 64
17:13:40.350044 IP truncated-ip - 6 bytes missing! 192.168.0.10 > 192.168.0.5: ICMP echo request, id 815, seq 7, length 64
17:13:41.390038 IP truncated-ip - 6 bytes missing! 192.168.0.10 > 192.168.0.5: ICMP echo request, id 815, seq 8, length 64
17:13:42.430047 IP truncated-ip - 6 bytes missing! 192.168.0.10 > 192.168.0.5: ICMP echo request, id 815, seq 9, length 64

packet goes out, as assumed, and of course will be answered by my (old) router. for same on r2 i need to install tcpdump (have to temporary boot with working kernel, install and again with gmac-kernel, have done this with ethtool before).

strange is the “truncated-ip”-information…

because i cannot use ssh (no connection to lan) i tcpdumped to file

[17:29] root@bpi-r2:~# tcpdump -nni lan0 icmp > lan0ping.log &
[17:30] root@bpi-r2:~# tcpdump -nni eth0 icmp > eth0ping.log &

pinged again, see incoming pings on my (old) router, but booth files on R2 stay empty :thinking: after killing tcpdump the lan0-log contains the outgoing pings:

[17:34] root@bpi-r2:~# tailf lan0ping.log 
17:31:34.745963 IP 192.168.0.10 > 192.168.0.5: ICMP echo request, id 791, seq 4,
 length 64
17:31:35.785962 IP 192.168.0.10 > 192.168.0.5: ICMP echo request, id 791, seq 5,
 length 64
17:31:36.825984 IP 192.168.0.10 > 192.168.0.5: ICMP echo request, id 791, seq 6,
 length 64
17:31:37.865970 IP 192.168.0.10 > 192.168.0.5: ICMP echo request, id 791, seq 7,
 length 64
17:31:38.905964 IP 192.168.0.10 > 192.168.0.5: ICMP echo request, id 791, seq 8,
 length 64
17:31:39.945963 IP 192.168.0.10 > 192.168.0.5: ICMP echo request, id 791, seq 9,
 length 64
17:31:40.985961 IP 192.168.0.10 > 192.168.0.5: ICMP echo request, id 791, seq 10
, length 64
17:31:42.025960 IP 192.168.0.10 > 192.168.0.5: ICMP echo request, id 791, seq 11
, length 64
17:31:43.065960 IP 192.168.0.10 > 192.168.0.5: ICMP echo request, id 791, seq 12
, length 64

BPI-R2 OpenWrt(LEDE) Souce code : 2018-04-11
Strange packet loss
(dorabmon) #18

Hardware NAT should be still working on current DSA configuration with single GMAC.

I think Software HNAT can work. and also Hardware NAT has to work!

You could add HWNAT on the top of current DSA configuration at first if you just want to have a verification for Hardware NAT.

Otherwise, you will meet tons of frustration during you’re integrating second GMAC into existing DSA framework.

This is still hard work for general people in specifically you don’t have deep thoughts on the DSA core.


(Frank W.) #19

Please don’t mix hnat and second gmac…this thread is for second gmac. hnat have separate thread where no traffic on wan is reported if using patched mtk_eth_soc.c


(dorabmon) #20

Yes, I knew and agreed it’s good to split these topics into two threads. So I just replied my thoughts to above message.


(Frank W.) #21

Actual i had only applied gmac-patches from lede made for 4.14 without modify anything,just testing. I had thought that they should work,but they did not so i do some debug see here:

@garywang is this information enough or do you need additional infos?


(Frank W.) #22

tested tcpdump on my router again with a working ping:

[root@bananapi ~]# tcpdump -nni eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:02:16.781003 IP 192.168.0.21 > 192.168.0.5: ICMP echo request, id 20647, seq 1, length 64
18:02:16.781234 IP 192.168.0.5 > 192.168.0.21: ICMP echo reply, id 20647, seq 1, length 64
18:02:17.781727 IP 192.168.0.21 > 192.168.0.5: ICMP echo request, id 20647, seq 2, length 64
18:02:17.781913 IP 192.168.0.5 > 192.168.0.21: ICMP echo reply, id 20647, seq 2, length 64
18:02:18.781675 IP 192.168.0.21 > 192.168.0.5: ICMP echo request, id 20647, seq 3, length 64
18:02:18.781869 IP 192.168.0.5 > 192.168.0.21: ICMP echo reply, id 20647, seq 3, length 64
18:02:19.781757 IP 192.168.0.21 > 192.168.0.5: ICMP echo request, id 20647, seq 4, length 64
18:02:19.781967 IP 192.168.0.5 > 192.168.0.21: ICMP echo reply, id 20647, seq 4, length 64
18:02:20.781712 IP 192.168.0.21 > 192.168.0.5: ICMP echo request, id 20647, seq 5, length 64
18:02:20.781912 IP 192.168.0.5 > 192.168.0.21: ICMP echo reply, id 20647, seq 5, length 64

as you can see with same command now there are outgoing icmp echo reply’s…so i assume that the problem is that the outgoing packets from gmac-kernel are getting damaged (truncated-ip - 6 bytes missing!) so my current router does not send a reply…

contacted John crispin (author from the patches on lede), hoping for a reply

added task to lede-project: https://bugs.lede-project.org/index.php?do=details&task_id=1403


(Frank W.) #23

No answer till now…has anybody contact to lede-team or john crispin and give him a hint for that?

@garywang @moore


(Paweł Kalemba) #24

Maybe you can try his email? Its in commits. Here


(Frank W.) #25

i have contacted john via this email…also the lede-bug-report is not confirmed yet


(fkpwolf) #26

Thanks for the patch. I am curious: why add second gmac? If MT7632 connect switch with 2 RGMII, the bandwidth will be doubled? How to test this kind of performance boost?


(Frank W.) #27

i will use that for LAN/WAN-separation…if wan is connected to one and LAN to the other you can test it with traffic wan-lan :slight_smile:


(xbgmsharp) #28

To confirm my understand on kernel 4.14 to enable native/hardware switch/bridge we need 2gmac as available in hardware. From what i can see in the forum it is not functional yet. How can i test it?


(Frank W.) #29

if the lan-port-separation is disabled anyhow you need a lan/wan-separation, so you need the 2nd gmac between switch (mt7530) and cpu (mt7623)…

the gmacs are recognized in debian based systems as ethx…currently you have only eth0…with the lede-patches you’ll see also eth1, but here eth0 will not work because outgoing frames are damaged

if the second gmac works it may be possible to bridge the lan-ports in the switch, not via software in CPU.

currently you can bridge the ports with bridge-utils, but imho this is done in CPU. so if Traffic goes from LAN0 to LAN1 (assuming ports are bridged via br0) this way:

LAN0 => SWITCH => ETH0 => CPU (br0) => ETH0 => SWITCH => LAN1

this means, if you you have full bi-directional traffic on these ports you will get only the half speed

@Ryder.Lee @garywang @moore @linkerosa please correct me if i’m wrong…and can you please contact john to look at the lede-4.14-patches for 2nd gmac


(xbgmsharp) #30

Thanks for the clarification, therefor i should not use the DSA code and ensure the 2Gmac to get full speed.


(Wolfgang) #31

Regarding speed, it should be noted that GMAC1 supports ‘Turbo-RGMII’ mode (trgmii), running at 2.5 Gbps instead of the usual 1Gbps. In theory this should leave enough bandwidth for two bidirectional gig-e links running simultaneously over a single GMAC.