Issues with NAT for LAN on 4.14.80 kernel

singinwhale · December 19, 2018, 6:30pm

Hey there,

I am running frank-w’s prebuilt deb_stretch_4.14.80_SD on my BPI-R2 and wanted to setup LAN -> WAN NAT for lan0-lan3 and also ap0 so I have my LAN and Wifi on the same subnet. Funny thing is that the NAT only works for ap0 but not for lan0-lan3 even though they are on the same bridge br0. Even when I remove the ap0 from the bridge it does not work.

when looking at the trafffic with TCPdump everything seems fine. Pings are answered fine and tcptraffic is exchanged as well, however when I open a page with firefox it never gets past the TLS-Handshake-Stage. When i use the same computer on the Wifi firefox loads the pages as it is supposed to.

Any ideas?

/etc/network/interfaces:

auto eth0
iface eth0 inet manual
  pre-up ip link set $IFACE up
  post-down ip link set $IFACE down

#2nd gmac (cpu-port) only in 4.14.53+
auto eth1
iface eth1 inet manual
  pre-up ip link set $IFACE up
  post-down ip link set $IFACE down

#wan-port as dhcp-client
allow-hotplug wan
iface wan inet dhcp

auto lan0
iface lan0 inet manual
auto lan1
iface lan1 inet manual
auto lan2
iface lan2 inet manual
auto lan3
iface lan3 inet manual

auto br0
iface br0 inet static
    address 192.168.0.1
    netmask 255.255.0.0
    bridge_ports lan0 lan1 lan2 lan3
    bridge_fd 5
    bridge_stp no



source-directory /etc/network/interfaces.d

*ip addr:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 7a:0a:45:32:3b:1a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::780a:45ff:fe32:3b1a/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether ba:e8:c7:15:84:38 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b8e8:c7ff:fe15:8438/64 scope link
       valid_lft forever preferred_lft forever
4: wan@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ba:e8:c7:15:84:38 brd ff:ff:ff:ff:ff:ff
    inet 10.10.61.54/20 brd 10.10.63.255 scope global wan
       valid_lft forever preferred_lft forever
    inet6 fe80::b8e8:c7ff:fe15:8438/64 scope link
       valid_lft forever preferred_lft forever
5: lan0@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN group default qlen 1000
    link/ether 7a:0a:45:32:3b:1a brd ff:ff:ff:ff:ff:ff
6: lan1@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN group default qlen 1000
    link/ether 7a:0a:45:32:3b:1a brd ff:ff:ff:ff:ff:ff
7: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN group default qlen 1000
    link/ether 7a:0a:45:32:3b:1a brd ff:ff:ff:ff:ff:ff
8: lan3@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN group default qlen 1000
    link/ether 7a:0a:45:32:3b:1a brd ff:ff:ff:ff:ff:ff
9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:08:22:96:b6:fe brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/16 brd 192.168.255.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::8:22ff:fe96:b6fe/64 scope link
       valid_lft forever preferred_lft forever
10: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:08:22:96:b6:fe brd ff:ff:ff:ff:ff:ff
11: ap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether 02:08:22:96:b6:fe brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8:22ff:fe96:b6fe/64 scope link
       valid_lft forever preferred_lft forever

iptables -vnL && iptables -t nat -vnL:

Chain INPUT (policy ACCEPT 1028 packets, 105K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
13525 1911K ACCEPT     all  --  br0    wan     0.0.0.0/0            0.0.0.0/0
19798   14M ACCEPT     all  --  wan    br0     0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 953 packets, 112K bytes)
 pkts bytes target     prot opt in     out     source               destination
Chain PREROUTING (policy ACCEPT 3212 packets, 139K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 343 packets, 32359 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 272 packets, 19422 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 4 packets, 1336 bytes)
 pkts bytes target     prot opt in     out     source               destination
 2971  107K MASQUERADE  all  --  *      wan     0.0.0.0/0            0.0.0.0/0

frank-w · December 19, 2018, 8:32pm

Iptables look good so far. I don’t use bridge here in my setup.

Can you try remove a lan-port out of the bridge and test with this?

singinwhale · December 20, 2018, 2:19pm

Tried that now. That does not change anything unfortunately. I liked that the 4 lan ports were a single interface in my previous 4.4 install. Is there a way to use eth0 or eth1 (whichever is the 4 ports)? Maybe that helps the kernel speed things up.

frank-w · December 20, 2018, 2:28pm

imho this is not possible, if you drop dsa-driver you have no defined connection between eth0/1 and wan/lan. i don’t know how exactly it’s done in 4.4…i guess there is a hardcoded connection

you use dhcp on your bridge? did all client get valid config? maybe here is the problem

you can also run tcpdump on lanx+wan to see whats happening here

i expect you have routing activated for all interfaces…

echo 1 > /proc/sys/net/ipv4/ip_forward

there are also interface-based entries possible…

you can ping lanx from your client? can you also ping wan-address from it?

hmm…btw. you need “auto wan” before “allow-hotplug wan” to bring it up on bootup…

and please comment out lines for lan0-lan3 if using a bridge

and make a clean reboot after it

i hope it fixes that strange LOWERLAYERDOWN on lanx while br0 is up (or is there nothing physically connected?)

singinwhale · December 20, 2018, 5:32pm

Yeah lan1-3 are just not connected that’s why it says LOWERLAYERDOWN.

DHCP is on for br0 but I used a manual setup for lan0 on both ends. Ping works to the gateway and also to 1.1.1.1. The ping rtt is about 10ms higher than via ap0 which is also strange. The pi is not doing anything else right now besides being a router so there shouldn’t be any cpu bottlenecks.

frank-w · December 20, 2018, 6:03pm

lan0 is also LOWERLAYERDOWN in your output…

have you tried removing lan0-lan3-config and adding “auto wan”?

so your interfaces should look like this:

auto eth0
iface eth0 inet manual
  pre-up ip link set $IFACE up
  post-down ip link set $IFACE down

#2nd gmac (cpu-port) only in 4.14.53+
auto eth1
iface eth1 inet manual
  pre-up ip link set $IFACE up
  post-down ip link set $IFACE down

#wan-port as dhcp-client
auto wan
allow-hotplug wan
iface wan inet dhcp

auto br0
iface br0 inet static
    address 192.168.0.1
    netmask 255.255.0.0
    bridge_ports lan0 lan1 lan2 lan3
    bridge_fd 5
    bridge_stp no

source-directory /etc/network/interfaces.d

what is IP 1.1.1.1? i mean ping from your client (connected on lan0)

br0: 192.168.0.1
wan: 10.10.61.54

singinwhale · December 20, 2018, 11:47pm

1.1.1.1 is just the Cloudflare DNS server. Similar to Google’s 8.8.8.8. I always use it for connectivity checks. Also wan is not brought up automatically when I add allow-hotplug wan just auto wan is enough. Removing the lan0-lan3 manual block didn’t change anything either.

In my log lan0 also says LOWERLEVELDOWN because it was not plugged in at the point I retrieved the output as I was connected via ap0, because the lans don’t work. Even SSH is affected as it seems. When I connect to the pi via SSH, it stops replying whenever I send a command that produces more than a couple lines of output (’ip a’ for example).

It feels like the lan ports are just not up to speed at all. It could be the same problem that @troumad has here regarding the packetloss.

singinwhale · December 21, 2018, 2:04am

I did some more testing just now and now I set up my interfaces differently:

I removed ap0 from the bridge
ap0 is assigned it its own subnet on 192.168.2.1/24
ap0 has its own dhcp range as well

I did a tcpdump on lan0 again and noticed when I ctrl-c’ed that it said this:

^C
1372 packets captured
1701 packets received by filter
329 packets dropped by kernel
1 packet dropped by interface

looks quite suspicious to me tbh. Why would the kernel drop any packets though?

frank-w · December 21, 2018, 5:39am

Can you look if hwqos/hwnat is enabled on system and try to disable?

singinwhale · December 21, 2018, 11:58am

Tried with and without the hwnat module. No difference in behavior

frank-w · December 21, 2018, 12:04pm

if you can recompile kernel and install it you can drop HWQOS, i have got a message that this causes some problems, but i can’t reproduce them

run “./build.sh defconfig” and comment out this line (at the end) with #

CONFIG_NET_MEDIATEK_HW_QOS=m

then importconfig and build

else i guess we need help from an expert. imho hwqos is part of hwnat and should not be active if hwnat is dropped

edit: found modifications in base-ethernet-driver, so it can break traffic also if hwnat is unloaded

./drivers/net/ethernet/mediatek/mtk_eth_soc.c:27:#if defined(CONFIG_NET_MEDIATEK_HW_QOS)
./drivers/net/ethernet/mediatek/mtk_eth_soc.c:739:#ifdef CONFIG_NET_MEDIATEK_HW_QOS
./drivers/net/ethernet/mediatek/mtk_eth_soc.c:2136:#if defined(CONFIG_NET_MEDIATEK_HW_QOS)
./drivers/net/ethernet/mediatek/mtk_eth_soc.c:2143:#if defined(CONFIG_NET_MEDIATEK_HW_QOS)

singinwhale · December 21, 2018, 12:29pm

boy, the kernel was always one of those things I always avoided to touch because I don’t have a clue what I’m doing there. But I guess I will have to give it a shot. While I’m at it: should I use your 4.19 kernel or just use the 4.14 kernel again?

frank-w · December 21, 2018, 12:30pm

try 4.19 first, hwnat/hwqos is not supported there and maybe bug is fixed

singinwhale · December 21, 2018, 12:43pm

Maybe a dumb question but is it enough to just install the debian-package from your github releases page?

frank-w · December 21, 2018, 12:44pm

you can try it…have not tested this much

singinwhale · December 21, 2018, 2:53pm

I have loaded the new 4.19.11 kernel now but the issue remains.

For any one interested: I installed @frank-w’s DEB-Package for Kernel 4.19.11-testing from his Github and then changed the /boot/bananapi/bpi-r2/linux/uEnv.txt so it loads the new uImage_4.19.11-main file by changing the boot1 entry which is loaded by default.

Is there a build of the 4.4 kernel with working wifi AP with WPA2 somewhere? Then I would roll back to that one for now.

frank-w · December 21, 2018, 2:55pm

you find compiled 4.4 on my gdrive or compile it from my repo

https://drive.google.com/open?id=1kkFyxeHoskszI7CpSsL6Wi6ROjDrBLOB

note that you need to assign ip-adress to eth0 instead of lanx

Btw boot1 should use var kernel which is easier to change than boot1.imho deb should do it automaticly…

Have you same strange packetdrops with 4.19 in tcpdump?

mhm…lan-ports need to be set to up after bridge is created…before there is no traffic and the port+bridge are down. make now some ping-tests to br0-ip…or do you have only problem with forwarded traffic?

singinwhale · December 21, 2018, 3:44pm

true that makes more sense will change it back later.

no its not just the forwarded traffic that has these issues. direct pings over the wire have the same issues regarding packet loss. about 10% of pings to the BPI get lost. I don’t think it’s an issues with my network setup. The interfaces are in their right states when they have to be, no issues there.

I downgraded to 4.4 now and have a look there. hostapd does its thing nicely but iptables complains:

 modprobe: ERROR: ../libkmod/libkmod-module.c:192 kmod_module_parse_depline() ctx=0x441008 path=/lib/modules/4.4.160-BPI-R2-Kernel/kernel/net/ipv4/netfilter/ip_tables.ko error=No such file or directory

is it not compiled in your build?

frank-w · December 21, 2018, 3:49pm

--- 192.168.0.18 ping statistics ---
1001 packets transmitted, 1001 received, 0% packet loss, time 1023906ms
rtt min/avg/max/mdev = 0.285/0.449/0.671/0.060 ms

root@bpi-r2:~# tcpdump -i br0 >/dev/null
[  347.217490] device br0 entered promiscuous mode
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C2649 packets cap[ 1317.278048] device br0 left promiscuous mode
tured
2650 packets received by filter
0 packets dropped by kernel
root@bpi-r2:~# uname -r
4.19.11-bpi-r2-main
root@bpi-r2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 32:c1:83:3a:48:5a brd ff:ff:ff:ff:ff:ff
3: wan@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
    link/ether 32:c1:83:3a:48:5a brd ff:ff:ff:ff:ff:ff
4: lan0@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 32:c1:83:3a:48:5a brd ff:ff:ff:ff:ff:ff
5: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 32:c1:83:3a:48:5a brd ff:ff:ff:ff:ff:ff
6: lan2@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master br0 state DOWN group default qlen 1000
    link/ether 32:c1:83:3a:48:5a brd ff:ff:ff:ff:ff:ff
7: lan3@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 32:c1:83:3a:48:5a brd ff:ff:ff:ff:ff:ff
8: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 32:c1:83:3a:48:5a brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.18/24 brd 192.168.0.255 scope global br0
       valid_lft forever preferred_lft forever
9: wan.60@wan: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
    link/ether 32:c1:83:3a:48:5a brd ff:ff:ff:ff:ff:ff
    inet 192.168.60.1/24 brd 192.168.60.255 scope global wan.60
       valid_lft forever preferred_lft forever
10: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 32:b6:a2:b4:7e:cc brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.1/24 brd 10.0.3.255 scope global lxcbr0
       valid_lft forever preferred_lft forever

direct pings do not have any issues here

regarding iptables…imho i used the config from official repo…only added things. after getting 4.14 running i do not use 4.4…i only use it to compare on problems

have you unpacked the lib/modules-folder in BPI-ROOT? config contains IPtables as module so it should be included…

currently i’m not getting lan-port up on system-boot…i always need to enable it manually after boot

https://wiki.debian.org/BridgeNetworkConnections says that this line for each port is enough before bridge:

iface eth0 inet manual

so i configured it like this:

iface lan1 inet manual
iface lan2 inet manual

auto br0
iface br0 inet static
    address 192.168.0.18
    netmask 255.255.255.0
    bridge_ports lan1 lan2
    bridge_fd 5
    bridge_stp no

singinwhale · December 21, 2018, 4:17pm

okay I went back to kernel 4.14 now because I couldn’t get 4.4 to work and I want 4.14 or newer anyways. And yes I unpacked both folders to their respective directories.

I checked the ping stats again and I am not getting any kernel dropped packets on the direct route either anymore which is strange.

I have to go now and probably won’t be back for a week or so. Maybe something will come up in the meantime…