New netfilter flow table based HNAT

Banana Pi BPI-R4/BPI-R3 Mini/BPI-R3/BPI-R64/BPI-R2
63

In my case i use bridge(lan0-3+wifi) for lan and wan, but also i use lan0-3 and wan in firewall:

Bridge:

r2-gentoo /dev # brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.6ac4b308d134       no              lan0
                                                        lan1
                                                        lan2
                                                        lan3
                                                        wlp1s0

ip addr:

r2-gentoo /dev # ip a s br0
16: br0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6a:c4:b3:08:d1:34 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.1/24 brd 10.0.1.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::68c4:b3ff:fe08:d134/64 scope link 
       valid_lft forever preferred_lft forever

firewall:

nft list ruleset
....

        flowtable f {
                hook ingress priority filter + 1
                devices = { lan2, lan1, lan0, wan }
                flags offload;
        }
....

All my ethernet devices are connected via lan0-3 which are members of br0.

I suppose ppp and bridge has no offload flags as they are virtual interfaces, and you still need to use physical interfaces in your firewall

P.S. Only in offload flowtable of cource :wink: you still need ppp8 for nat

64

Thanks i try it…after thinking about it,basicly it makes sense as each virtual device is only software and only adds headers to packets (ppp,vlan) and hardware only knows hardware ports…

For ppp i have packets containing

Dot1q-header+ppp-header+payload

And sent out of wan

For bridge there are no additional headers as traffic are only grouped on software side. Hardware does not see it.

btw. where have the flow-table be linked before or after other rules in forward-chain?

chain FORWARD {
    type filter hook forward priority 0; policy drop;
    #ip protocol { tcp, udp } flow add @f
    oifname $ifexternal ip saddr $iprangesblocked jump REJECTED comment "block internal ip ranges to have only internal access"
    #ipv6 in ipv4 tunnel
    udp dport {41,43,44,58,59,60} jump REJECTED comment "block ipv6 in ipv4 tunnel"

    oifname $ifwan ip saddr 192.168.0.9 jump REJECTED comment "Block internet-access for cisco switch"
    oifname $ifwan tcp dport domain jump REJECTED comment "block external dns in forward"

seems at position i have prepared, it breaks forwarding, but imho accept/drop exits the chain and so never reach position of “flow add” defined.

so it looks like a bug in ppp-handling (maybe bug while porting to 5.10)

table ip filter {
    flowtable f {
        hook ingress priority 10
        #devices = $ifoffload;
        #use HW interfaces here!
        devices = { wan, lan0, lan1, lan2, lan3 }
        flags offload
    }

if i look into entries, i see only UNB, many with “new=0.0.0.0:0->0.0.0.0:0 eth=00:00:00:00:00:00->00:00:00:00:00:00 etype=0000 vlan=0,0”, but some (also with public ips) with mac-addresses and the vlan

01ac6 UNB IPv4 5T orig=192.168.0.21:52136->195.20.250.26:443 new=217.61.147.xx:57010->217.72.196.71:443 eth=02:11:02:03:04:05->00:00:5e:00:01:02 etype=0101 vlan=140,0 ib1=1000019d ib2=007ff020

i wonder why target-IP has changed…imho only source needs change to public-ip (IP and vlan 140 are correct)

Imho Ethtype should be 8100/88a8 for vlan or maybe 8863/4 for pppoe and not 101

i’ve found out, that i can install a pppoe-server in ubuntu for testing (apt install pppoe), but this needs some configuration of course

ok, seems only a etype-problem…activated it and made request to https://wiki.ubuntuusers.de/tcpdump/ (see packet with tcpdump on ppp-interface, not wan.140 or wan, but i see pppoe-packets with ip-adress-information, seems like port-filter does not work in this case)

# tcpdump -n -i ppp8 port 443
16:32:01.256873 IP 80.245.76.249.33630 > 213.95.41.4.443: Flags [.], ack 764, win 501, options [nop,nop,TS val 128775260 ecr 153299079], length 0
16:32:01.261032 IP 213.95.41.4.443 > 80.245.76.249.33634: Flags [P.], seq 4984:5298, ack 1985, win 262, options [nop,nop,TS val 153299081 ecr 128775254], length 314

# nslookup 213.95.41.4
4.41.95.213.in-addr.arpa	name = ha.ubuntu-eu.org.

cat /sys/kernel/debug/mtk_ppe/entries | grep BND | grep 140 | grep '213.95.41.4'
00c22 BND IPv4 5T orig=192.168.0.21:33676->213.95.41.4:443 new=80.245.76.249:33676->213.95.41.4:443 eth=02:11:02:03:04:05->00:00:5e:00:01:02 etype=0101 vlan=140,0 ib1=214949a7 ib2=007ff020

mac is the one for wan.140 (wan has another, as i need to set it for both vlans)

if etype in entries is really the ethernet-type it needs to be 8100/88XX (vlan/pppoe). i guess first ethernettype needs to be 8100 for vlan

mhm, on my previous test with afair wan+lan3 i got etype with bits 12/8 set, so again it looks like the ethtype is set to the dsa-port here and not the vlan…but i do not see if all headers are added

did some more tests with a local pppoe-server and it seems that is working with 5.12 and not with 5.10…so i guess i miss anything while porting

it seems flowtable itself is breaking, not the hw-offload (it is broken with disabled flags too, but working if “add flow” line is disabled), but offload with 5.10-hnat works without ppp…very strange

it looks like with 5.10 i have only a mtu problem with flowtable active, seems like flowtable breaks the normal Path-discovery/fragmentation behaviour. without flowtable i can access websites through ppp-tunnel (mtu 1492), with flowtable (also without offload) i got only connection refused. if i reduce mtu, it works with flowtable+offload. idk why i don’t get problems with 5.12 or without flowtable. i will try now mss-fix-settings

https://wiki.nftables.org/wiki-nftables/index.php/Mangling_packet_headers

i got answer about location of rule here: https://marc.info/?l=netfilter&m=162012856832116&w=2

flow add should be last in forward to process all other rules before. forward chain is only done for SYN and SYN-ACK (first 2 tcp packets per connection)

65

Have You succeeded with tests? I’ve tested 5.10 + pppoe, with MSSFIX on the server’s side - works without any problems.

1 Like
66

not yet due to missing time :frowning: timeslots are currently only max 30min, too short for setting up the complete test environment