WED and beamforming on BPI-R3

Banana Pi Router design BPI-R3/R3 Mini(MT7986)
99

Here are my nft rules:

table inet filter {
        flowtable f {
                hook ingress priority filter
                devices = { lan0, lan1, lan2, lan3, wan }
                flags offload
        }

        chain input {
                type filter hook input priority filter; policy drop;
                iifname "br-lan" accept comment "Allow local network to access the router"
                iifname "wan" ct state { established, related } accept comment "Allow established traffic"
                iifname "wan" icmp type { destination-unreachable, echo-request, time-exceeded } counter  accept comment "Allow select ICMP"
                iifname "wan" counter packets drop comment "Drop all other unsolicited traffic from wan"
                iifname "lo" accept comment "Accept everything from loopback interface"
        }

        chain forward {
                type filter hook forward priority filter; policy drop;
                ip protocol { tcp, udp } flow add @f
                iifname "br-lan" oifname "wan" accept comment "Allow trusted LAN to WAN"
                iifname "wan" oifname "br-lan" ct state established,related accept comment "Allow established back to LANs"
        }
}
table ip nat {
        chain postrouting {
                type nat hook postrouting priority srcnat; policy accept;
                oifname "wan" masquerade
        }
}

The offloading seems to work for the cable-based interfaces. If I try to add wlan0 and wlan1 to that the service fails to start with following error:

systemd[1]: Reloaded nftables firewall.
systemd[1]: Reloading nftables firewall...
0iqi9cnxh7dy79sjqaldp26bcs4fnlcx-nftables-rules[11751]: /nix/store/0iqi9cnxh7dy79sjqaldp26bcs4fnlcx-nftables-rules:8:14-14: Error: Could not process rule: Operation not supported
0iqi9cnxh7dy79sjqaldp26bcs4fnlcx-nftables-rules[11751]:    flowtable f {
0iqi9cnxh7dy79sjqaldp26bcs4fnlcx-nftables-rules[11751]: /nix/store/0iqi9cnxh7dy79sjqaldp26bcs4fnlcx-nftables-rules:25:30-44: Error: Could not process rule: No such file or directory
0iqi9cnxh7dy79sjqaldp26bcs4fnlcx-nftables-rules[11751]:     ip protocol { tcp, udp } flow offload @f
0iqi9cnxh7dy79sjqaldp26bcs4fnlcx-nftables-rules[11751]:

wlan0 and wlan1 are managed by hostapd.

100

The packets follow the normal forwarding rules, until they are established. Then offload kicks in and forwarding rules are totally bypassed for the established connection.

Does it mean that the priority of flowtable doesnā€™t matter?

101

Apart from the wlan issue:

                                         userspace process
                                          ^              |
                                          |              |
                                     _____|____     ____\/___
                                    /          \   /         \
                                    |   input  |   |  output |
                                    \__________/   \_________/
                                         ^               |
                                         |               |
      _________      __________      ---------     _____\/_____
     /         \    /          \     |Routing |   /            \
  -->  ingress  ---> prerouting ---> |decision|   | postrouting|--> neigh_xmit
     \_________/    \__________/     ----------   \____________/          ^
       |      ^                          |               ^                |
   flowtable  |                     ____\/___            |                |
       |      |                    /         \           |                |
    __\/___   |                    | forward |------------                |
    |-----|   |                    \_________/                            |
    |-----|   |                 'flow offload' rule                       |
    |-----|   |                   adds entry to                           |
    |_____|   |                     flowtable                             |
       |      |                                                           |
      / \     |                                                           |
     /hit\_no_|                                                           |
     \ ? /                                                                |
      \ /                                                                 |
       |__yes_________________fastpath bypass ____________________________|

               Fig.1 Netfilter hooks and flowtable interactions

Looks like your tcp/udp packets are going, without applying the forwarding rules, once the flowtable entry is addedā€¦ Then all tcp/udp packets follow the fastpath, not only the established. This may not be a great idea when wan is added to the flowtable devices.

So hook ingress is always before hook forward, see schematic. The priority of hook ingress only priorities within the hooks ingress.

102

Great catch! Thank you. I know that this is a little bit off-topic but would you recommend narrowing more the offload rule to only apply for example to established http connections like:

 # offload established HTTP connections
 tcp dport { 80, 443 } ct state established flow offload @f counter packets 0 bytes 0

or disable offloading for wan interface entirely?

103

I still have to experiment with it myself, but it should be ok when only using it for established traffic, since this is what we filter in the input/forward rules anywayā€¦

104

Imho the right way should be offload established traffic and accept http(s) separately (2 rules)

105

Agree. First the traffic needs to pass the forward rules twice (incoming and reply), so an accept rule needs to match with the desired traffic.

Then a flow offload rule needs to also match the desired traffic, so a flow table entry will be created after the traffic has gone through including the reply.

106

@dangowrt can you tell me how i can setup wed with nftables? Or is bridger a hard requirement? I do not plan to bridge wifi devices atm

maybe you can point us to the changes and build-code for bridger

107

Now you made me think. I thought that I donā€™t need bridger but I am not sure anymore.

Flow offloading happens in situations where flows are actually being tracked, ie. when using NAT/routing. To use WED on a dumb access point requires the bridger package which does a bit of BPF magic to get bridged streams tracked (and hence offloaded).

In my case I have a bridge that includes (lan0-3 and wlan0-1) but I also have some routing configured between that bridę and the WAN interface.

Do I need the bridger in this case to benefit from WED?

108

How about offloading wlan to lan? That is not routed or through nat. Even there is a nat configured, but it is for wan.

So need to remove wlan from the bridge and forward traffic, in order to get hw-offlioading to/from wlan started?

I need wlan in the bridge for 802.11r roamingā€¦

109

No, or at least not for traffic between WAN and Wi-Fi interfaces. Traffic between LAN and Wi-Fi wonā€™t be offloaded without bridger.

110

Thanks, for now that will be enough for me. Do you know how I can enable enable flow table for wlan interface using nftables? If I try to add a wlan interface to my existing flowtable I am getting following error: Error: Could not process rule: Operation not supported

111

Do you mean that wed is not usable with non openwrt linux userspace? Bridger seems to be only compatible with openwrt.

Why it is not similar to wan/lan/ppp offloading?

112

Did you set the wed_enable=Y module parameter for the mt7915e module?

113

It very much is. And thatā€™s why it depends on connection tracking and flows, just like any PPE forwarding offload function. However, if you just put two interfaces into a bridge on Linux there is no connections tracking on layer-3 what-so-ever, thus PPE cannot bind to anything and no offloading will happen.

bridger is a small tool which installs eBPF programs on tc layer on the relevant bridge member interfaces which enables connection tracking on them, hence allowing for bridged flows to be offloaded.

114

i do not want to bridge wifi and lanā€¦

but i cannot get it working too


root@bpi-r3:~# cat /etc/nftables.conf                                                                                                                       
#!/usr/sbin/nft -f

define iflan="lanbr0"
define ifinternal={$iflan,"wlan0","wlan1"}

flush ruleset

table inet filter {
        flowtable f {
                hook ingress priority filter
                devices = { lan0, lan1, lan2, lan3, wan }
                #devices = { lan0, lan1, lan2, lan3, wan, wlan0, wlan1 }
                flags offload
        }
        chain input {
                type filter hook input priority filter;policy drop;
                limit rate 5/second icmp type {echo-request, echo-reply} accept comment "limit icmp to 5/s"
                iifname "lo" accept comment "accept loopback"
                iifname $ifinternal accept comment "allow traffic from internal interfaces"
                reject
        }
        chain forward {
                type filter hook forward priority filter;policy drop;
                ct state invalid counter drop comment "early drop of invalid packets"
                ct state established flow offload @f counter
                ct state {established,related} accept comment "allow connections initiated"
        }
        chain output {
                type filter hook output priority filter;
        }
}

table ip nat {
        chain postrouting {
                type nat hook postrouting priority srcnat; policy accept;
                oifname "wan" masquerade
        }
}

if i enable the line with wlan devices (of course disable the other one) nftables cannot startā€¦i dontā€™t get any specific error, only this:

root@bpi-r3:~# nft -f /etc/nftables.conf                                                                                                                    
/etc/nftables.conf:9:19-19: Error: Could not process rule: Operation not supported
        flowtable f {
                  ^
/etc/nftables.conf:25:24-38: Error: Could not process rule: No such file or directory
                ct state established flow offload @f counter
                                     ^^^^^^^^^^^^^^^

root@bpi-r3:~# cat /sys/module/mt7915e/parameters/wed_enable                                                                                     
Y
root@bpi-r3:~# cat /sys/kernel/debug/mtk_ppe/bind                                                                                                
cat: /sys/kernel/debug/mtk_ppe/bind: No such file or directory
root@bpi-r3:~#

edit: noticed iā€™m currently running 6.1.50 on that deviceā€¦i guess need to switch to later kernelā€¦also with 6.5.0 i have same errorā€¦

of course i have debugfs mountedā€¦i see entries like wed0 and wed1 there, but no bind

root@bpi-r3:~# ls /sys/kernel/debug/
bdi               dynamic_debug       mmc0      pwm         suspend_stats
block             extfrag             mtd       ras         swiotlb
clear_warn_once   f2fs                opp       regmap      tracing
clk               fault_around_bytes  phy       regulator   ubi
debug_enabled     gpio                pinctrl   sched       ubifs
device_component  hid                 pm_genpd  sfp-1       usb
devices_deferred  ieee80211           ppe0      sfp-2       wakeup_sources
dma_pools         iio                 ppe1      slab        wed0
dmaengine         memblock            psci      sleep_time  wed1
root@bpi-r3:~# ls /sys/kernel/debug/wed0/
regidx  regval  rxinfo  txinfo  
root@bpi-r3:~# ls /sys/kernel/debug/wed1/
regidx  regval  rxinfo  txinfo

see no mtk_ppe (searched for ppe) in menuconfigā€¦

115

No I did not. I will set it and report back.

In the meantime, can some one explain to me the difference between:

$ cat /sys/module/mt7915e/parameters/wed_enable      
N

and

$  zgrep 'WED' /proc/config.gz                                                                                                                                                                                                                          
CONFIG_NET_MEDIATEK_SOC_WED=y

Why one returns Y and the other returns N?

116

config.gz says your kernel has the wed driver builtinā€¦the other tells options passed to the wifi driver module

root@bpi-r3:~# cat /etc/modprobe.d/wed.conf                                                                                                                 
options mt7915e wed_enable=Y
117

Thanks @frank-w :+1:

I enabled the option for kernel module. I can now see

$ cat /sys/module/mt7915e/parameters/wed_enable                                                                                                                                                                                                        
Y

However if I try to add a wlan interface to flowtable I fails with the same error as before:

nftables-rules[2565]: /nix/store/ivczmcr22329bbqrvc4cfhf256f7qp2q-nftables-rules:8:14-14: Error: Could not process rule: Operation not supported
nftables-rules[2565]:    flowtable f {
nftables-rules[2565]:              ^
nftables-rules[2565]: /nix/store/ivczmcr22329bbqrvc4cfhf256f7qp2q-nftables-rules:25:55-69: Error: Could not process rule: No such file or directory
nftables-rules[2565]:     ip protocol { tcp, udp } ct state { established } flow offload @f comment "Offload tcp/udp established traffic"
nftables-rules[2565]: 

$ uname -a                                                                                                                                                                                                                                             
Linux surfer 6.4.11 #1-NixOS SMP Wed Aug 16 16:32:31 UTC 2023 aarch64 GNU/Linux
118

@dangowrt

Since kernel 5.3 brought the module nf_conntrack_bridge to have conntrack available at the bridge layer, allowing stateful firewalling

If we need the flow to be tracked can nftables conntrack be setup on a bridge as well? Perhapse something like so:

table bridge filter {
        chain input {
                type filter hook input priority -200; policy accept;
        }

        chain forward {
                type filter hook forward priority -200; policy accept;
        }

        chain output {
                type filter hook output priority -200; policy accept;
        }
}

And then adding the fast-path in these rules?