Can you tell me the right version and how i can test it?
It’s probably V2, same as MT7622.
Thanks, i try to add mt7623 with version 2 like it’s done for mt7621 and 7622.
Can you give an example for configure nft with hw nat offload?
Not sure about nftables, I only know openwrt way.
One of these examples should help:
table netdev filter {
chain ingress {
type filter hook ingress device eth0 priority 0; flags offload;
ip daddr 192.168.0.10 tcp dport 22 drop
}
}
or
table inet flowoffload {
flowtable f {
hook ingress priority 0
devices = { eno1, eno2, ens1f0, ens1f1, virbr0, virbr-cube-kvm }
}
chain flowoffload {
type filter hook forward priority 0; policy accept;
ip protocol { tcp, udp } flow offload @f
ip6 nexthdr { tcp, udp } flow offload @f
counter
ct state established,related counter accept
ip protocol { tcp, udp } accept
ip6 nexthdr { tcp, udp } accept
}
}
or
table inet x {
flowtable f {
hook ingress priority 0 devices = { eth0, eth1 };
}
chain forward {
type filter hook forward priority 0; policy drop;
# offload established connections
ip protocol { tcp, udp } flow offload @f
ip6 nexthdr { tcp, udp } flow offload @f
counter packets 0 bytes 0
# established/related connections
ct state established,related counter accept
# allow initial connection
ip protocol { tcp, udp } accept
ip6 nexthdr { tcp, udp } accept
}
}Hi, I’m trying to add ipv6 support, but with no success. I’ve attached my changes, I suspect my mtk_flow_mangle_ipv6 implementation is wrong, any ideas? @Ryder.Lee
mtk_ppe_offload.c (11.5 KB)
i guess we need to add the flowtable to a nat-chain, something like this:
table ip filter {
chain input {
type filter hook input priority 0; policy accept;
}
chain output {
type filter hook output priority 0; policy accept;
}
chain forward {
type filter hook forward priority 0; policy accept;
}
}
table ip nat {
flowtable f {
hook ingress priority 10
devices = { wan, lan0 }
}
chain post {
type nat hook postrouting priority 0; policy accept;
oifname "wan" masquerade
#add flowtable here
}
chain pre {
type nat hook prerouting priority 0; policy accept;
}
}
but i got
Error: Could not process rule: Operation not supported
add rule nat post ip protocol tcp flow offload @f
@Ryder.Lee any idea? can i only add flowtable to forwarding-chain like this picture suggests: https://wiki.nftables.org/wiki-nftables/index.php/Flowtable
loaded modules:
nft_flow_offload 16384 0
nf_flow_table_ipv4 16384 1
nf_flow_table 36864 2 nf_flow_table_ipv4,nft_flow_offload
nf_log_ipv4 16384 0
nf_log_common 16384 1 nf_log_ipv4
nft_log 16384 0
nft_reject_ipv4 16384 0
nf_reject_ipv4 16384 1 nft_reject_ipv4
nft_reject 16384 1 nft_reject_ipv4
nft_limit 16384 0
nft_ct 20480 0
nft_masq 16384 1
nft_fib_ipv4 16384 0
nft_fib 16384 1 nft_fib_ipv4
nft_nat 16384 0
nft_chain_nat 16384 2
nf_nat 45056 3 nft_nat,nft_chain_nat,nft_masq
nf_conntrack 135168 6 nft_ct,nft_nat,nf_flow_table,nft_flow_offload,nft_masq,nf_nat
nf_defrag_ipv6 20480 1 nf_conntrack
nf_defrag_ipv4 16384 1 nf_conntrack
nft_counter 16384 0
nf_tables 200704 15 nft_ct,nft_nat,nf_flow_table_ipv4,nft_reject,nft_reject_ipv4,nft_limit,nft_flow_offload,nft_chain_nat,nft_masq,nft_fib,nft_fib_ipv4,nft_counter,nft_log
nfnetlink 20480 1 nf_tables
seems like the series was not sent to mediatek-mailinglist in cc…
i can apply the flow offload rule to forward-chain, but is this right?? how can i check if offloading works (looking on top only while running iperf over it seems like not the best method for it)
table ip filter {
flowtable f {
hook ingress priority 0; devices = { wan, lan0 };
}
chain input {
type filter hook input priority 0; policy accept;
}
chain output {
type filter hook output priority 0; policy accept;
}
chain forward {
type filter hook forward priority 0; policy accept;
ip protocol { tcp, udp } flow offload @f
}
}
table ip nat {
chain post {
type nat hook postrouting priority 0; policy accept;
oifname "wan" masquerade
}
chain pre {
type nat hook prerouting priority 0; policy accept;
}
}
With this config (added lan3 to flowtables devices) i started test
192.168.90.122 (client,iperf3 -c) => 192.168.90.1 [lan3] / 192.168.0.12 [wan] (test-r2) => 192.168.0.10 (main-r2,iperf3 -s)
and looked in debugfs:
root@bpi-r2:~# cat /sys/kernel/debug/mtk_ppe/entries
0051e UNB IPv4 5T orig=192.168.0.21:58552->239.255.255.250:1900 new=0.0.0.0:0->0.0.0.0:0 eth=00:00:00:00:00:00->00:00:00:00:00:00 etype=0000 vlan=0,0 ib1=5000014c ib2=00000000
00742 UNB IPv4 5T orig=192.168.0.10:5201->192.168.0.12:45878 new=0.0.0.0:0->0.0.0.0:0 eth=00:00:00:00:00:00->00:00:00:00:00:00 etype=0000 vlan=0,0 ib1=10001e49 ib2=00000000
0128c UNB IPv4 5T orig=192.168.90.122:37058->193.227.196.10:22067 new=0.0.0.0:0->0.0.0.0:0 eth=00:00:00:00:00:00->00:00:00:00:00:00 etype=0000 vlan=0,0 ib1=1000024d ib2=00000000
01516 UNB IPv4 5T orig=192.168.90.122:52304->162.125.7.20:443 new=0.0.0.0:0->0.0.0.0:0 eth=00:00:00:00:00:00->00:00:00:00:00:00 etype=0000 vlan=0,0 ib1=10000048 ib2=00000000
01672 UNB IPv4 5T orig=193.227.196.10:22067->192.168.0.12:37058 new=0.0.0.0:0->0.0.0.0:0 eth=00:00:00:00:00:00->00:00:00:00:00:00 etype=0000 vlan=0,0 ib1=1000024d ib2=00000000
01b40 UNB IPv4 5T orig=192.168.0.21:48287->239.255.255.250:1900 new=0.0.0.0:0->0.0.0.0:0 eth=00:00:00:00:00:00->00:00:00:00:00:00 etype=0000 vlan=0,0 ib1=5000014c ib2=00000000
01f3a UNB IPv4 5T orig=192.168.90.122:45878->192.168.0.10:5201 new=0.0.0.0:0->0.0.0.0:0 eth=00:00:00:00:00:00->00:00:00:00:00:00 etype=0000 vlan=0,0 ib1=10000049 ib2=00000000
01f3e UNB IPv4 5T orig=192.168.90.122:45876->192.168.0.10:5201 new=0.0.0.0:0->0.0.0.0:0 eth=00:00:00:00:00:00->00:00:00:00:00:00 etype=0000 vlan=0,0 ib1=10000049 ib2=00000000
but see only UNB (entries changing while running iperf) instead of BND. NAT is working as expected as i see on iperf3-server wan-ip of test-r2 instead of clients ip
Accepted connection from 192.168.0.12, port 45876
[ 5] local 192.168.0.10 port 5201 connected to 192.168.0.12 port 45878
with and without flowoffload i don’t see much cpu-load and speed is ~93MBit/s (client has only 100Mbit-card) added flow-version 2 to mt7623…maybe this is wrong?
tried same on r64 with same result
maybe this is related:
root@bpi-r64:~# ethtool -k wan | grep offload
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: on
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
macsec-hw-offload: off [fixed]
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
root@bpi-r64:~#
The new patches do not add xt_flowoffload like in openwrt and i’m not sure if i need additional flag (like hw in owrt). Tries to find any information for hw offloading but found nothing that helps here
This probably means HWNAT hooks are set up using the new way (TC).
.ndo_setup_tc = mtk_eth_setup_tc
If you remove ip protocol { tcp, udp } flow offload @f does ethtool show hw-tc-offload off?
seems like this flag (and the others too) is on by default (after bootup without any nftables ruleset configured):
root@bpi-r64:~# ethtool -k wan | grep offload
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: on
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
macsec-hw-offload: off [fixed]
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
root@bpi-r64:~#
root@bpi-r64:~# nft list ruleset
root@bpi-r64:~#I just tried the new nftables based openwrt firewall and flow offload doesn’t seem to work.
fw4 print:
flowtable f {
hook ingress priority 0; devices = { eth0, eth1 };
}
chain forward {
type filter hook forward priority filter; policy drop;
# offload established connections
ip protocol { tcp, udp } flow offload @f
ip6 nexthdr { tcp, udp } flow offload @f
ct state established,related accept comment "!fw4: Allow forwarded established and related flows"
iifname "br-lan" jump forward_lan comment "!fw4: Handle lan IPv4/IPv6 forward traffic"
iifname "pppoe-wan" jump forward_wan comment "!fw4: Handle wan IPv4/IPv6 forward traffic"
jump handle_reject
}
I think I was able to reduce CPU usage by setting flowtable hook like this:
hook ingress priority 0; devices = { pppoe-wan, br-lan };
sirq went from 49% to 29% during 900 mbps speedtest.
Still no BND entries in mtk_ppe/entries.
According to this commit we need to specify HW_OFFLOAD flag but it says syntax is wrong.
It seems in old openwrt way that flag was hardcoded here, and it seems currently nft cannot enable flowtable hardware offloading.
I tried adding the flag manually in nf_flow_table_core but then nft complained it cannot read the configuration.
https://lwn.net/Articles/804384/ says
table inet x {
flowtable f {
hook ingress priority 10 devices = { eth0, eth1 }
flags offload #<<<<<<<<<<
}
chain y {
type filter hook forward priority 0; policy accept;
ip protocol tcp flow offload @f
}
}
This example above enables the fastpath for TCP traffic between devices
eth0 and eth1. Users can turn on the hardware offload through the
'offload' flag from the flowtable definition.
So i guess we only need flags offload in the flowtable as above example…i hope all kinds of devices work (nic,dsa,bridge,ppp). And ethtool likely tells support not activation of flags
I see commit description youve posted shows this flag in the example too 
but i got error while adding the flags
nft-nat-flowoffload.nft:6:17-21: Error: syntax error, unexpected flags
(tried with and without semicolon in line before)
based on https://www.spinics.net/lists/netfilter/msg59253.html i set tc-offload on all interfaces (they are shown as on before) again to on
root@bpi-r64:~# ethtool -K wan hw-tc-offload on
root@bpi-r64:~# ethtool -K lan3 hw-tc-offload on
root@bpi-r64:~# ethtool -K lan0 hw-tc-offload on
i guess ntf is too old and does not support the flags
root@bpi-r64:~# nft --version
nftables v0.9.0 (Fearless Fosdick)I already updated to latest nftables but with no luck.
root@OpenWrt:/# nft --version
nftables v0.9.8 (E.D.S.)
I guess only solution for now is to patch nf_flow_table (or maybe another file) so we set the flag directly in code for any flowtable.
have you added the “flags offload” option to your config? do you get an error?
cannot get libnftnl/nftables compiled…tried crosscompile too
Yes, still can’t set flags offload.
Latest nftables depends on libnftnl 1.1.9, that is why I could not compile initally.
It seems there is option to specify offload flag for a chain too, not for flowtable, this commit suggets: http://git.netfilter.org/nftables/commit/?id=d42bd56cff1a22301703d2b9d6d6fc937ea7cfbd
with crosscompile i cannot compile libnftl because of missing libmnl
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/…/…/…/…/arm-linux-gnueabihf/bin/ld: cannot find -lmnl
i can install libnml-dev but not for armhf
sudo apt install asciidoc libmnl-dev libnftnl-dev --no-install-recommends
but it seems to exist…maybe i need to add additional repo for it
https://packages.ubuntu.com/focal/armhf/libmnl-dev/filelist
sudo dpkg --add-architecture armhf
sudo apt update #shows many 404 for armhf...
sudo apt install asciidoc libmnl-dev:armhf --no-install-recommends
on native install (very slow) i hang on in configure-step
./configure: line 4091: syntax error near unexpected token `LIBMNL,'
./configure: line 4091: `PKG_CHECK_MODULES(LIBMNL, libmnl >= 1.0.4)'
have installed these:
apt-get install git make gcc dh-autoreconf bison flex asciidoc --no-install-recommends libmnl-dev pkg-config libgmp-dev libreadline-dev
had to run autogen.sh again…now it gets over this point…but no idea about crosscompile
The patch you’ve linked is for nft tool too,so i still need to get libnftnl and nft compiled too…but strange that you still cannot add offload flag to flowtable
I think flowtable just does not support offload flags for now.
I try to patch flow_offload_add:
+__set_bit(NF_FLOWTABLE_HW_OFFLOAD ,flow_table->flags);
if (nf_flowtable_hw_offload(flow_table)) {
__set_bit(NF_FLOW_HW, &flow->flags);
nf_flow_offload_add(flow_table, flow);
}
But still not offload, also kernel crash shortly.
[ 507.363674] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G S 5.10.23 #0
[ 507.371057] Hardware name: Bananapi BPI-R64 (DT)
[ 507.375667] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO BTYPE=--)
[ 507.381677] pc : flow_offload_add+0x9c/0x254 [nf_flow_table]
[ 507.387328] lr : flow_offload_add+0x54/0x254 [nf_flow_table]
[ 507.392977] sp : ffffffc010b3b420
[ 507.396282] x29: ffffffc010b3b420 x28: ffffff8001e0da50
[ 507.401587] x27: ffffff8002022e90 x26: ffffff8000044300
[ 507.406891] x25: 0000000000000000 x24: ffffff8002f53c00
[ 507.412196] x23: 0000000000000000 x22: ffffff8002f53c60
[ 507.417499] x21: ffffffc0109f6000 x20: ffffff8002f53c50
[ 507.422803] x19: ffffff8002f1a200 x18: 0000000000000000
[ 507.428108] x17: 00001ed380d5b700 x16: 0000000000000001
[ 507.433412] x15: ffffffc010b3b370 x14: ffffffc010b3b368
[ 507.438716] x13: 0000000000000002 x12: ffffffc010b3b360
[ 507.444020] x11: ffffff8000c3a000 x10: 0000000002d64d22
[ 507.449324] x9 : 0000000000000000 x8 : ffffff8002f1a300
[ 507.454629] x7 : 00000000a8139c6a x6 : 0000000008e83a30
[ 507.459933] x5 : 00000000881c520d x4 : ffffff8002f7d4d8
[ 507.465237] x3 : 000000010000ca5e x2 : 0000000001499700
[ 507.470541] x1 : 0000000000000000 x0 : 00000000014a615e
[ 507.475846] Call trace:
[ 507.478287] flow_offload_add+0x9c/0x254 [nf_flow_table]
[ 507.483591] 0xffffffc008a46900
[ 507.486727] nft_do_chain+0xc8/0x440 [nf_tables]
[ 507.491338] nft_data_dump+0x1b44/0x2370 [nf_tables]
[ 507.496297] nf_hook_slow+0x48/0xe0
[ 507.499780] ip6_forward+0x454/0x8b0
[ 507.503347] ipv6_rcv+0xb0/0xe0
[ 507.506482] __netif_receive_skb_one_core+0x44/0x50
[ 507.511351] __netif_receive_skb+0x14/0x60
[ 507.515438] netif_receive_skb+0x20/0x94
[ 507.519355] br_pass_frame_up+0x124/0x160
[ 507.523355] br_handle_frame_finish+0x2b8/0x444
[ 507.527877] br_handle_frame+0x35c/0x3ac
[ 507.531791] __netif_receive_skb_core+0x2a0/0xaf0
[ 507.536486] __netif_receive_skb_list_core+0xd0/0x1e4
[ 507.541529] netif_receive_skb_list_internal+0x168/0x260
[ 507.546831] napi_complete_done+0x64/0x1c0
[ 507.550921] mtk_napi_rx+0x56c/0x680
[ 507.554487] __napi_poll+0x34/0x140
[ 507.557967] net_rx_action+0xd4/0x280
[ 507.561621] _stext+0x124/0x28c
[ 507.564756] irq_exit+0xd8/0xf4
[ 507.567890] __handle_domain_irq+0x7c/0xdc
[ 507.571980] gic_handle_irq+0x68/0x8c
[ 507.575633] el1_irq+0xc0/0x180
[ 507.578767] arch_cpu_idle+0x14/0x2c
[ 507.582335] do_idle+0xc4/0x140
[ 507.585467] cpu_startup_entry+0x20/0x50
[ 507.589382] rest_init+0xb8/0xc4
[ 507.592601] arch_call_rest_init+0xc/0x14
[ 507.596601] start_kernel+0x46c/0x484
[ 507.600258] Code: 0b020000 b9000820 b9410281 52800017 (f9400020)
[ 507.606342] ---[ end trace f25d78f236d013d9 ]---Hi, frank.
You can try to add sid repo to install latest nftables (and dep libs). https://packages.debian.org/sid/nftables it has nftables 0.9.8.
Short howto:
1st in yout /etc/apt/source.list change your distro name to sid i.e
deb http://deb.debian.org/debian buster main contrib non-free ->
deb http://deb.debian.org/debian sid main contrib non-free
- apt-update
3.install/update nftables (read carefully what packets are affected, just cancel if you don’t like it)
- replace sid to your distro, and repeat apt-update.
1 Like
thanks i try this on r2 then…but first i try to get it working on r64 with the compile-variant (currently natively on r64 as crosscompile does not work, maybe i try a armhf-chroot on my laptop as third option similar to openwrt build-processs and afair like i’ve done for openssl) as it looks like we need additional patches
for crosscompile, maybe i need to crosccompile libnml too and add lib while configure like it’s done here: https://stackoverflow.com/questions/63435221/issues-cross-compiling-libnftnl-for-arm but have not yet found source for it…
contacted author pablo naira ayuso of the flags-patch above about the missing flags support in flowoffload chain
Ok, I finally got it to work, this is the patch I made in nf_flow_table_core.c
int nf_flow_table_init(struct nf_flowtable *flowtable)
{
int err;
INIT_DEFERRABLE_WORK(&flowtable->gc_work, nf_flow_offload_work_gc);
flow_block_init(&flowtable->flow_block);
init_rwsem(&flowtable->flow_block_lock);
err = rhashtable_init(&flowtable->rhashtable,
&nf_flow_offload_rhash_params);
if (err < 0)
return err;
queue_delayed_work(system_power_efficient_wq,
&flowtable->gc_work, HZ);
mutex_lock(&flowtable_lock);
list_add(&flowtable->list, &flowtables);
mutex_unlock(&flowtable_lock);
+flowtable->flags = NF_FLOWTABLE_HW_OFFLOAD;
return 0;
}thanks but this always enables hw offload independed of config for each flowtable, right?
i still hang on native compilation…i got ld-errors for libnftnl (have used default prefix for the lib while install…). tried to build it like the bash-script there, but still got the ld-errors
Yes, this was only for testing new openwrt firewall based on nftables. nftables needs to support offload flag in future to configure it there.
Now to see if my ipv6 patch for mtk_ppe_offload works.