New netfilter flow table based HNAT

frank-w · March 20, 2021, 2:28pm

still have not got nft compiled…this is my (crosscompile) way:

git clone https://git.netfilter.org/libmnl
git clone git://git.netfilter.org/libnftnl
git clone git://git.netfilter.org/nftables

#build libmnl for armhf
cd libmnl
sh autogen.sh
mkdir install
sh configure --host=arm-linux-gnueabihf --prefix=$(pwd)/install
make
make install

#build libnftnl
cd ../libnftnl/
sh autogen.sh
mkdir install
./configure --host=arm-linux-gnueabihf --prefix=$(pwd)/install LIBMNL_LIBS=../libmnl/install/lib LIBMNL_CFLAGS=../libmnl/include/
make
make install

#build nft
cd ../nftables/
sh autogen.sh
sh configure --host=arm-linux-gnueabihf --prefix=$(pwd)/install PKG_CONFIG_PATH="../libnftnl/install/lib/pkgconfig:../libmnl/install/lib/pkgconfig" LIBMNL_LIBS=../libmnl/install/lib LIBMNL_CFLAGS=../libmnl/include/

still got:

checking for LIBMNL... yes
checking for LIBNFTNL... yes
checking for __gmpz_init in -lgmp... no
configure: error: No suitable version of libgmp found

seems like this is also an armhf-lib required, but have not found any repo to build it…and cannot install it on x86_64 host seems like i can obtain source from here https://gmplib.org/#DOWNLOAD, but only mercurial repo and no git…download source-package (*.xz) is very slow

wget https://ftp.gnu.org/gnu/gmp/gmp-6.2.1.tar.xz
tar -xvf gmp-6.2.1.tar.xz
cd gmp-6.2.1/
./configure --host=arm-linux-gnueabihf --prefix=$(pwd)/install
make
make install

last step suggests make check, which fails

FAIL: t-bswap
FAIL: t-constants
FAIL: t-count_zeros
FAIL: t-hightomask
FAIL: t-modlinv
FAIL: t-popc
FAIL: t-parity
FAIL: t-sub

netherless i tried to include it into nftables-config, but this does also fail

sh configure --host=arm-linux-gnueabihf --prefix=$(pwd)/install PKG_CONFIG_PATH="../libnftnl/install/lib/pkgconfig:../libmnl/install/lib/pkgconfig:../gmp-6.2.1/install/lib/pkgconfig" LIBMNL_LIBS=../libmnl/install/lib LIBMNL_CFLAGS=../libmnl/include/ LIBGMP_LIBS=../libmnl/install/lib LIBGMP_CFLAGS=../gmp-6.2.1/include

checking for __gmpz_init in -lgmp... no
configure: error: No suitable version of libgmp found

but i got configure working with --with-mini-gmp --without-cli

without-cli disables readline library…i’m not sure this is right as nft is a cli-tool

for make i had to use absolute names (instead of …/)

BASE="$(pwd)/../"; sh configure --host=arm-linux-gnueabihf --prefix=$(pwd)/install PKG_CONFIG_PATH="$BASE/libnftnl/install/lib/pkgconfig:$BASE/libmnl/install/lib/pkgconfig" LIBMNL_LIBS=$BASE/libmnl/install/lib LIBMNL_CFLAGS=$BASE/libmnl/include/ --with-mini-gmp --without-cli

native-way still stops on ld-error

but now i’m on same state as native compile

  CCLD     nft
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_attr_nest_start'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_socket_bind'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_nlmsg_batch_current'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_attr_nest_end'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_attr_put_u16'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_nlmsg_fprintf'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_attr_put_u8'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_nlmsg_get_payload'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_attr_get_u32'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_attr_get_payload_len'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_nlmsg_batch_is_empty'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_nlmsg_get_payload_offset'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_attr_get_u16'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_nlmsg_get_payload_len'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_nlmsg_put_extra_header'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_attr_next'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_attr_get_u64'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_attr_put_u64'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_nlmsg_get_payload_tail'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_nlmsg_batch_next'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_socket_setsockopt'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_nlmsg_batch_stop'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_attr_parse'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_nlmsg_put_header'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_socket_get_portid'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_attr_get_str'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_nlmsg_batch_head'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_cb_run'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_attr_put_strz'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_attr_get_type'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_attr_get_u8'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_nlmsg_batch_start'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_attr_validate'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_attr_put_str'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_attr_ok'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_socket_sendto'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_socket_recvfrom'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_nlmsg_size'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_attr_put_u32'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_nlmsg_batch_size'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_cb_run2'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_socket_open'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_attr_get_payload'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /media/data_nvme/git/nft/libnftnl/install/lib/libnftnl.so.11: undefined reference to `mnl_attr_parse_nested'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_attr_put'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_socket_get_fd'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_socket_close'
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: ./.libs/libnftables.so: undefined reference to `mnl_attr_type_valid'

i tried to pass the lib dirs with LDFLAGS,LD_LIBRARY_PATH and --libdir= to configure…still get the linker-errors

tried also to configure libnftnl without relative paths

BASE="$(pwd)/..";
./configure --host=arm-linux-gnueabihf --prefix=$(pwd)/install LIBMNL_LIBS=$BASE/libmnl/install/lib LIBMNL_CFLAGS=$BASE/libmnl/include/
make clean
make
make install
cd ../nftables
sh configure --host=arm-linux-gnueabihf --prefix=$(pwd)/install PKG_CONFIG_PATH="$BASE/libnftnl/install/lib/pkgconfig:$BASE/libmnl/install/lib/pkgconfig" LIBMNL_LIBS=$BASE/libmnl/install/lib LIBMNL_CFLAGS=$BASE/libmnl/include
make clean
make

but still the same

graphine · March 20, 2021, 6:52pm

My ipv6 patch works in nftables, it was not working in iptables + xt_flowoffload.

940 mbps over ipv6 (pppoe) at no CPU cost. Pretty impressinve.

Now it’s time to explore HWQoS.

frank-w · March 20, 2021, 7:01pm

Can you send it as follow up to the series from post #1?

Maybe your fix for flowtable to get a real configurable solution

graphine · March 20, 2021, 7:22pm

I don’t know how to do that, never sent a patch before . Feel free to do so.

Also I’m not sure the mangle part is correct.

ipv6.patch (2.7 KB)

frank-w · March 20, 2021, 7:30pm

First i need to get it compiled mhm, seems to work if i leave the vars in the last configure step

BASE="$(pwd)/..";
./configure --host=arm-linux-gnueabihf --prefix=$(pwd)/install PKG_CONFIG_PATH="$BASE/libnftnl/install/lib/pkgconfig:$BASE/libmnl/install/lib/pkgconfig" --with-mini-gmp --without-cli

and on native compile seems like PKG_CONFIG_PATH was not set correctly (and needed to install libreadline-dev docbook-xsl xsltproc libxml2-utils)

For supporting flags in flowtable i guess we need token here similar to patch above: http://git.netfilter.org/nftables/tree/src/parser_bison.y?id=d42bd56cff1a22301703d2b9d6d6fc937ea7cfbd#n1886

I really do not understand bison,but it looks like here the token for devices is added. I guess flags can be next, of course this is only cli parser and needs to add functionality in code

frank-w · March 21, 2021, 3:10pm

got it working

root@bpi-r64:~# /home/frank/nftables/install/sbin/nft -f nft-nat-flowoffload.nft
root@bpi-r64:~# /home/frank/nftables/install/sbin/nft list ruleset
table ip filter {
        flowtable f {
                hook ingress priority filter + 1
                devices = { lan3, lan0, wan }
 flags offload; }

        chain input {
                type filter hook input priority filter; policy accept;
        }

        chain output {
                type filter hook output priority filter; policy accept;
        }

        chain forward {
                type filter hook forward priority filter; policy accept;
                ip protocol { tcp, udp } flow add @f
        }
}
table ip nat {
        chain post {
                type nat hook postrouting priority filter; policy accept;
                oifname "wan" masquerade
        }

        chain pre {
                type nat hook prerouting priority filter; policy accept;
        }
}
root@bpi-r64:~# cat /sys/kernel/debug/mtk_ppe/entries
root@bpi-r64:~# echo 1 > /proc/sys/net/ipv4/ip_forward
root@bpi-r64:~# cat /sys/kernel/debug/mtk_ppe/entries
0074e BND IPv4 5T orig=192.168.0.10:5201->192.168.0.18:36142 new=192.168.0.10:5201->192.168.80.122:36142 eth=36:c7:65:74:18:bc->00:13:77:b7:a7:62 etype=1000 vlan=0,0 ib1=21412f3e ib2=007ff020
00c00 UNB IPv6 5T orig=fe80:0000:0000:0000:6387:b15f:d347:b958:41038->ff12:0000:0000:0000:0000:0000:0000:8384:21027 eth=00:00:00:00:00:00->00:00:00:00:00:00 etype=0000 vlan=0,0 ib1=5a000039 ib2=00000000
012d6 UNB IPv4 5T orig=192.168.80.122:52327->192.168.80.1:5351 new=0.0.0.0:0->0.0.0.0:0 eth=00:00:00:00:00:00->00:00:00:00:00:00 etype=0000 vlan=0,0 ib1=5000003a ib2=00000000
01314 UNB IPv4 5T orig=192.168.0.21:33923->192.168.0.255:21027 new=0.0.0.0:0->0.0.0.0:0 eth=00:00:00:00:00:00->00:00:00:00:00:00 etype=0000 vlan=0,0 ib1=5000013c ib2=00000000
0171e BND IPv4 5T orig=192.168.80.122:36142->192.168.0.10:5201 new=192.168.0.18:36142->192.168.0.10:5201 eth=36:c7:65:74:18:bc->08:02:00:00:00:10 etype=0100 vlan=0,0 ib1=21412f3e ib2=007ff020
01722 BND IPv4 5T orig=192.168.80.122:36144->192.168.0.10:5201 new=192.168.0.18:36144->192.168.0.10:5201 eth=36:c7:65:74:18:bc->08:02:00:00:00:10 etype=0100 vlan=0,0 ib1=21412f42 ib2=007ff020
01772 BND IPv4 5T orig=192.168.0.10:5201->192.168.0.18:36144 new=192.168.0.10:5201->192.168.80.122:36144 eth=36:c7:65:74:18:bc->00:13:77:b7:a7:62 etype=1000 vlan=0,0 ib1=21412f42 ib2=007ff020
root@bpi-r64:~# cat /sys/kernel/debug/mtk_ppe/entries
0074e BND IPv4 5T orig=192.168.0.10:5201->192.168.0.18:36142 new=192.168.0.10:5201->192.168.80.122:36142 eth=36:c7:65:74:18:bc->00:13:77:b7:a7:62 etype=1000 vlan=0,0 ib1=21412f3e ib2=007ff020
0171e BND IPv4 5T orig=192.168.80.122:36142->192.168.0.10:5201 new=192.168.0.18:36142->192.168.0.10:5201 eth=36:c7:65:74:18:bc->08:02:00:00:00:10 etype=0100 vlan=0,0 ib1=21412f3e ib2=007ff020
01722 BND IPv4 5T orig=192.168.80.122:36144->192.168.0.10:5201 new=192.168.0.18:36144->192.168.0.10:5201 eth=36:c7:65:74:18:bc->08:02:00:00:00:10 etype=0100 vlan=0,0 ib1=21412f46 ib2=007ff020
01772 BND IPv4 5T orig=192.168.0.10:5201->192.168.0.18:36144 new=192.168.0.10:5201->192.168.80.122:36144 eth=36:c7:65:74:18:bc->00:13:77:b7:a7:62 etype=1000 vlan=0,0 ib1=21412f46 ib2=007ff020
root@bpi-r64:~#

traffic increases, but not much (80Mbit/s->96Mbit/s) as my test-device if 100Mbit/s only, but i see that cpu is not limiting anymore

except the little printing-issue in “list ruleset” it’s working as expected

edit: printing-issue also fixed:

frank@bpi-r64:~/nftables$ sudo install/sbin/nft list ruleset
table ip filter {
        flowtable f {
                hook ingress priority filter + 1
                devices = { lan3, lan0, wan }
                flags offload;
        }

Send patch to mailinglist…i hope it is right

https://patchwork.ozlabs.org/project/netfilter-devel/patch/[email protected]/

frank-w · March 22, 2021, 8:46am

one problem resides with cross-compiled version (i try it now on r2)…

i copied the content of install/lib of all crosscompiled libs folder to /usr/local/lib, ran “/sbin/ldconfig -v” and started my nft binary

root@bpi-r2:~# /home/frank/nft_install/sbin/nft
/home/frank/nft_install/sbin/nft: /usr/lib/arm-linux-gnueabihf/libnftnl.so.11: version `LIBNFTNL_13' not found (required by /usr/local/lib/libnftables.so.1)
/home/frank/nft_install/sbin/nft: /usr/lib/arm-linux-gnueabihf/libnftnl.so.11: version `LIBNFTNL_15' not found (required by /usr/local/lib/libnftables.so.1)
/home/frank/nft_install/sbin/nft: /usr/lib/arm-linux-gnueabihf/libnftnl.so.11: version `LIBNFTNL_14' not found (required by /usr/local/lib/libnftables.so.1)
/home/frank/nft_install/sbin/nft: /usr/lib/arm-linux-gnueabihf/libnftnl.so.11: version `LIBNFTNL_16' not found (required by /usr/local/lib/libnftables.so.1)

ldconfig shows the libs, but it seems they are not found

/usr/local/lib:
/sbin/ldconfig: /usr/local/lib/libmnl.so.0 is not a symbolic link

        libmnl.so.0 -> libmnl.so.0.2.0
/sbin/ldconfig: /usr/local/lib/libnftnl.so.11 is not a symbolic link

        libnftnl.so.11 -> libnftnl.so.11.5.0
/sbin/ldconfig: /usr/local/lib/libnftables.so.1 is not a symbolic link

        libnftables.so.1 -> libnftables.so.1.0.0

any idea?

seems system tries to load old libnftl

root@bpi-r2:~# ldd /home/frank/nft_install/sbin/nft
/home/frank/nft_install/sbin/nft: /usr/lib/arm-linux-gnueabihf/libnftnl.so.11: version `LIBNFTNL_13' not found (required by /usr/local/lib/libnftables.so.1)
/home/frank/nft_install/sbin/nft: /usr/lib/arm-linux-gnueabihf/libnftnl.so.11: version `LIBNFTNL_15' not found (required by /usr/local/lib/libnftables.so.1)
/home/frank/nft_install/sbin/nft: /usr/lib/arm-linux-gnueabihf/libnftnl.so.11: version `LIBNFTNL_14' not found (required by /usr/local/lib/libnftables.so.1)
/home/frank/nft_install/sbin/nft: /usr/lib/arm-linux-gnueabihf/libnftnl.so.11: version `LIBNFTNL_16' not found (required by /usr/local/lib/libnftables.so.1)
        linux-vdso.so.1 (0xbef7f000)
        libnftables.so.1 => /usr/local/lib/libnftables.so.1 (0xb6f08000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6e0e000)
        /lib/ld-linux-armhf.so.3 (0xb6f98000)
        libmnl.so.0 => /lib/arm-linux-gnueabihf/libmnl.so.0 (0xb6df9000)
        libnftnl.so.11 => /usr/lib/arm-linux-gnueabihf/libnftnl.so.11 (0xb6dd1000)

how can i tell nft to use the libs from /usr/local/bin?

got it with a (dirty) hack:

root@bpi-r2:~# mv /usr/lib/arm-linux-gnueabihf/libnftnl.so.11{,.bak}
root@bpi-r2:~# ln -fs /usr/local/lib/libnftnl.so.11 /usr/lib/arm-linux-gnueabihf/libnftnl.so.11                                                                          
root@bpi-r2:~# /home/frank/nft_install/sbin/nft
/home/frank/nft_install/sbin/nft: no command specified
root@bpi-r2:~#

DeadMeat · March 22, 2021, 9:06am

Hi Frank, please try to read man ld.so there is a description of library search order. I don’r remember details, but i hope this may help.

frank-w · March 22, 2021, 9:14am

thx alex…i know ld only for linking while compiling not as system-configuration…but looked through the very large manpage

found this:

For a native linker on an ELF system, if the file /etc/ld.so.conf exists, the list of directories found in that file.

/etc/ld.so.conf contains “include /etc/ld.so.conf.d/*.conf”

where i have these files (don’t know which one comes first or which is taken) /etc/ld.so.conf.d/arm-linux-gnueabihf.conf /etc/ld.so.conf.d/fakeroot-arm-linux-gnueabihf.conf /etc/ld.so.conf.d/libc.conf

i guess /etc/ld.so.conf.d/arm-linux-gnueabihf.conf is used for my binary which contains these lines:

/usr/local/lib/arm-linux-gnueabihf
/lib/arm-linux-gnueabihf
/usr/lib/arm-linux-gnueabihf

so maybe i should put my libs into /usr/local/lib/arm-linux-gnueabihf instead of /usr/local/lib/ directly

DeadMeat · March 22, 2021, 9:23am

As for me, the main thing is:

 If a shared object dependency does not contain a slash, then it is searched for in the following order:

       o  Using the directories specified in the DT_RPATH dynamic section attribute of the binary if present and DT_RUNPATH attribute does not exist.  Use of DT_RPATH is deprecated.

       o  Using the environment variable LD_LIBRARY_PATH, unless the executable is being run in secure-execution mode (see below), in which case this variable is ignored.

       o  Using the directories specified in the DT_RUNPATH dynamic section attribute of the binary if present.  Such directories are searched only to find those objects required by DT_NEEDED (direct dependencies)  entries  and  do
          not apply to those objects' children, which must themselves have their own DT_RUNPATH entries.  This is unlike DT_RPATH, which is applied to searches for all children in the dependency tree.

       o  From  the  cache  file  /etc/ld.so.cache,  which contains a compiled list of candidate shared objects previously found in the augmented library path.  If, however, the binary was linked with the -z nodeflib linker option,
          shared objects in the default paths are skipped.  Shared objects installed in hardware capability directories (see below) are preferred to other shared objects.

       o  In the default path /lib, and then /usr/lib.  (On some 64-bit architectures, the default paths for 64-bit shared objects are /lib64, and then /usr/lib64.)  If the binary was linked with the -z nodeflib linker option, this
          step is skipped.

So you still can place libs directly to one of these paths , but you may also try ro replace/modify LD_LIBRARY_PATH and to set needed path, so it should be checked earlier than /etc/ld.so.cache

frank-w · March 22, 2021, 9:43am

seems like this…but still error

root@bpi-r2:~# LD_LIBRARY_PATH=/usr/local/lib /home/frank/nft_install/sbin/nft
/home/frank/nft_install/sbin/nft: /usr/local/lib/libnftnl.so.11: version `LIBNFTNL_13' not found (required by /usr/local/lib/libnftables.so.1)

Zusammenfassung

seems this is only done for first iteration…not for depency of loaded shared lib

root@bpi-r2:~# ls -l /usr/local/lib
total 14096
drwxr-xr-x 2 root root     4096 Mar 22 10:26 arm-linux-gnueabihf
-rwxr-xr-x 1 root root      946 Mar 22 09:33 libmnl.la
-rwxr-xr-x 1 root root    64200 Mar 22 09:33 libmnl.so
-rwxr-xr-x 1 root root    64200 Mar 22 09:33 libmnl.so.0
-rwxr-xr-x 1 root root    64200 Mar 22 09:33 libmnl.so.0.2.0
-rw-r--r-- 1 root root  3984680 Mar 22 09:28 libnftables.a
-rwxr-xr-x 1 root root     1197 Mar 22 09:28 libnftables.la
-rwxr-xr-x 1 root root  2669132 Mar 22 09:28 libnftables.so
-rwxr-xr-x 1 root root  2669132 Mar 22 09:28 libnftables.so.1
-rwxr-xr-x 1 root root  2669132 Mar 22 09:28 libnftables.so.1.0.0
-rwxr-xr-x 1 root root      964 Mar 22 09:29 libnftnl.la
-rwxr-xr-x 1 root root  1057580 Mar 22 09:29 libnftnl.so
-rwxr-xr-x 1 root root    97240 Mar 22 10:27 libnftnl.so.11
-rwxr-xr-x 1 root root  1057580 Mar 22 09:29 libnftnl.so.11.5.0
drwxrwsr-x 3 root staff    4096 Oct  2  2019 python3.7

put now the libs in the created folder /usr/local/lib/arm-linux-gnueabihf and run ldconfig -v

root@bpi-r2:~# ldconfig -v
ldconfig: Path `/lib/arm-linux-gnueabihf' given more than once
ldconfig: Path `/usr/lib/arm-linux-gnueabihf' given more than once
/usr/local/lib/arm-linux-gnueabihf:
ldconfig: /usr/local/lib/arm-linux-gnueabihf/libmnl.so.0 is not a symbolic link

        libmnl.so.0 -> libmnl.so.0.2.0
ldconfig: /usr/local/lib/arm-linux-gnueabihf/libnftnl.so.11 is not a symbolic link

        libnftnl.so.11 -> libnftnl.so.11.5.0
ldconfig: /usr/local/lib/arm-linux-gnueabihf/libnftables.so.1 is not a symbolic link

        libnftables.so.1 -> libnftables.so.1.0.0
/lib/arm-linux-gnueabihf:
...
/usr/local/lib:
ldconfig: /usr/local/lib/libmnl.so.0 is not a symbolic link

        libmnl.so.0 -> libmnl.so.0.2.0
ldconfig: /usr/local/lib/libnftnl.so.11 is not a symbolic link

        libnftnl.so.11 -> libnftnl.so.11.5.0
ldconfig: /usr/local/lib/libnftables.so.1 is not a symbolic link

        libnftables.so.1 -> libnftables.so.1.0.0

and got same

root@bpi-r2:~# /home/frank/nft_install/sbin/nft
/home/frank/nft_install/sbin/nft: /usr/local/lib/arm-linux-gnueabihf/libnftnl.so.11: version `LIBNFTNL_13' not found (required by /usr/local/lib/arm-linux-gnueabihf/libnftables.so.1)

this is strange as the file exists and is the new version of the file…i guess again the older version is used

ldd shows right one

root@bpi-r2:~# ldd /usr/local/lib/arm-linux-gnueabihf/libnftnl.so.11
        linux-vdso.so.1 (0xbefcb000)
        libmnl.so.0 => /usr/local/lib/arm-linux-gnueabihf/libmnl.so.0 (0xb6f51000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6e57000)
        /lib/ld-linux-armhf.so.3 (0xb6f8d000)
root@bpi-r2:~# ldd /usr/local/lib/arm-linux-gnueabihf/libnftables.so.1
/usr/local/lib/arm-linux-gnueabihf/libnftables.so.1: /usr/local/lib/arm-linux-gnueabihf/libnftnl.so.11: version `LIBNFTNL_13' not found (required by /usr/local/lib/arm-l
inux-gnueabihf/libnftables.so.1)
/usr/local/lib/arm-linux-gnueabihf/libnftables.so.1: /usr/local/lib/arm-linux-gnueabihf/libnftnl.so.11: version `LIBNFTNL_15' not found (required by /usr/local/lib/arm-l
inux-gnueabihf/libnftables.so.1)
/usr/local/lib/arm-linux-gnueabihf/libnftables.so.1: /usr/local/lib/arm-linux-gnueabihf/libnftnl.so.11: version `LIBNFTNL_14' not found (required by /usr/local/lib/arm-l
inux-gnueabihf/libnftables.so.1)
/usr/local/lib/arm-linux-gnueabihf/libnftables.so.1: /usr/local/lib/arm-linux-gnueabihf/libnftnl.so.11: version `LIBNFTNL_16' not found (required by /usr/local/lib/arm-l
inux-gnueabihf/libnftables.so.1)
        linux-vdso.so.1 (0xbec93000)
        libmnl.so.0 => /usr/local/lib/arm-linux-gnueabihf/libmnl.so.0 (0xb6e5d000)
        libnftnl.so.11 => /usr/local/lib/arm-linux-gnueabihf/libnftnl.so.11 (0xb6e35000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6d3b000)
        /lib/ld-linux-armhf.so.3 (0xb6eed000)

mhm…version seems to be set to 1 not 13+

root@bpi-r2:~# readelf -h /usr/local/lib/arm-linux-gnueabihf/libnftnl.so.11
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           ARM
  Version:                           0x1

recopied the libs to /usr/local/lib/arm-linux-gnueabihf/ followed by calling ldconfig -v and it seems to work

root@bpi-r2:~# mkdir -p /usr/local/lib/arm-linux-gnueabihf/
root@bpi-r2:~# cp /home/frank/libnftnl/install/lib/* /usr/local/lib/arm-linux-gnueabihf/
root@bpi-r2:~# cp /home/frank/nftables/install/lib/* /usr/local/lib/arm-linux-gnueabihf/
root@bpi-r2:~# ldconfig -v
...
root@bpi-r2:~# /home/frank/nftables/install/sbin/nft
/home/frank/nftables/install/sbin/nft: no command specified

strange that my crosscompiled version still reports

nft-nat-flowoffload.nft:5:17-21: Error: syntax error, unexpected flags
                flags offload;
                ^^^^^

have recompiled it (after make clean) and copied again…seems i need to replace libs (previous version was without my patch) too now i can import the ruleset

and yes, it works for r2 too

root@bpi-r2:~# cat /sys/kernel/debug/mtk_ppe/entries
0016a UNB IPv4 5T orig=192.168.0.10:5201->192.168.0.12:50210 new=192.168.0.10:5201->192.168.90.122:50210 eth=46:fb:52:43:15:e2->00:13:77:b7:a7:62 etype=1000 vlan=0,0 ib1=10000092 ib2=007ff020
0111e BND IPv4 5T orig=192.168.90.122:50212->192.168.0.10:5201 new=192.168.0.12:50212->192.168.0.10:5201 eth=08:00:00:00:00:01->08:02:00:00:00:10 etype=0100 vlan=0,0 ib1=21412092 ib2=007ff020
root@bpi-r2:~#

@graphine sent out series with my and your Patch https://patchwork.kernel.org/project/linux-mediatek/list/?series=452805

uploaded compiled versions of nftables to my gdrive https://drive.google.com/drive/folders/1hajKvqQa96WRrAy52fQX90i59I1s0h-i?usp=sharing

Just unpack content of lib-folders to folder /usr/local/lib/arm-linux-gnueabihf on r2 (create first) or /usr/local/lib/aarch64-linux-gnu on r64 and nft binary to folder in PATH (/usr/local/sbin)

frank-w · March 25, 2021, 9:37am

v2 of the series is merged to net-next (5.13):

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/log/

nftables-Patch is also applied

http://git.netfilter.org/nftables/commit/?id=8e6cc9f373854ed580156ec6f01bcd97786fa9f7

“my” kernel-patches (mt7623 support and ipv6 offload) are still open

ipv6 offload seems not working:

2 top most commits break ipv6-nat
without them ipv6 NAT works, but hw flow offload not working

this is my ipv6 ruleset:

flush ruleset
table ip6 filter {
    flowtable f {
        hook ingress priority 1
        devices = { lan3, lan0, wan }
        flags offload;
    }
    chain input {
        type filter hook input priority 0; policy accept;
    }

    chain output {
        type filter hook output priority 0; policy accept;
    }

    chain forward {
        type filter hook forward priority 0; policy accept;
        ip6 nexthdr { tcp, udp } flow add @f
    }
}
table ip6 nat {
    chain post {
        type nat hook postrouting priority 0; policy accept;
        oifname "wan" masquerade
    }

    chain pre {
        type nat hook prerouting priority 0; policy accept;
    }
}

Btw. I did backporting of hnat patches for 5.10 to have it working on last lts kernel:

https://github.com/frank-w/BPI-R2-4.14/tree/5.10-hnat

Maybe someone will try it too

frank-w · March 31, 2021, 9:35am

Seems like graphine do only flow offloading and no ipv6 nat (like i try to test). Current patch for adding ipv6 mangle offload breaks ipv6 nat. Pablo means nat for ipv6 seems not to be supported by driver.

I know “no one needs ipv6 nat at the moment” because it breaks end to end connectivity and is not really needed because there are enough adresses available. But maybe if anyone have only a /64 prefix and wants to use 2 subnets (maybe vlan or any other kind of subnetting) nat may be useful…

At least hw offloading should not break nat functionality…

This is currently the case…@ryder.lee can you address this to the hnat-specialists?

Ryder.Lee · March 31, 2021, 9:57pm

Pablo is the netfilter maintainer and leading member, so this is actually his call though unless you can convince him.

moore · April 1, 2021, 12:31am

NAT was invented to solve the shortage of ipv4 address. I agree Pablo’s opinion that no need to support ipv6 NAT. In my opinion, It is more useful to suppprt MAP-E or MAP-T offload since some operators use those protocols in ipv4 to ipv6 transition stage.

frank-w · April 1, 2021, 1:42pm

What is the preferred way if someone has only a /64 ipv6 but wants more than 1 subnet? That was a possible usecase for nat in ipv6

dangowrt · April 4, 2021, 12:45pm

The solution would be to delegate e.g. /80 to downstream network and use DHCPv6 instead of SLAAC, well, if Android would support that. I’ve also used ND proxy for IPv6 and NAT for IPv4 in that situation, but then you still end up with a single /64. Actually most ISPs I’ve seen until now hand out /56 if requesting that in first place (most of the time that implies having to use your own router handling that DHCPv6 request on the WAN interface as ISP supplied routers often don’t allow to configure the size of the prefix to be requested nor delegating sub-prefixes to multiple local networks)

frank-w · April 4, 2021, 1:19pm

which ND proxy do you use here (and how does this work, i only know the “normal” way to have an own /64 on each subnet and radvd for advertising it)? have not done much with ipv6. i have a /48 over tunnelbroker.net, so i have enough subnets for using radvd…by default somebody get a /64 first and after a while (collecting points) a larger subnet can be “purchased” with these points, like i did some time ago

dangowrt · April 4, 2021, 3:48pm

Oh yeah, I remember playing the HE IPv6 qualification game a long time ago. By now, most commercial private VDSL and DOCSIS connections I’ve seen already provide native IPv6 and usually delegate /64 or even /56 to the CPE.

ND proxy is a Linux kernel feature (like ARP proxy on v4), you can enable that using sysctl. In OpenWrt we got our own IPv6 management stack (as ISC stack would be overkill and dnsmasq doesn’t allow doing IPv6 delegation properly). See

DeadMeat · April 5, 2021, 8:33am

Do you mean sysctl featue e.g net.ipv6.conf.wan.proxy_ndp = 1, and ip -6 neigh add proxy <client_ipv6_addr> dev <wan>?

It’s quite inconvenient to use unless you use static ip for every client, or your ipv6 management tracks any client and it’s ip.

For general purpose linux distro i used ndppd.