BPI-R2 Kernel 4.14 HNAT

(Frank W.) #35

as rainfall83 wrote, change following line

in hnat.c

(rainfall83) #36

As my testing result:

  1. When this setting is set to ‘1’, every LAN <–> WAN stream will be bound.
  2. When this setting is set to ‘0x20’, the traffic only exchanging few packets will not be bound. For example, control stream of iperf3 will not be bound. Everyone who wants above behavior can change the value.

@frank-w I refer to the patches in kernel 4.9 from graywang’s github to port hnat to kernel4.4 with 1cpu port dsa. I am interested in how does it look like in kernel 4.14 with 2cpu port dsa, so I check some points and put the reference from your github in this thread.

(Frank W.) #37

Then in my 4.9 (was native 4.9.44 with garys lede patches) seems to not working as stated by xbgmsharp.this 4.9-branch contains also second gmac-patches from garys repo…

my 4.14-hnat uses same code only ported to newer iptables callbacks

(rainfall83) #38

Yes, state=UNBIND means it is not leveraging the hardware.
I do not know how to make an entry state=BIND manually.
I only can try to observe network stack to make sure the LAN <–> WAN stream can pass the checking in reasonable nf_hook registered by mtkhnat, and then reach this line.

[Reasonable path]

eth0 --> lanX --> xxx_pre_routing() hook --> Network stack & SW NAT --> xxx_post_routing() hook --> wan ---> eth0 or eth1, depending on dsa setting.

(Frank W.) #39

In 4.14 (with 2nd gmac) the path imho should be this:

Client => lanx => (eth0) => nfprerouting-hook => sw/hwnat (postrouting) => (eth1) => wan/ppp => internet

reverse way (answer from internet-server):

Server => wan/ppp => (eth1) => sw/hw nat (prerouting) => postrouting => (eth0) => lanx => client

(xbgmsharp) #40

I updated the file drivers/net/ethernet/mediatek/mtk_hnat/hnat.c with

cr_set_field(host->ppe_base + PPE_BNDR, BIND_RATE, 0x20);

However now iptables is very slow and all sys entries are still set as UNBIND.

# cat /sys/kernel/debug/hnat/all_entry

Is there anything else require?

(Frank W.) #41

as i understand @rainfall83 right with this:

then this change is not needed…and all traffic should be bound. i tested only init-procedure (will be called if module loaded), because i don’t know how to debug it deeper

(xbgmsharp) #42

OK then i will revert my change.

(Frank W.) #43

i can confirm, that on 4.14-hnat also all entries are UNBIND:

root@bpi-r2-ubuntu:~# cat /sys/kernel/debug/hnat/all_entry

how can we debug this?

[  380.129873] [HNAT] hnat_probe: (drivers/net/ethernet/mediatek/mtk_hnat/hnat.c
:252) of_node=hnat
[  380.129900] [HNAT] hnat_probe: (drivers/net/ethernet/mediatek/mtk_hnat/hnat.c
:265) res:-559325376
[  380.129966] [HNAT] hnat_probe: (drivers/net/ethernet/mediatek/mtk_hnat/hnat.c
:273) host-fe_base:-503775232
[  380.130236] [HNAT] hnat_probe: (drivers/net/ethernet/mediatek/mtk_hnat/hnat.c
:281) err (debugfs):0
[  380.133039] [HNAT] hnat_probe: (drivers/net/ethernet/mediatek/mtk_hnat/hnat.c
:293) err (hnat_start):0
[  380.133044] [HNAT] hnat_probe: (drivers/net/ethernet/mediatek/mtk_hnat/hnat.c
:299) register reached
[  380.323695] [HNAT] hnat_probe: (drivers/net/ethernet/mediatek/mtk_hnat/hnat.c
:303) ret (hnat_register):0

As far as i know,traffic is received by hnat-module,but i don’t know why it is unbind…there are several unclear checks (mtk_hnat_nf_post_routing) before bind is set (skb_to_hnat_info)

(Frank W.) #44

i’ve added some printk’s to mtk_hnat_nf_post_routing, and looked which are printed…

first result: they are less than i thought…only 2 entries (i thought that this is done by each packet received from driver)

Jul 18 12:03:25 bpi-r2-ubuntu kernel: [  115.123949] [HNAT] mtk_hnat_nf_post_rou
ting: (drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c:171) ct
Jul 18 12:03:25 bpi-r2-ubuntu kernel: [  115.124015] [HNAT] mtk_hnat_nf_post_rou
ting: (drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c:178) help

my code:

167     ct = nf_ct_get(skb, &ctinfo);
168     if (!ct)
169         return 0;
170 printk(KERN_WARNING "[HNAT] %s: (%s:%i) ct",
171  __FUNCTION__, __FILE__, __LINE__);
173     /* rcu_read_lock()ed by nf_hook_slow */
174     help = nfct_help(ct);
175     if (help && rcu_dereference(help->helper))
176         return 0;
177 printk(KERN_WARNING "[HNAT] %s: (%s:%i) help",
178  __FUNCTION__, __FILE__, __LINE__);
180     if ((FROM_GE_WAN(skb) || FROM_GE_LAN(skb)) &&
181         skb_hnat_is_hashed(skb) &&
182         (skb_hnat_reason(skb) == HIT_BIND_KEEPALIVE_DUP_OLD_HDR))
183         return -1;
184 printk(KERN_WARNING "[HNAT] %s: (%s:%i) from_wan/lan",
185  __FUNCTION__, __FILE__, __LINE__);

as you see the last message does not came up…so that condition seems to be the Problem…after adding a printk for the check-parts…i get much more messages (also the missing “from_wan/lan”)…seems previous test has cached something on client-side

180 printk(KERN_WARNING "[HNAT] %s: (%s:%i) check: from wan: %i,from lan:%i,skb-hashed:%i,skb-reason:%i",
181  __FUNCTION__, __FILE__, __LINE__,FROM_GE_WAN(skb), FROM_GE_LAN(skb),skb_hnat_is_hashed(skb),skb_hnat_reason(skb));

what i wonder: i did not get any message where from “lan:1”

everytime the “from lan/wan”-message comes up wan=1…maybe its a detection-problem of lan-ports

Jul 18 12:21:24 bpi-r2-ubuntu kernel: [  440.135087] [HNAT] mtk_hnat_nf_post_routing: (drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c:181) check: from wan: 1,from lan:0,skb-hashed:1,skb-reason:14
Jul 18 12:21:24 bpi-r2-ubuntu kernel: [  440.135156] [HNAT] mtk_hnat_nf_post_routing: (drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c:189) from_wan/lan

skb-hashed is mostly 1, and skb-reason is 12,13,14,15, 23 (here all values are 0).

drivers/net/ethernet/mediatek/mtk_hnat/hnat.h:371:#define FROM_GE_LAN(skb)	(HNAT_SKB_CB(skb)->iif == FOE_MAGIC_GE_LAN)
drivers/net/ethernet/mediatek/mtk_hnat/hnat.h:375:#define FOE_MAGIC_GE_LAN	0x7272
drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c:223:		HNAT_SKB_CB(skb)->iif = FOE_MAGIC_GE_LAN;

the flag is set in prerouting-nf-hook:

224     if (IS_WAN(state->in))
225         HNAT_SKB_CB(skb)->iif = FOE_MAGIC_GE_WAN;
226     else if (IS_LAN(state->in))
227         HNAT_SKB_CB(skb)->iif = FOE_MAGIC_GE_LAN;

which leads us to IS_LAN-macro:

drivers/net/ethernet/mediatek/mtk_hnat/hnat.h:402:#define IS_LAN(dev)		(!strncmp(dev->name, "lan", 3))

lol…after i added a printk to pre_routing-function i see my problem:

Jul 18 12:48:11 bpi-r2-ubuntu kernel: [  117.792015] [HNAT] mtk_hnat_nf_pre_routing: (drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c:221) in-device: ap0

i tested only over wifi-device :wink: because i need my laptop to build kernel…i think IS_LAN should be changed to include ap0 and wlan* devices or maybe !wan (but how about the unknown-devices - last step in prerouting)…

there are maybe some other problems:


121     if (IS_LAN(dev))
122         gmac = NR_GMAC1_PORT;
123     else if (IS_WAN(dev))
124         gmac = NR_GMAC2_PORT;

imho i cannot append ap0 to IS_LAN because then traffic is routed to GMAC1_PORT…maybe we need a IS_WIFI :slight_smile:

if i test over lan0 i have also BIND and FIN-states :slight_smile:

cat /sys/kernel/debug/hnat/all_entry | grep -v 'UNBIIND'

also tested with 4.14-hnat-branch…same result

uname -r
cat /sys/kernel/debug/hnat/all_entry | grep -v 'UNBIND'

so i assume that hnat works, @rainfall83 am i right? how can we add this behaviour to ap0 and wlanx?

so everything that is not wan (dts-setting), lan (first 3 letters from interface name, so also lanbrx) and bridge-device (first 2 letters “br”) get tagging invalid…maybe we can create a additional tag (e.g. FOE_WIFI) and add a IS_WIFI-macro (check if devicename is ap0 or contains “wlan”).

btw. in /proc/interrupts i see no decisive difference between with and without mtkhnat-module

ok, it takes some time…i started a download (ubuntu-image) and after some seconds, the interrupts counting ~4/second…in debugfs i see more bind than unbind-entries

(Frank W.) #45

to get it running for wifi maybe we can change the code to check only for wan (because traffic should not go from wan to wan…and wan is always part of the check):

if (FROM_GE_WAN(skb) || IS_WAN(out)) {

instead of

if ((FROM_GE_WAN(skb) || FROM_GE_LAN(skb)) &&


if (FROM_GE_WAN(skb) || IS_WAN(out)) {

instead of

if ((IS_LAN(out) && FROM_GE_WAN(skb)) || (IS_WAN(out) && FROM_GE_LAN(skb))) {

but first i want to merge current version, if someone confirm it is working

maybe then we can merge the block to only check once for wan-port and make additional checks inside

only check for wan seems not to work at least without tagging in mtk_hnat_nf_pre_routing

this is what i tried (no traffic with this patch on top of 4.14-hnat-branch): hnat_test_wifi.diff (2,5 KB) i have bind-entries for ap0-connection, but traffic seems to routed wrong or dropped

(Frank W.) #46

@jackzeng please use this thread if you are working on hnat (wifi/bridge-support).

(ZB) #47

Ok,frank,I am going to try your repo to see the xhci problem.

(ZB) #48

Hello, Frank, I’m trying to port Hnat to kernel 4.4, but after I ported one version code, eth will be disabled. do you have any suggestions?

(Frank W.) #49

current hnat-code depends on second gmac. I don’t know if it is implemented same as in 4.9/4.14. also forwarding saves sourceport which needs name lanx or wan (needs dsa-driver).

Why do you put such effort in old kernel 4.4?

can you help me with uboot (mmc-offset) and bluetooth (also not working in 4.4) and wifi (wmt-tools)?

(Frank W.) #50

What is status of official hnat-patch (new framework)? Last info i have is currently under review…

(Frank W.) #51

@Ryder.Lee can you give me an update on this?

framework should be this: https://www.kernel.org/doc/Documentation/networking/nf_flowtable.txt right? how far is hnat/hwqos implementation (4.19)?

(moore liu) #52

Per my understanding, framework support software acceleration and openwrt community enhanced it to support hardware acceleration in openwrt trunk. we should wait until upstream framework support hw acceleration before sending patch, thanks.

(Frank W.) #53

Can i see the progress anywhere?

(moore liu) #54

We are studying upstream framework and just found hnat feature can only be supported in openwrt trunk. After we have overview about framework, we will discuss with netfilter core developer about hw nat plan, and hope mt7623 can be the first platform to support upstream hnat feature, thanks.