BPI-R2 Kernel 4.14 HNAT

rainfall83 · July 13, 2018, 7:10am

Yes, state=UNBIND means it is not leveraging the hardware.
I do not know how to make an entry state=BIND manually.
I only can try to observe network stack to make sure the LAN <–> WAN stream can pass the checking in reasonable nf_hook registered by mtkhnat, and then reach this line.

github.com

frank-w/BPI-R2-4.14/blob/hnat/drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c#L140




	entry.ipv4_hnapt.iblk2.dp = gmac;
	entry.ipv4_hnapt.iblk2.port_mg = 0x3f;
	entry.ipv4_hnapt.iblk2.port_ag = (skb->mark >> 3) & 0x1f;
	if (IS_LAN(dev))
		entry.ipv4_hnapt.iblk2.port_ag += 32;
	entry.bfib1.time_stamp = readl((host->fe_base + 0x0010)) & (0xFFFF);
	entry.ipv4_hnapt.bfib1.ttl = 1;
	entry.ipv4_hnapt.bfib1.cah = 1;
	entry.ipv4_hnapt.bfib1.ka = 1;
	entry.bfib1.state = BIND;


	entry.ipv4_hnapt.sip = foe->ipv4_hnapt.sip;
	entry.ipv4_hnapt.dip = foe->ipv4_hnapt.dip;
	entry.ipv4_hnapt.sport = foe->ipv4_hnapt.sport;
	entry.ipv4_hnapt.dport = foe->ipv4_hnapt.dport;


	memcpy(foe, &entry, sizeof(entry));


	return 0;
}

[Reasonable path]

eth0 --> lanX --> xxx_pre_routing() hook --> Network stack & SW NAT --> xxx_post_routing() hook --> wan ---> eth0 or eth1, depending on dsa setting.

frank-w · July 13, 2018, 3:26pm

In 4.14 (with 2nd gmac) the path imho should be this:

Client => lanx => (eth0) => nfprerouting-hook => sw/hwnat (postrouting) => (eth1) => wan/ppp => internet

reverse way (answer from internet-server):

Server => wan/ppp => (eth1) => sw/hw nat (prerouting) => postrouting => (eth0) => lanx => client

xbgmsharp · July 16, 2018, 1:22pm

I updated the file drivers/net/ethernet/mediatek/mtk_hnat/hnat.c with

cr_set_field(host->ppe_base + PPE_BNDR, BIND_RATE, 0x20);

However now iptables is very slow and all sys entries are still set as UNBIND.

# cat /sys/kernel/debug/hnat/all_entry
(fb8c1c80)0x00072|state=UNBIND|type=IPV4_HNAPT|

Is there anything else require?

frank-w · July 16, 2018, 2:12pm

as i understand @rainfall83 right with this:

then this change is not needed…and all traffic should be bound. i tested only init-procedure (will be called if module loaded), because i don’t know how to debug it deeper

xbgmsharp · July 16, 2018, 3:34pm

OK then i will revert my change.

frank-w · July 17, 2018, 9:46am

i can confirm, that on 4.14-hnat also all entries are UNBIND:

log

root@bpi-r2-ubuntu:~# cat /sys/kernel/debug/hnat/all_entry
(fb8c5180)0x00146|state=UNBIND|type=IPV4_HNAPT|192.168.0.10:53->192.168.0.11:581
99=>0.0.0.0:0->0.0.0.0:0|00:00:00:00:00:00=>00:00:00:00:00:00|etype=0x0000|info1
=0x50000119|info2=0x0|vlan1=0|vlan2=0
(fb8c5880)0x00162|state=UNBIND|type=IPV4_HNAPT|192.168.0.10:53->192.168.0.11:288
5=>0.0.0.0:0->0.0.0.0:0|00:00:00:00:00:00=>00:00:00:00:00:00|etype=0x0000|info1=
0x50000118|info2=0x0|vlan1=0|vlan2=0
(fb8c6a80)0x001aa|state=UNBIND|type=IPV4_HNAPT|192.168.0.10:53->192.168.0.11:417
61=>0.0.0.0:0->0.0.0.0:0|00:00:00:00:00:00=>00:00:00:00:00:00|etype=0x0000|info1
=0x50000119|info2=0x0|vlan1=0|vlan2=0
(fb8cf800)0x003e0|state=UNBIND|type=IPV4_HNAPT|192.168.0.10:53->192.168.0.11:273
96=>0.0.0.0:0->0.0.0.0:0|00:00:00:00:00:00=>00:00:00:00:00:00|etype=0x0000|info1
=0x50000016|info2=0x0|vlan1=0|vlan2=0
(fb8d8e00)0x00638|state=UNBIND|type=IPV4_HNAPT|192.168.0.10:53->192.168.0.11:475
92=>0.0.0.0:0->0.0.0.0:0|00:00:00:00:00:00=>00:00:00:00:00:00|etype=0x0000|info1
=0x50000119|info2=0x0|vlan1=0|vlan2=0
(fb8e1c00)0x00870|state=UNBIND|type=IPV4_HNAPT|192.168.0.10:53->192.168.0.11:788
4=>0.0.0.0:0->0.0.0.0:0|00:00:00:00:00:00=>00:00:00:00:00:00|etype=0x0000|info1=
0x50000019|info2=0x0|vlan1=0|vlan2=0
(fb8e6580)0x00996|state=UNBIND|type=IPV4_HNAPT|52.88.66.112:80->192.168.0.11:456
68=>0.0.0.0:0->0.0.0.0:0|00:00:00:00:00:00=>00:00:00:00:00:00|etype=0x0000|info1
=0x10000018|info2=0x0|vlan1=0|vlan2=0
(fb8e6780)0x0099e|state=UNBIND|type=IPV4_HNAPT|52.88.66.112:80->192.168.0.11:456
64=>0.0.0.0:0->0.0.0.0:0|00:00:00:00:00:00=>00:00:00:00:00:00|etype=0x0000|info1
=0x10000018|info2=0x0|vlan1=0|vlan2=0
(fb8ee700)0x00b9c|state=UNBIND|type=IPV4_HNAPT|52.88.66.112:80->192.168.0.11:456
65=>0.0.0.0:0->0.0.0.0:0|00:00:00:00:00:00=>00:00:00:00:00:00|etype=0x0000|info1
=0x10000019|info2=0x0|vlan1=0|vlan2=0
(fb8f0300)0x00c0c|state=UNBIND|type=IPV4_HNAPT|192.168.0.10:53->192.168.0.11:524
66=>0.0.0.0:0->0.0.0.0:0|00:00:00:00:00:00=>00:00:00:00:00:00|etype=0x0000|info1
=0x50000119|info2=0x0|vlan1=0|vlan2=0
(fb8f4d00)0x00d34|state=UNBIND|type=IPV4_HNAPT|192.168.0.10:53->192.168.0.11:727
8=>0.0.0.0:0->0.0.0.0:0|00:00:00:00:00:00=>00:00:00:00:00:00|etype=0x0000|info1=
0x50000119|info2=0x0|vlan1=0|vlan2=0
(fb8f5300)0x00d4c|state=UNBIND|type=IPV4_HNAPT|192.168.0.10:53->192.168.0.11:523
06=>0.0.0.0:0->0.0.0.0:0|00:00:00:00:00:00=>00:00:00:00:00:00|etype=0x0000|info1
=0x50000119|info2=0x0|vlan1=0|vlan2=0
(fb8f6680)0x00d9a|state=UNBIND|type=IPV4_HNAPT|52.88.66.112:80->192.168.0.11:456
66=>0.0.0.0:0->0.0.0.0:0|00:00:00:00:00:00=>00:00:00:00:00:00|etype=0x0000|info1
=0x10000119|info2=0x0|vlan1=0|vlan2=0
(fb8fe600)0x00f98|state=UNBIND|type=IPV4_HNAPT|52.88.66.112:80->192.168.0.11:456
67=>0.0.0.0:0->0.0.0.0:0|00:00:00:00:00:00=>00:00:00:00:00:00|etype=0x0000|info1
=0x10000118|info2=0x0|vlan1=0|vlan2=0
(fb8ff800)0x00fe0|state=UNBIND|type=IPV4_HNAPT|52.88.66.112:80->192.168.0.11:456
63=>0.0.0.0:0->0.0.0.0:0|00:00:00:00:00:00=>00:00:00:00:00:00|etype=0x0000|info1
=0x10000118|info2=0x0|vlan1=0|vlan2=0

how can we debug this?

dmesg

[  380.129873] [HNAT] hnat_probe: (drivers/net/ethernet/mediatek/mtk_hnat/hnat.c
:252) of_node=hnat
[  380.129900] [HNAT] hnat_probe: (drivers/net/ethernet/mediatek/mtk_hnat/hnat.c
:265) res:-559325376
[  380.129966] [HNAT] hnat_probe: (drivers/net/ethernet/mediatek/mtk_hnat/hnat.c
:273) host-fe_base:-503775232
[  380.130236] [HNAT] hnat_probe: (drivers/net/ethernet/mediatek/mtk_hnat/hnat.c
:281) err (debugfs):0
[  380.133039] [HNAT] hnat_probe: (drivers/net/ethernet/mediatek/mtk_hnat/hnat.c
:293) err (hnat_start):0
[  380.133044] [HNAT] hnat_probe: (drivers/net/ethernet/mediatek/mtk_hnat/hnat.c
:299) register reached
[  380.323695] [HNAT] hnat_probe: (drivers/net/ethernet/mediatek/mtk_hnat/hnat.c
:303) ret (hnat_register):0

As far as i know,traffic is received by hnat-module,but i don’t know why it is unbind…there are several unclear checks (mtk_hnat_nf_post_routing) before bind is set (skb_to_hnat_info)

github.com

frank-w/BPI-R2-4.14/blob/b3ba09aea285ecdac5228805411f1e7a6c68382a/drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c#L194


			return 0;


		entry = &host->foe_table_cpu[skb_hnat_entry(skb)];
		if (entry_hnat_is_bound(entry))
			return 0;


		if (skb_hnat_reason(skb) == HIT_UNBIND_RATE_REACH &&
		    skb_hnat_alg(skb) == 0) {
			if (fn && fn(skb, out))
				return 0;
			skb_to_hnat_info(skb, out, entry);
		}
	}


	return 0;
}


static unsigned int mtk_hnat_nf_pre_routing(void *priv,
					    struct sk_buff *skb,
					    const struct nf_hook_state *state)
{

frank-w · July 18, 2018, 10:13am

i’ve added some printk’s to mtk_hnat_nf_post_routing, and looked which are printed…

first result: they are less than i thought…only 2 entries (i thought that this is done by each packet received from driver)

Jul 18 12:03:25 bpi-r2-ubuntu kernel: [  115.123949] [HNAT] mtk_hnat_nf_post_rou
ting: (drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c:171) ct
Jul 18 12:03:25 bpi-r2-ubuntu kernel: [  115.124015] [HNAT] mtk_hnat_nf_post_rou
ting: (drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c:178) help

my code:

167     ct = nf_ct_get(skb, &ctinfo);
168     if (!ct)
169         return 0;
170 printk(KERN_WARNING "[HNAT] %s: (%s:%i) ct",
171  __FUNCTION__, __FILE__, __LINE__);
172
173     /* rcu_read_lock()ed by nf_hook_slow */
174     help = nfct_help(ct);
175     if (help && rcu_dereference(help->helper))
176         return 0;
177 printk(KERN_WARNING "[HNAT] %s: (%s:%i) help",
178  __FUNCTION__, __FILE__, __LINE__);
179
180     if ((FROM_GE_WAN(skb) || FROM_GE_LAN(skb)) &&
181         skb_hnat_is_hashed(skb) &&
182         (skb_hnat_reason(skb) == HIT_BIND_KEEPALIVE_DUP_OLD_HDR))
183         return -1;
184 printk(KERN_WARNING "[HNAT] %s: (%s:%i) from_wan/lan",
185  __FUNCTION__, __FILE__, __LINE__);

as you see the last message does not came up…so that condition seems to be the Problem…after adding a printk for the check-parts…i get much more messages (also the missing “from_wan/lan”)…seems previous test has cached something on client-side

180 printk(KERN_WARNING "[HNAT] %s: (%s:%i) check: from wan: %i,from lan:%i,skb-hashed:%i,skb-reason:%i",
181  __FUNCTION__, __FILE__, __LINE__,FROM_GE_WAN(skb), FROM_GE_LAN(skb),skb_hnat_is_hashed(skb),skb_hnat_reason(skb));

what i wonder: i did not get any message where from “lan:1”

everytime the “from lan/wan”-message comes up wan=1…maybe its a detection-problem of lan-ports

Jul 18 12:21:24 bpi-r2-ubuntu kernel: [  440.135087] [HNAT] mtk_hnat_nf_post_routing: (drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c:181) check: from wan: 1,from lan:0,skb-hashed:1,skb-reason:14
Jul 18 12:21:24 bpi-r2-ubuntu kernel: [  440.135156] [HNAT] mtk_hnat_nf_post_routing: (drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c:189) from_wan/lan

skb-hashed is mostly 1, and skb-reason is 12,13,14,15, 23 (here all values are 0).

drivers/net/ethernet/mediatek/mtk_hnat/hnat.h:371:#define FROM_GE_LAN(skb)	(HNAT_SKB_CB(skb)->iif == FOE_MAGIC_GE_LAN)
drivers/net/ethernet/mediatek/mtk_hnat/hnat.h:375:#define FOE_MAGIC_GE_LAN	0x7272
drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c:223:		HNAT_SKB_CB(skb)->iif = FOE_MAGIC_GE_LAN;

the flag is set in prerouting-nf-hook:

224     if (IS_WAN(state->in))
225         HNAT_SKB_CB(skb)->iif = FOE_MAGIC_GE_WAN;
226     else if (IS_LAN(state->in))
227         HNAT_SKB_CB(skb)->iif = FOE_MAGIC_GE_LAN;

which leads us to IS_LAN-macro:

drivers/net/ethernet/mediatek/mtk_hnat/hnat.h:402:#define IS_LAN(dev)		(!strncmp(dev->name, "lan", 3))

lol…after i added a printk to pre_routing-function i see my problem:

Jul 18 12:48:11 bpi-r2-ubuntu kernel: [  117.792015] [HNAT] mtk_hnat_nf_pre_routing: (drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c:221) in-device: ap0

i tested only over wifi-device because i need my laptop to build kernel…i think IS_LAN should be changed to include ap0 and wlan* devices or maybe !wan (but how about the unknown-devices - last step in prerouting)…

there are maybe some other problems:

drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c

121     if (IS_LAN(dev))
122         gmac = NR_GMAC1_PORT;
123     else if (IS_WAN(dev))
124         gmac = NR_GMAC2_PORT;

imho i cannot append ap0 to IS_LAN because then traffic is routed to GMAC1_PORT…maybe we need a IS_WIFI

if i test over lan0 i have also BIND and FIN-states

cat /sys/kernel/debug/hnat/all_entry | grep -v 'UNBIIND'
(fb8f6300)0x00d8c|state=FIN|type=IPV4_HNAPT|192.168.4.22:54306->78.46.4.143:443=>192.168.0.11:5306->78.46.4.143:443|a6:eb:a4:f6:6f:a1=>08:00:00:00:00:00|etype=0x8100|info1=0x315102e4|info2=0x3f040|vlan1=1|vlan2=0
(fb8f6980)0x00da6|state=UNBIND|type=IPV4_HNAPT|192.168.4.22:33698->13.32.8.17:443=>192.168.0.11:33698->13.32.8.17:443|a6:eb:a4:f6:6f:a1=>08:00:00:00:00:00|etype=0x8100|info1=0x100000e1|info2=0x3f040|vlan1=1|vlan2=0
(fb8f8b80)0x00e2e|state=BIND|type=IPV4_HNAPT|192.168.4.22:58824->162.125.66.3:443=>192.168.0.11:58824->162.125.66.3:443|a6:eb:a4:f6:6f:a1=>08:00:00:00:00:00|etype=0x8100|info1=0x215182e0|info2=0x3f040|vlan1=1|vlan2=0

also tested with 4.14-hnat-branch…same result

uname -r
4.14.54-bpi-r2-hnat
cat /sys/kernel/debug/hnat/all_entry | grep -v 'UNBIND'
(fb8e9980)0x00a66|state=BIND|type=IPV4_HNAPT|52.88.66.112:80->192.168.0.11:58332=>52.88.66.112:80->192.168.4.22:58332|2a:2b:78:8b:8f:d0=>00:13:77:b7:a7:62|etype=0x8100|info1=0x215180fb|info2=0x83f020|vlan1=2|vlan2=0
(fb8e9b80)0x00a6e|state=FIN|type=IPV4_HNAPT|52.88.66.112:80->192.168.0.11:58328=>52.88.66.112:80->192.168.4.22:58328|2a:2b:78:8b:8f:d0=>00:13:77:b7:a7:62|etype=0x8100|info1=0x315180fa|info2=0x83f020|vlan1=2|vlan2=0
(fb8f9d00)0x00e74|state=FIN|type=IPV4_HNAPT|192.168.4.22:58328->52.88.66.112:80=>192.168.0.11:58328->52.88.66.112:80|7e:81:57:f2:23:c0=>08:00:00:00:00:00|etype=0x8100|info1=0x315180fa|info2=0x3f040|vlan1=1|vlan2=0
(fb8f9f00)0x00e7c|state=BIND|type=IPV4_HNAPT|192.168.4.22:58332->52.88.66.112:80=>192.168.0.11:58332->52.88.66.112:80|7e:81:57:f2:23:c0=>08:00:00:00:00:00|etype=0x8100|info1=0x215180fb|info2=0x3f040|vlan1=1|vlan2=0
....

so i assume that hnat works, @rainfall83 am i right? how can we add this behaviour to ap0 and wlanx?

github.com

frank-w/BPI-R2-4.14/blob/b3ba09aea285ecdac5228805411f1e7a6c68382a/drivers/net/ethernet/mediatek/mtk_hnat/hnat_nf_hook.c#L210




static unsigned int mtk_hnat_nf_pre_routing(void *priv,
					    struct sk_buff *skb,
					    const struct nf_hook_state *state)
{
	if (IS_WAN(state->in))
		HNAT_SKB_CB(skb)->iif = FOE_MAGIC_GE_WAN;
	else if (IS_LAN(state->in))
		HNAT_SKB_CB(skb)->iif = FOE_MAGIC_GE_LAN;
	else if (!IS_BR(state->in))
		HNAT_SKB_CB(skb)->iif = FOE_INVALID;


	return NF_ACCEPT;
}


static unsigned int hnat_get_nexthop(struct sk_buff *skb, const struct net_device *out) {


	u32 nexthop;
	struct neighbour *neigh;
	struct dst_entry *dst = skb_dst(skb);
	struct rtable *rt = (struct rtable *)dst;

so everything that is not wan (dts-setting), lan (first 3 letters from interface name, so also lanbrx) and bridge-device (first 2 letters “br”) get tagging invalid…maybe we can create a additional tag (e.g. FOE_WIFI) and add a IS_WIFI-macro (check if devicename is ap0 or contains “wlan”).

btw. in /proc/interrupts i see no decisive difference between with and without mtkhnat-module

ok, it takes some time…i started a download (ubuntu-image) and after some seconds, the interrupts counting ~4/second…in debugfs i see more bind than unbind-entries

frank-w · July 19, 2018, 9:42am

to get it running for wifi maybe we can change the code to check only for wan (because traffic should not go from wan to wan…and wan is always part of the check):

if (FROM_GE_WAN(skb) || IS_WAN(out)) {

instead of

if ((FROM_GE_WAN(skb) || FROM_GE_LAN(skb)) &&

and

if (FROM_GE_WAN(skb) || IS_WAN(out)) {

instead of

if ((IS_LAN(out) && FROM_GE_WAN(skb)) || (IS_WAN(out) && FROM_GE_LAN(skb))) {

but first i want to merge current version, if someone confirm it is working

maybe then we can merge the block to only check once for wan-port and make additional checks inside

only check for wan seems not to work at least without tagging in mtk_hnat_nf_pre_routing

this is what i tried (no traffic with this patch on top of 4.14-hnat-branch): hnat_test_wifi.diff (2,5 KB) i have bind-entries for ap0-connection, but traffic seems to routed wrong or dropped

frank-w · July 28, 2018, 6:43am

@jackzeng please use this thread if you are working on hnat (wifi/bridge-support).

Jackzeng · July 28, 2018, 8:49am

Ok，frank，I am going to try your repo to see the xhci problem.

Jackzeng · October 10, 2018, 7:46am

Hello, Frank, I’m trying to port Hnat to kernel 4.4, but after I ported one version code, eth will be disabled. do you have any suggestions？

frank-w · October 10, 2018, 10:06am

current hnat-code depends on second gmac. I don’t know if it is implemented same as in 4.9/4.14. also forwarding saves sourceport which needs name lanx or wan (needs dsa-driver).

Why do you put such effort in old kernel 4.4?

can you help me with uboot (mmc-offset) and bluetooth (also not working in 4.4) and wifi (wmt-tools)?

frank-w · October 28, 2018, 12:03pm

What is status of official hnat-patch (new framework)? Last info i have is currently under review…

frank-w · December 18, 2018, 11:56am

@Ryder.Lee can you give me an update on this?

framework should be this: https://www.kernel.org/doc/Documentation/networking/nf_flowtable.txt right? how far is hnat/hwqos implementation (4.19)?

moore · December 29, 2018, 5:35am

Per my understanding, framework support software acceleration and openwrt community enhanced it to support hardware acceleration in openwrt trunk. we should wait until upstream framework support hw acceleration before sending patch, thanks.

frank-w · December 29, 2018, 3:24pm

Can i see the progress anywhere?

moore · December 30, 2018, 1:51am

We are studying upstream framework and just found hnat feature can only be supported in openwrt trunk. After we have overview about framework, we will discuss with netfilter core developer about hw nat plan, and hope mt7623 can be the first platform to support upstream hnat feature, thanks.

cioby23 · March 9, 2019, 10:02pm

Any news about this feature being implemented soon ?

moore · March 9, 2019, 10:44pm

We have sent 3 patches to openwrt and kernel.org to fix mt7621 hnat issues. All patches were merged, but OpenWRT user reported system become unstable and we are trying to find the root cause, thanks.

SamAlex · May 28, 2019, 1:08pm

How many connection supported by hnat? I tests BPI-R2 by ixia, one UDP flow it works good. But I saw only 4 records in debug/hnat/all_entry when run test with 128 UDP flows.