[SOLVED] HQoS for BPI-R4 / MT7988A

To MediaTek & folks:

Your hardware QoS looks good and works great (with my limited tests) until LXC (Linux containers) kicks in. There is a little nagging compatibility issue with LXC in your HQoS supporting codes in PPE.

Whenever lxc-start or lxc-stop is invoked from the command line, your codes spit out this horrible warning (but seems harmless to system stability from my check):

[   13.722547] ------------[ cut here ]------------
[   13.727178] sysctl net/netfilter/nf_conntrack_qos: data points to kernel global data: 0xffffffc080dfa6cc
[   13.736736] WARNING: CPU: 2 PID: 3539 at register_net_sysctl_sz+0xe0/0x140
[   13.831491] CPU: 2 UID: 0 PID: 3539 Comm: lxc-start Tainted: G           O       6.12.55 #0
[   13.839835] Tainted: [O]=OOT_MODULE
[   13.843315] Hardware name: Banana Pi BPI-R4 (2x SFP+) (DT)
[   13.848790] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   13.939903] Call trace:
[   13.942341]  register_net_sysctl_sz+0xe0/0x140
[   13.946777]  nf_conntrack_pernet_init+0x150/0x2c0
[   13.951477]  ops_init+0x70/0x128
[   13.954701]  setup_net+0xc0/0x2f0
[   13.958009]  copy_net_ns+0x13c/0x2bc
[   13.961579]  create_new_namespaces+0x110/0x2c0
[   13.966016]  unshare_nsproxy_namespaces+0x5c/0xa0
[   13.970710]  ksys_unshare+0x11c/0x3b4
[   13.974368]  __arm64_sys_unshare+0x10/0x20
[   13.978458]  invoke_syscall.constprop.0+0x4c/0xd0
[   13.983155]  do_el0_svc+0x3c/0xd0
[   13.986461]  el0_svc+0x18/0x60
[   13.989509]  el0t_64_sync_handler+0x118/0x124
[   13.993858]  el0t_64_sync+0x150/0x154
[   13.997513] ---[ end trace 0000000000000000 ]---

The issue looks like your addition of sysctl variable “nf_conntrack_qos” isn’t namespace compatible. Would be good if you look into this and get it addressed.

My tests done in OpenWrt master branch + your PPE patches.

Cheers

Figured out a fix and answering my own question. Here is the fix:

--- a/net/netfilter/nf_conntrack_standalone.c
+++ b/net/netfilter/nf_conntrack_standalone.c
@@ -1147,6 +1147,7 @@ static int nf_conntrack_standalone_init_sysctl(struct net *net)
 	table[NF_SYSCTL_CT_CHECKSUM].data = &net->ct.sysctl_checksum;
 	table[NF_SYSCTL_CT_LOG_INVALID].data = &net->ct.sysctl_log_invalid;
 	table[NF_SYSCTL_CT_ACCT].data = &net->ct.sysctl_acct;
+	table[NF_SYSCTL_CT_QOS].data = &net->ct.sysctl_qos;
 #ifdef CONFIG_NF_CONNTRACK_EVENTS
 	table[NF_SYSCTL_CT_EVENTS].data = &net->ct.sysctl_events;
 #endif
1 Like

I am facing a similar issue. How did you arrive at this solution, @kvic?

  1. Did you submit your changes upstream?
  2. Does networking within your container work, and what is the output of ip a s in AND outside of the container?

Thanks, mate.

I think your patch is useful because now I get a different trace, see below. I’d like to know your process for generating the patch, thanks.

Trace

Networking is broken in the container - I cannot ping 1.1.1.1 or bing.com.

[60838.058268] ------------[ cut here ]------------                
[60838.062896] sysctl net/netfilter/nf_conntrack_tcp_no_window_check: data points to kernel global data: 0xffffffc081351300           
[60838.073843] WARNING: CPU: 3 PID: 22077 at register_net_sysctl_sz+0x1a4/0x1e8                                                       
[60838.080892] Modules linked in: nf_nat_amanda nf_conntrack_amanda qcserial pppoe ppp_async option l2tp_ppp cdc_mbim wireguard usb_wwan qmi_wwan pptp pppox ppp_mppe ppp_generic nft_fib_inet nf_flow_table_inet mt7996e(O) mt76_connac_lib(O) mt76(O) mac80211(O) libchacha20poly1305 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute chacha_neon ch341 cfg80211(O) cdc_ncm cdc_ether cdc_eem xt_u32 xt_time xt_tcpudp xt_tcpmss xt_string xt_statistic xt_state xt_socket xt_recent xt_quota2(O) xt_quota xt_psd(O) xt_policy xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_lscan(O) xt_limit xt_length2(O) xt_length xt_ipv4options(O) xt_iprange xt_ipp2p(O) xt_iface(O) xt_hl xt_helper xt_hashlimit xt_geoip(O) xt_fuzzy(O) xt_esp xt_ecn xt_dscp xt_coova(O) xt_conntrack xt_connmark xt_connlimit xt_connlabel xt_connbytes xt_condition(O) xt_comment xt_cluster xt_cgroup xt_bpf xt_asn(O) xt_addrtype xt_TRACE xt_TPROXY xt_TEE xt_TCPMSS xt_TARPIT(O) xt_SYSRQ(O) xt_RTPENGINE(O) xt_REDIRECT xt_PROTO(O) xt_NFQUEUE xt_NFLOG                                          [60838.081053]  xt_NETMAP xt_MASQUERADE xt_LUA(O) xt_LOGMARK(O) xt_LOG xt_LED xt_IPMARK(O) xt_HL xt_FLOWOFFLOAD xt_DSCP xt_DHCPMAC(O) xt_DELUDE(O) xt_CT xt_CLASSIFY xt_CHECKSUM xt_CHAOS(O) xt_ACCOUNT(O) usbserial usbnet ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda ts_kmp ts_fsm ts_bm spidev slhc sfp sch_cake rtc_pcf8563 poly1305_neon nft_xfrm nft_tproxy nft_socket nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject_bridge nft_reject nft_redir nft_quota nft_queue nft_numgen nft_nat nft_meta_bridge nft_masq nft_log nft_limit nft_hash nft_fwd_netdev nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_dup_netdev nft_dup_ipv6 nft_dup_ipv4 nft_ct nft_connlimit nft_compat nft_chain_nat nfnetlink_queue nfnetlink_log nfnetlink_cttimeout nfnetlink_cthelper nf_tproxy_ipv6 nf_tproxy_ipv4 nf_tables nf_socket_ipv6 nf_socket_ipv4 nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_rtsp(O) nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_log_syslog                                     [60838.170727]  nf_flow_table_netlink nf_dup_netdev nf_dup_ipv6 nf_dup_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_sane nf_conntrack_rtsp(O) nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast nf_conntrack_bridge nf_conncount netatop(O) mdio_i2c macvlan libcurve25519_generic libchacha iptable_raw iptable_nat iptable_mangle iptable_filter ipt_rpfilter ipt_ah ipt_ECN ip6table_raw ip6t_rpfilter ip_tables ina2xx ebtables ebt_vlan ebt_stp ebt_snat ebt_redirect ebt_pkttype ebt_nflog ebt_mark_m ebt_mark ebt_log ebt_limit ebt_ip6 ebt_ip ebt_dnat ebt_arpreply ebt_arp ebt_among ebt_802_3 compat_xtables(O) compat(O) cdc_wdm cdc_acm br_netfilter at24 arptable_filter arpt_mangle arp_tables crypto_safexcel fuse ntfs3 cls_bpf act_bpf act_connmark sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact sg pwm_fan i2c_gpio i2c_algo_bit gpio_pcf857x                                       [60838.260832]  gpio_pca953x i2c_mux_pca954x i2c_mux_pca9541 i2c_mux_gpio i2c_mux ledtrig_usbport xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_nat nf_nat ip6t_NPT ip6t_rt ip6t_mh ip6t_ipv6header ip6t_hbh ip6t_frag ip6t_eui64 ip6t_ah ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ksmbd nls_ucs2_utils cifs_arc4 asn1_decoder ip_gre gre ifb l2tp_netlink l2tp_core oid_registry netlink_diag ip_tunnel veth tun snd_rawmidi snd_seq_device snd_pcm_oss snd_mixer_oss snd_hwdep snd_compress snd_pcm snd_timer snd soundcore ntfs autofs4 nls_utf8 nls_iso8859_1 nls_cp437 marvell10g marvell vxlan udp_tunnel ip6_udp_tunnel crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg sha512_arm64 sha1_ce cts arc4 uas                                        [60838.350761]  usb_storage                                        [60838.385439] mxl862xx mdio-bus:10 mxl_lan5: resolve link status: system iface=0                                                     [60838.440466]  uleds leds_gpio xhci_plat_hcd xhci_pci xhci_mtk_hcd xhci_hcd nvme nvme_core gpio_button_hotplug(O) vfat fat exfat usbcore usb_common aeon_as21xxx(O) aquantia an8801 mii                 [60838.440494] CPU: 3 PID: 22077 Comm: runc:[1:CHILD] Tainted: G           O       6.6.93 #0                                          [60838.440500] Hardware name: Bananapi BPI-R4-PRO-4E (DT)          [60838.440502] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)                                                        [60838.440507] pc : register_net_sysctl_sz+0x1a4/0x1e8             [60838.440513] lr : register_net_sysctl_sz+0x1a4/0x1e8             [60838.440516] sp : ffffffc0885c3bd0                               [60838.440517] x29: ffffffc0885c3bd0 x28: ffffffc081351300 x27: ffffffc081516fe0                                                      [60838.440523] x26: ffffffc081516fa8 x25: ffffffc080e42dd8 x24: ffffff80d6ee6580                                                      [60838.440527] x23: ffffffc080e35890 x22: ffffff80c5ebf000 x21: 0000000000000020                                                      [60838.440531] x20: 0000000000000022 x19: ffffff80c5ebf800 x18: 00000000000001a2                                                      [60838.528129] x17: 6e696f7020617461 x16: 64203a6b63656863 x15: ffffffc08136cd68                                                      [60838.535253] x14: 00000000000004e6 x13: 000000000000e1a2 x12: 00000000ffffffea                                                      [60838.542377] x11: 000000010000c1a2 x10: ffffffc0813c4d68 x9 : ffffffc08136cba0                                                      [60838.549500] x8 : 0000000000002730 x7 : c00000010000c1a2 x6 : 0000000000000001                                                      [60838.556624] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000                                                      [60838.563747] x2 : ffffff80ff79c6b0 x1 : ffffffc07e466000 x0 : 000000000000006c                                                      [60838.570870] Call trace:                                         [60838.573306]  register_net_sysctl_sz+0x1a4/0x1e8                 [60838.577827]  nf_conntrack_pernet_init+0x16c/0x2e0               [60838.582525]  ops_init+0x3c/0x13c                                [60838.585747]  setup_net+0x110/0x318                              [60838.589141]  copy_net_ns+0xc8/0x228                             [60838.592620]  create_new_namespaces+0x110/0x2ac                  [60838.597059]  unshare_nsproxy_namespaces+0x5c/0xa0               [60838.601754]  ksys_unshare+0x110/0x31c                           [60838.605407]  __arm64_sys_unshare+0x10/0x1c                      [60838.609493]  invoke_syscall.constprop.0+0x4c/0xe0               [60838.614187]  do_el0_svc+0x3c/0xbc                               [60838.617492]  el0_svc+0x18/0x4c                                  [60838.620540]  el0t_64_sync_handler+0x118/0x124                   [60838.624890]  el0t_64_sync+0x150/0x154                           [60838.628544] ---[ end trace 0000000000000000 ]---