[BPI-R2] Kernel 4.14 issues

frank-w · February 5, 2019, 10:03am

Hi,

i currently see panics in kmem_cache_alloc / build_skb in 4.14.95, have not yet figured out why this crash happens or how this can be reproduced. Just to inform you, that not to use this…4.14.78 is working, but not testing between

Zusammenfassung

[ 1851.777568] Internal error: Oops: 5 [#1] SMP ARM
[ 1851.782161] Modules linked in: xt_addrtype veth pppoe pppox ppp_generic slhc sit tunnel4 ip_tunnel brid
ge ip6t_REJECT nf_reject_ipv6 xt_mac ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables i
pt_REJECT nf_reject_ipv4 xt_limit xt_iprange xt_recent xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_m
asquerade_ipv4 xt_nat tun xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_con
ntrack xt_mark iptable_mangle mtk_thermal thermal_sys mt6577_auxadc pwm_mediatek spi_mt65xx evbug mtk_pmic
_keys rtc_mt6397 nvmem_mtk_efuse nfsd ip_tables x_tables ipv6 [last unloaded: mt76]
[ 1851.835334] CPU: 0 PID: 3428 Comm: SoftwareVsyncTh Not tainted 4.14.95-bpi-r2-main #49
[ 1851.843192] Hardware name: Mediatek Cortex-A7 (Device Tree)
[ 1851.848722] task: da788600 task.stack: da77e000
[ 1851.853226] PC is at kmem_cache_alloc+0xd0/0x21c
[ 1851.857807] LR is at 0x6fc3d
[ 1851.860663] pc : [<c028c0f4>] lr : [<0006fc3d>] psr: 20070113
[ 1851.866881] sp : da77fd70 ip : a0070113 fp : da77fda4
[ 1851.872064] r10: 00000001 r9 : de950900 r8 : c08958f0
[ 1851.877249] r7 : de801cc0 r6 : 01080020 r5 : c1226790 r4 : de801cc0
[ 1851.883727] r3 : 00000000 r2 : df5a39d4 r1 : 1e4e9000 r0 : 0006fc3c
[ 1851.890207] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
[ 1851.897290] Control: 10c5387d Table: 9bba006a DAC: 00000051
[ 1851.902992] Process SoftwareVsyncTh (pid: 3428, stack limit = 0xda77e218)
[ 1851.909729] Stack: (0xda77fd70 to 0xda780000)
[ 1851.914054] fd60: 1e4e9000 c019d1cc 20070193 01080020
[ 1851.922175] fd80: 00000080 00000000 de801cc0 01080020 00000000 00000000 da77fdd4 da77fda8
[ 1851.930302] fda0: c08958f0 c028c030 00000000 df59eb9c 01080020 db7b8300 c119f040 db7b8314
[ 1851.938428] fdc0: 0000001c 00000000 da77fe0c da77fdd8 c08b965c c08958a8 da77fdfc da77fde8
[ 1851.946553] fde0: c0149494 db7b8300 db7b8318 00000001 db7b8314 00000003 00000000 ffffe000
[ 1851.954678] fe00: da77fe34 da77fe10 c08be820 c08b961c c11bd17c 00000100 c08be68c db7b8330
[ 1851.962803] fe20: 00000200 00000000 da77fe6c da77fe38 c019bdc4 c08be698 c019db48 df5a4140
[ 1851.970928] fe40: 00000000 2617d280 000001af db7b8330 df59c440 da77fea8 c11bd17c 00000200
[ 1851.979053] fe60: da77fea4 da77fe70 c019c008 c019bd84 df5a4140 c1103c6c 00000000 da77fea4
[ 1851.987179] fe80: df59c440 da77fea4 c1102d00 c10b3440 1e4e9000 00000282 da77feec da77fea8
[ 1851.995304] fea0: c019c158 c019bf20 00000000 df59d5f8 da77ff0c da77fec0 c019ea04 c01af384
[ 1852.003429] fec0: a54fec74 000001af c1102084 00000001 c11bcd08 00000100 00000002 ffffe000
[ 1852.011554] fee0: da77ff4c da77fef0 c0101614 c019c090 00000010 de819000 e1003000 00400040
[ 1852.019679] ff00: c1102d00 00025e29 c0b023a8 0000000a c11f4c40 c1103c6c c10af310 c1102080
[ 1852.027804] ff20: c0186bf8 c10b7b3c 00000000 00000000 00000001 de819000 e1003000 6e5f9d25
[ 1852.035929] ff40: da77ff5c da77ff50 c012c998 c01014f4 da77ff84 da77ff60 c01818f4 c012c8c0
[ 1852.044054] ff60: c11043f8 c1168010 e100200c e1002000 da77ffb0 e1003000 da77ffac da77ff88
[ 1852.052178] ff80: c01014a0 c0181890 b6d8875e 20070030 ffffffff 10c5387d 10c5387d a54fec30
[ 1852.060303] ffa0: 00000000 da77ffb0 c010e614 c0101450 00000b98 000003e8 b6d88688 0002eaba
[ 1852.068427] ffc0: a556b64c a54fec30 b6f86ce8 a54fec70 a54fec44 a54fec30 6e5f9d25 00000000
[ 1852.076551] ffe0: 2d4ca952 a54febf0 b361d339 b6d8875e 20070030 ffffffff 00000000 00000000
[ 1852.084692] [<c028c0f4>] (kmem_cache_alloc) from [<c08958f0>] (__alloc_skb+0x54/0x140)
[ 1852.092568] [<c08958f0>] (__alloc_skb) from [<c08b965c>] (__neigh_notify+0x4c/0xdc)
[ 1852.100181] [<c08b965c>] (__neigh_notify) from [<c08be820>] (neigh_timer_handler+0x194/0x278)
[ 1852.108657] [<c08be820>] (neigh_timer_handler) from [<c019bdc4>] (call_timer_fn+0x4c/0x19c)
[ 1852.116959] [<c019bdc4>] (call_timer_fn) from [<c019c008>] (expire_timers+0xf4/0x170)
[ 1852.124742] [<c019c008>] (expire_timers) from [<c019c158>] (run_timer_softirq+0xd4/0x1cc)
[ 1852.132870] [<c019c158>] (run_timer_softirq) from [<c0101614>] (__do_softirq+0x12c/0x378)
[ 1852.140999] [<c0101614>] (__do_softirq) from [<c012c998>] (irq_exit+0xe4/0x15c)
[ 1852.148266] [<c012c998>] (irq_exit) from [<c01818f4>] (__handle_domain_irq+0x70/0xc4)
[ 1852.156049] [<c01818f4>] (__handle_domain_irq) from [<c01014a0>] (gic_handle_irq+0x5c/0xa0)
[ 1852.164349] [<c01014a0>] (gic_handle_irq) from [<c010e614>] (__irq_usr+0x54/0x80)
[ 1852.171779] Exception stack(0xda77ffb0 to 0xda77fff8)
[ 1852.176795] ffa0: 00000b98 000003e8 b6d88688 0002eaba
[ 1852.184920] ffc0: a556b64c a54fec30 b6f86ce8 a54fec70 a54fec44 a54fec30 6e5f9d25 00000000
[ 1852.193043] ffe0: 2d4ca952 a54febf0 b361d339 b6d8875e 20070030 ffffffff
[ 1852.199615] Code: e121f00c e35a0000 0a000002 e5943014 (e79a3003) 
[ 1852.205765] ---[ end trace eeb67939bd037a21 ]---
[ 1852.210418] Kernel panic - not syncing: Fatal exception in interrupt
[ 1852.216750] CPU2: stopping
[ 1852.219449] CPU: 2 PID: 164 Comm: systemd-journal Tainted: G D 4.14.95-bpi-r2-main #49
[ 1852.228431] Hardware name: Mediatek Cortex-A7 (Device Tree)
[ 1852.233982] [<c0113358>] (unwind_backtrace) from [<c010d580>] (show_stack+0x20/0x24)
[ 1852.241680] [<c010d580>] (show_stack) from [<c0aa8e18>] (dump_stack+0x94/0xa8)
[ 1852.248860] [<c0aa8e18>] (dump_stack) from [<c011124c>] (handle_IPI+0x358/0x378)
[ 1852.256212] [<c011124c>] (handle_IPI) from [<c01014e0>] (gic_handle_irq+0x9c/0xa0)
[ 1852.263734] [<c01014e0>] (gic_handle_irq) from [<c010e24c>] (__irq_svc+0x6c/0x90)
[ 1852.271165] Exception stack(0xdde57eb0 to 0xdde57ef8)
[ 1852.276183] 7ea0: 004e6190 beffffff 00000055 00000010
[ 1852.284308] 7ec0: ffffe000 00000051 004e6180 80808080 deb6d01c fefefeff 00000ff0 dde57f34
[ 1852.292431] 7ee0: 00000fe0 dde57f00 c04a9fd4 c0aa688c 20000013 ffffffff
[ 1852.299004] [<c010e24c>] (__irq_svc) from [<c0aa688c>] (__get_user_4+0x0/0x20)
[ 1852.306177] CPU1: stopping
[ 1852.308869] CPU: 1 PID: 402 Comm: rs:main Q:Reg Tainted: G D 4.14.95-bpi-r2-main #49
[ 1852.317679] Hardware name: Mediatek Cortex-A7 (Device Tree)
[ 1852.323219] [<c0113358>] (unwind_backtrace) from [<c010d580>] (show_stack+0x20/0x24)
[ 1852.330914] [<c010d580>] (show_stack) from [<c0aa8e18>] (dump_stack+0x94/0xa8)
[ 1852.338092] [<c0aa8e18>] (dump_stack) from [<c011124c>] (handle_IPI+0x358/0x378)
[ 1852.345443] [<c011124c>] (handle_IPI) from [<c01014e0>] (gic_handle_irq+0x9c/0xa0)
[ 1852.352964] [<c01014e0>] (gic_handle_irq) from [<c010e24c>] (__irq_svc+0x6c/0x90)
[ 1852.360395] Exception stack(0xdc0d5da0 to 0xdc0d5de8)
[ 1852.365413] 5da0: c1232c48 a0030013 00000007 00004fbe c1232c48 dec3a000 00000000 00000076
[ 1852.373537] 5dc0: 00000fff dcd36400 dc18a076 dc0d5dfc dc0d5e00 dc0d5df0 c052f398 c0ac2f24
[ 1852.381658] 5de0: 80030013 ffffffff
[ 1852.385127] [<c010e24c>] (__irq_svc) from [<c0ac2f24>] (_raw_spin_unlock_irqrestore+0x30/0x34)
[ 1852.393686] [<c0ac2f24>] (_raw_spin_unlock_irqrestore) from [<c052f398>] (uart_write+0x128/0x194)
[ 1852.402505] [<c052f398>] (uart_write) from [<c0514660>] (n_tty_write+0x1c8/0x454)
[ 1852.409942] [<c0514660>] (n_tty_write) from [<c05103c4>] (tty_write+0x19c/0x378)
[ 1852.417296] [<c05103c4>] (tty_write) from [<c029fd28>] (__vfs_write+0x38/0x130)
[ 1852.424563] [<c029fd28>] (__vfs_write) from [<c029ffb4>] (vfs_write+0xb4/0x178)
[ 1852.431829] [<c029ffb4>] (vfs_write) from [<c02a01a8>] (SyS_write+0x4c/0xa0)
[ 1852.438833] [<c02a01a8>] (SyS_write) from [<c0108c80>] (ret_fast_syscall+0x0/0x54)
[ 1852.446352] CPU3: stopping
[ 1852.449046] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G D 4.14.95-bpi-r2-main #49
[ 1852.457338] Hardware name: Mediatek Cortex-A7 (Device Tree)
[ 1852.462878] [<c0113358>] (unwind_backtrace) from [<c010d580>] (show_stack+0x20/0x24)
[ 1852.470574] [<c010d580>] (show_stack) from [<c0aa8e18>] (dump_stack+0x94/0xa8)
[ 1852.477753] [<c0aa8e18>] (dump_stack) from [<c011124c>] (handle_IPI+0x358/0x378)
[ 1852.485103] [<c011124c>] (handle_IPI) from [<c01014e0>] (gic_handle_irq+0x9c/0xa0)
[ 1852.492625] [<c01014e0>] (gic_handle_irq) from [<c010e24c>] (__irq_svc+0x6c/0x90)
[ 1852.500055] Exception stack(0xde947f38 to 0xde947f80)
[ 1852.505071] 7f20: 00000000 000fa8e8
[ 1852.513195] 7f40: 1e51c000 c011f8e0 ffffe000 c1103ccc c1103c6c c11bc85f c0d2c0e0 00000001
[ 1852.521320] 7f60: 00000000 de947f94 de947f98 de947f88 c01098ac c01098b0 60000013 ffffffff
[ 1852.529447] [<c010e24c>] (__irq_svc) from [<c01098b0>] (arch_cpu_idle+0x48/0x4c)
[ 1852.536799] [<c01098b0>] (arch_cpu_idle) from [<c0ac2d88>] (default_idle_call+0x30/0x3c)
[ 1852.544841] [<c0ac2d88>] (default_idle_call) from [<c016e3b0>] (do_idle+0xec/0x150)
[ 1852.552451] [<c016e3b0>] (do_idle) from [<c016e6b0>] (cpu_startup_entry+0x28/0x2c)
[ 1852.559974] [<c016e6b0>] (cpu_startup_entry) from [<c0110c60>] (secondary_start_kernel+0x170/0x194)
[ 1852.568962] [<c0110c60>] (secondary_start_kernel) from [<801018ec>] (0x801018ec)
[ 1852.576318] ---[ end Kernel panic - not syncing: Fatal exception in interrupt

regards Frank

frank-w · February 8, 2019, 6:57am

I have tested 4.14.90…it seems stable. And got answer in mailinglist. versions .92 till .97 should be affected, in .98 this should be fixed. I will update my r2 today

fix-commit: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/commit/?h=linux-4.14.y&id=dc489ad6a2f2fcebdaecb7b8532e0d02d272ac6a

buggy commit is named

net: ipv4: do not handle duplicate fragments as overlapping

buggy versions of 4.19 should be 14-19

cioby23 · February 10, 2019, 10:20pm

Hi Frank,

Can you please enable this option CONFIG_DMI=y in your kernel image ?

I’m trying to obtain some information about the Banana PI R2 device from the DMI table but it seems currently it’s not enabled.

Also is MT76 driver enabled on your 4.19 kernel ? I have an MT7612e card which is not recognized.

frank-w · February 10, 2019, 10:30pm

i have included Mt76-driver in separate 4.19 branch because i had not tested it (had only one card in my main-router). See Mt76 from openwrt

Regarding dmi i try to add it (if it is available for arm)…seems it exists and depends on efi…mhm,we have no efi on arm

Can you give an example for reading out your info so i can test it?

cioby23 · February 11, 2019, 10:15am

Hello,

Regarding the DMI stuff I’m trying to read some information available in the /sys/devices/virtual/dmi/id folder For example this line cat /sys/devices/virtual/dmi/id/product_name

displays the product name of the device (currently I obtain this information fron dmesg output which is not very elegant)

I’m working on a web interface for the BananaPI-R2 and trying to read the system information

frank-w · February 11, 2019, 11:57am

have uploaded a kernel with dmi enabled and mt76-driver: https://drive.google.com/open?id=1MY8owoBdyv060u0QcE0GVC1YEtIPgu5E

please test and give feedback

cioby23 · February 11, 2019, 1:13pm

Hi I’ll test later today. How can I install this tar.gz archive kernel image ? Previously i have used the generated debs from your github repo.

frank-w · February 11, 2019, 1:14pm

unpack it and you have 2 folders, BPI-BOOT and BPI-ROOT, simply copy content of it to the matching partition

Automatic deb-creation is only activated for main-branches,to have not all branches in releases-list

btw. 4.14.98 is stable last 3 days…

cioby23 · February 11, 2019, 4:48pm

I have unpacked both folders as recommended and changed the kernel image in uEnv.txt to uImage_nodt and after rebooting it gets stuck at Starting kernel …

Below is the output from the serial console

bootargs=board=bpi-r2 console=earlyprintk console=tty1 fbcon=map:0 console=ttyS0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait vmalloc=496M debug=7 initcall_d4 7709248 bytes read in 386 ms (19 MiB/s)

Booting kernel from Legacy Image at 80200000 …

Image Name: Linux Kernel 4.19.20-dmi Image Type: ARM Linux Kernel Image (uncompressed) Data Size: 7709184 Bytes = 7.4 MiB Load Address: 80008000 Entry Point: 80008000 Verifying Checksum … OK Loading Kernel Image … OK

Starting kernel …

frank-w · February 11, 2019, 4:51pm

you have to use the uImage-file (without _nodt), the uImage_nodt has no appended devicetree so you have to load it manually (needed for dt-overlays, requires new uboot and a changed boot-command in uboot)

cioby23 · February 11, 2019, 5:35pm

OK I have booted with the new uImage and it loaded fine.

The mt76 driver was automatically loaded and I prepared a basic hostapd config. I can see the SSID on my phone and it gets stuck at the Obtaining an IP address. I need to check what’s wrong with DNSmasq.

For the other part DMI it does not work at all. This is the message I get from dmesg

[ 0.420969] DMI not present or invalid.

Also the folder /sys/devices/virtual/dmi is not present on the system. Maybe this is related to EFI

This is what I found in the dmesg logs.

[ 0.000000] efi: Getting EFI parameters from FDT:

[ 0.000000] efi: UEFI not found.

[ 0.009215] EFI services will not be available.

frank-w · February 11, 2019, 5:48pm

As i said,i don’t know if dmi/efi works on arm because you have no efi-bios on it.

For wifi make sure you have defined ip-address+netmask for wifi-device (maybe after starting hostapd) and in dnsmasq listen to this device and dhcp-range has to be in same subnet as device-ip

Maybe you can read out devicetree (sysfs or via dtc) to get hardware-specific information: https://unix.stackexchange.com/questions/265890/is-it-possible-to-get-the-information-for-a-device-tree-using-sys-of-a-running

cioby23 · February 11, 2019, 6:53pm

Yes if I run

cat /sys/firmware/devicetree/base/model

it returns the model --> Bananapi BPI-R2

Regarding mt76 driver I managed to connect to the SSID and obtain an IP address. I run some basic speed tests and so far it works fine.

frank-w · February 17, 2019, 1:32pm

what have you changed to get an IP-address? i also configured dnsmasq on my test-device,

configured wlan1-address
enabled wlan1 to dnsmasq listen to
configured dhcp-range
configured gateway

but i receive no dhcp-requests (in syslog)

Jan 19 20:23:35 bpi-r2 systemd[1]: Starting dnsmasq - A lightweight DHCP and caching DNS server...                                                           
Jan 19 20:23:35 bpi-r2 dnsmasq[1812]: dnsmasq: Syntaxpr�üfung OK.                                                                                            
Jan 19 20:23:35 bpi-r2 dnsmasq[1820]: gestartet, Version 2.76, Cachegr�ö�ße 150                                                                              
Jan 19 20:23:35 bpi-r2 dnsmasq[1820]: �Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect in
otify                                                                                                                                                        
Jan 19 20:23:35 bpi-r2 dnsmasq-dhcp[1820]: DHCP, IP-Bereich 10.0.3.100 -- 10.0.3.150, Lease Zeit2d                                                           
Jan 19 20:23:35 bpi-r2 dnsmasq-dhcp[1820]: DHCP, IP-Bereich 192.168.111.100 -- 192.168.111.150, Lease Zeit2d                                                 
Jan 19 20:23:35 bpi-r2 dnsmasq[1820]: lese /etc/resolv.conf                                                                                                  
Jan 19 20:23:35 bpi-r2 dnsmasq[1820]: Benutze Namensserver 192.168.0.10#53                                                                                   
Jan 19 20:23:35 bpi-r2 dnsmasq[1820]: /etc/hosts gelesen - 4 Adressen                                                                                        
Jan 19 20:23:35 bpi-r2 systemd[1]: Started dnsmasq - A lightweight DHCP and caching DNS server. 

root@bpi-r2:~# ip addr show wlan1                                                                                                                            
10: wlan1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000                                                     
    link/ether f8:62:aa:50:15:c8 brd ff:ff:ff:ff:ff:ff                                                                                                       
    inet 192.168.111.1/24 brd 192.168.111.255 scope global wlan1                                                                                             
       valid_lft forever preferred_lft forever

client is waiting for IP-address and hostapd prints this:

Association Response to the STA has already been sent, but no TX status yet known - ignore Class 3 frame issue with <mac>

it seems that this is a problem with the openwrt-mt76-driver…reverted back to default 4.19-driver