[BPI-R64] PCIe issues

Banana Pi BPI-R4/BPI-R3 Mini/BPI-R3/BPI-R64/BPI-R2
1

Hi,

as described here we have issues getting second pcie slot (CN8) working. i tried now my wifi-card in the other slot and loaded the ath10 kernel driver and it crashes:

details 
[   10.459496] ath10k_pci 0000:01:00.0: assign IRQ: got 140                     
[   10.481601] pci 0000:00:00.0: enabling device (0000 -> 0002)           
[   10.488761] pci 0000:00:00.0: enabling bus mastering         
[   10.494914] ath10k_pci 0000:01:00.0: enabling device (0000 -> 0002)
[   10.502656] ath10k_pci 0000:01:00.0: enabling bus mastering                  
[   10.509667] Unable to handle kernel paging request at virtual address 0000000
[   10.517607] Mem abort info:                                                  
[   10.520450]   ESR = 0x96000005                                               
[   10.523529]   EC = 0x25: DABT (current EL), IL = 32 bits                     
[   10.528855]   SET = 0, FnV = 0                                               
[   10.531917]   EA = 0, S1PTW = 0                                              
[   10.535066] Data abort info:                                                 
[   10.537939]   ISV = 0, ISS = 0x00000005                                      
[   10.541781]   CM = 0, WnR = 0                                                
[   10.544759] user pgtable: 4k pages, 39-bit VAs, pgdp=0000000079416000        
[   10.551209] [0000000400000040] pgd=0000000000000000, pud=0000000000000000    
[   10.558010] Internal error: Oops: 96000005 [#1] PREEMPT SMP                  
[   10.563576] Modules linked in: ath10k_pci(+) ath10k_core mt7622(+) ath mt76 t
[   10.573673] CPU: 0 PID: 123 Comm: systemd-udevd Not tainted 5.4.0-r64-main #2
[   10.580886] Hardware name: Bananapi BPI-R64 (DT)                             
[   10.585496] pstate: 00000005 (nzcv daif -PAN -UAO)                           
[   10.590287] pc : mutex_can_spin_on_owner+0x30/0x5c                           
[   10.595070] lr : mutex_can_spin_on_owner+0x24/0x5c                           
[   10.599851] sp : ffffffc010ceb580                                            
[   10.603156] x29: ffffffc010ceb580 x28: 0000000000080000                      
[   10.608461] x27: ffffffc010ceb838 x26: 0000000000000001                      
[   10.613766] x25: 000000000000008d x24: 0000000000000002                      
[   10.619071] x23: 000000000000008d x22: ffffff803e155500                      
[   10.624375] x21: ffffff803dc0bc00 x20: ffffffc010838000                      
[   10.629680] x19: ffffff803e155500 x18: 000000000000000a                      
[   10.634984] x17: 0000000000000000 x16: 0000000000000000                      
[   10.640289] x15: 000000000000008d x14: ffffff8039792604                      
[   10.645593] x13: ffffffffffffffff x12: 0000000000000010                      
[   10.650898] x11: 0101010101010101 x10: ffffff803dc0bc68                      
[   10.656203] x9 : 0000000000080000 x8 : ffffff803dc0bc60                      
[   10.661507] x7 : ffffff803e155500 x6 : ffffff800336a400                      
[   10.666811] x5 : 0000000400000003 x4 : 0000000400000003                      
[   10.672116] x3 : ffffffc0108f6000 x2 : ffffff800336a400                      
[   10.677420] x1 : ffffff800336a400 x0 : 0000000400000000                      
[   10.682726] Call trace:                                                      
[   10.685169]  mutex_can_spin_on_owner+0x30/0x5c                               
[   10.689608]  __mutex_lock.isra.9+0x58/0x2a4                                  
[   10.693784]  __mutex_lock_slowpath+0x10/0x18                                 
[   10.698047]  mutex_lock+0x44/0x68                                            
[   10.701358]  mtk_pcie_irq_domain_alloc+0x38/0xc8                             
[   10.705970]  irq_domain_alloc_irqs_hierarchy+0x14/0x1c                       
[   10.711100]  irq_domain_alloc_irqs_parent+0x14/0x24                          
[   10.715970]  msi_domain_alloc+0x90/0x130                                     
[   10.719886]  irq_domain_alloc_irqs_hierarchy+0x14/0x1c                       
[   10.725017]  __irq_domain_alloc_irqs+0x140/0x2b4                             
[   10.729626]  msi_domain_alloc_irqs+0x134/0x2c4                               
[   10.734063]  pci_msi_setup_msi_irqs+0x28/0x38                                
[   10.738412]  __pci_enable_msi_range+0x208/0x30c                              
[   10.742935]  pci_enable_msi+0x18/0x28                                        
[   10.746604]  ath10k_pci_probe+0x50c/0x6d8 [ath10k_pci]                       
[   10.751739]  pci_device_probe+0xb4/0x144                                     
[   10.755658]  really_probe+0x238/0x3f8                                        
[   10.759314]  driver_probe_device+0x114/0x124                                 
[   10.763577]  device_driver_attach+0x40/0x68                                  
[   10.767753]  __driver_attach+0x134/0x138                                     
[   10.771668]  bus_for_each_dev+0x78/0xbc                                      
[   10.775498]  driver_attach+0x20/0x28                                         
[   10.779066]  bus_add_driver+0x1a8/0x1ec                                      
[   10.782894]  driver_register+0xac/0xe4                                       
[   10.786638]  __pci_register_driver+0x40/0x48                                 
[   10.790913]  ath10k_pci_init+0x28/0x1000 [ath10k_pci]                        
[   10.795959]  do_one_initcall+0x74/0x178                                      
[   10.799790]  do_init_module+0x58/0x2fc                                       
[   10.803533]  load_module+0x113c/0x1608                                       
[   10.807276]  __do_sys_finit_module+0xd0/0xf0                                 
[   10.811539]  __arm64_sys_finit_module+0x18/0x20                              
[   10.816063]  el0_svc_common.constprop.1+0xfc/0x168                           
[   10.820846]  el0_svc_handler+0x44/0x70                                       
[   10.824586]  el0_svc+0x8/0xc                                                 
[   10.827465] Code: 94005b39 f9400260 f27df000 54000080 (b9404013)             
[   10.833552] ---[ end trace 3741f6ce457a2bec ]---

has anyone an idea if the problem is in pcie-driver/tphy or in ath10k?

used Kernel 5.4.0-r64-main, as i get similar crash on 4.19 i guess it’s also a pcie driver issue

[    5.870751] Call trace:
[    5.873272]  __mutex_lock.isra.1+0x238/0x498
[    5.877672]  __mutex_lock_slowpath+0x10/0x18
[    5.882070]  mutex_lock+0x2c/0x34
[    5.885486]  mtk_pcie_irq_domain_alloc+0x38/0xc8

@sinovoip @ryder.lee @moore can you please try to use pcie-card on both slots with device driver (e.g. wifi driver) so that card is fully initialized and not only recognized with lspci?

as far as i currently know crash happens in mtk_pcie_irq_domain_alloc (drivers/pci/controller/pcie-mediatek.c) on try (or directly after) setting mutex-lock, port is not NULL

[   11.530288] DEBUG: Passed mtk_pcie_irq_domain_alloc 441 port:0x00000000b3aaf77c                                                                 
[   11.537629] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000260    
....
mtk_pcie_irq_domain_alloc+0x5c                                        

(gdb) list *(mtk_pcie_irq_domain_alloc+0x5c)                                                                                                       
0xffffffc0102e68cc is in mtk_pcie_irq_domain_alloc (drivers/pci/controller/pcie-mediatek.c:447).                                                   
442                                                                                                                                                
443             WARN_ON(nr_irqs != 1);                                                                                                             
444             mutex_lock(&port->lock);                                                                                                           
445                                                                                                                                                
446                                                                                                                                                
447             printk(KERN_ALERT "DEBUG: Passed %s %d\n",__FUNCTION__,__LINE__);                                                                  
448             bit = find_first_zero_bit(port->msi_irq_in_use, MTK_MSI_IRQS_NUM);

printk on 447 is not printed and i have no pointer dereference so we still in mutex_lock, i guess port->lock is not yet initialized…this is done in mtk_pcie_allocate_msi_domains which is done before (i had checked)…so i don’t understand the crash till now

i found out that the mutex is already locked…

if (mutex_is_locked(&port->lock)) printk(KERN_ALERT "DEBUG: %s mutex already locked\n",__FUNCTION__); //before mutex_lock()

[   11.395077] DEBUG: mtk_pcie_irq_domain_alloc mutex already locked

so i added an mutex_unlock() in the condition and got no crash on bootup (but it seems that this function is not executed further), and later a rcu_preempt self-detected stall happens

log 
[  788.702325] rcu: INFO: rcu_preempt self-detected stall on CPU
[  788.708075] rcu:     1-....: (193565 ticks this GP) idle=a7e/1/0x4000000000000002 softirq=5105/5105 fqs=96767 
[  788.717806]  (t=194286 jiffies g=805 q=7450)
[  788.722068] Task dump for CPU 1:
[  788.725288] systemd-udevd   R  running task        0   122    119 0x0000002b
[  788.732331] Call trace:
[  788.734773]  dump_backtrace+0x0/0x160
[  788.738428]  show_stack+0x14/0x1c
[  788.741738]  sched_show_task+0xf8/0x130
[  788.745566]  dump_cpu_task+0x40/0x114
[  788.749222]  rcu_dump_cpu_stacks+0xc8/0xd4
[  788.753310]  rcu_sched_clock_irq+0x31c/0x7d4
[  788.757574]  update_process_times+0x2c/0x50
[  788.761750]  tick_sched_handle.isra.12+0x3c/0x44
[  788.766360]  tick_sched_timer+0x54/0x94
[  788.770187]  __hrtimer_run_queues+0xe4/0x13c
[  788.774448]  hrtimer_interrupt+0xb8/0x1c0
[  788.778452]  arch_timer_handler_phys+0x28/0x3c
[  788.782893]  handle_percpu_devid_irq+0x58/0xf8
[  788.787328]  generic_handle_irq+0x18/0x2c
[  788.791330]  __handle_domain_irq+0x94/0x98
[  788.795418]  gic_handle_irq+0x70/0xac
[  788.799072]  el1_irq+0xb8/0x180
[  788.802207]  __cmpwait_case_32+0x18/0x1c
[  788.806121]  do_raw_spin_lock+0x48/0x6c
[  788.809952]  _raw_spin_lock+0x20/0x2c
[  788.813608]  __mutex_unlock_slowpath.isra.19+0x70/0x114
[  788.818825]  mutex_unlock+0x2c/0x34
[  788.822307]  mtk_pcie_irq_domain_alloc+0x7c/0x148
[  788.827004]  irq_domain_alloc_irqs_hierarchy+0x14/0x1c
[  788.832134]  irq_domain_alloc_irqs_parent+0x14/0x24
[  788.837005]  msi_domain_alloc+0x90/0x130
[  788.840919]  irq_domain_alloc_irqs_hierarchy+0x14/0x1c
[  788.846050]  __irq_domain_alloc_irqs+0x140/0x2b4
[  788.850658]  msi_domain_alloc_irqs+0x134/0x2c4
[  788.855094]  pci_msi_setup_msi_irqs+0x28/0x38
[  788.859443]  __pci_enable_msi_range+0x208/0x30c
[  788.863966]  pci_enable_msi+0x18/0x28
[  788.867633]  ath10k_pci_probe+0x50c/0x6d8 [ath10k_pci]
[  788.872765]  pci_device_probe+0xb4/0x144
[  788.876682]  really_probe+0x238/0x3f8
[  788.880336]  driver_probe_device+0x114/0x124
[  788.884598]  device_driver_attach+0x40/0x68
[  788.888774]  __driver_attach+0x134/0x138
[  788.892688]  bus_for_each_dev+0x78/0xbc
[  788.896515]  driver_attach+0x20/0x28
[  788.900082]  bus_add_driver+0x1a8/0x1ec
[  788.903911]  driver_register+0xac/0xe4
[  788.907652]  __pci_register_driver+0x40/0x48
[  788.911920]  ath10k_pci_init+0x28/0x1000 [ath10k_pci]
[  788.916964]  do_one_initcall+0x74/0x178
[  788.920793]  do_init_module+0x58/0x2fc
[  788.924535]  load_module+0x113c/0x1608
[  788.928277]  __do_sys_finit_module+0xd0/0xf0
[  788.932538]  __arm64_sys_finit_module+0x18/0x20
[  788.937061]  el0_svc_common.constprop.1+0xfc/0x168
[  788.941843]  el0_svc_handler+0x44/0x70
[  788.945583]  el0_svc+0x8/0xc
[  795.618344] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 1-... } 195971 jiffies s: 21 root: 0x2/.
[  795.629067] rcu: blocking rcu_node structures:
[  795.634072] Task dump for CPU 1:
[  795.637567] systemd-udevd   R  running task        0   122    119 0x0000002b
[  795.644842] Call trace:
[  795.647530]  __switch_to+0xcc/0x118
[  795.651238]  0xffffffc010808960

after poweroff and some reboots this seems not happen anymore…but strange why mutex is locked on entering mtk_pcie_irq_domain_alloc

mutex_lock is only used in mtk_pcie_irq_domain_alloc / mtk_pcie_irq_domain_free and there is no ovious problem with missing unlock, mutex_init initializes to unlocked state…so it lookes the mutex is locked anywhere else…but struct holding the lock-pointer is defined in pcie-mediatek.c so imho it can’t be used anywhere else

2

can you sand a email with the descriptions to [email protected]?

3

Hi,

I am facing a similar issue.

Following is my OS

Linux bpi-iot-ros-ai 4.19.49-BPI-R64-Kernel #1 SMP Wed Dec 11 14:22:23 IST 2019 aarch64 aarch64 aarch64 GNU/Linux

I am trying to get Google Coral mini PCIe work on BPI R64 V1.1. When I insert the chip on the CN25 slot and boot the OS (https://drive.google.com/open?id=1zrOSS2QJPirSwoK5yJFx10SiOtxRjXPt). I am able to detect the Coral with the lspci command

When I load the PCIe drivers (.ko), the OS reboots as follows

pi@bpi-iot-ros-ai:~/gasket-dkms-1.0$ sudo insmod apex.ko
[ 1076.819466] Internal error: Oops: 96000044 [#1] SMP
[ 1076.824498] Modules linked in: apex(+) gasket
[ 1076.828996] Process insmod (pid: 2611, stack limit = 0x000000007192814c)
[ 1076.835908] CPU: 1 PID: 2611 Comm: insmod Not tainted 4.19.49-BPI-R64-Kernel #1
[ 1076.843441] Hardware name: Bananapi BPI-R64 (DT)
[ 1076.848199] pstate: 80000005 (Nzcv daif -PAN -UAO)
[ 1076.853141] pc : osq_lock+0x64/0x174
[ 1076.856825] lr : __mutex_lock.isra.1+0xb0/0x4e8
[ 1076.861490] sp : ffffff800ae735b0
[ 1076.864900] x29: ffffff800ae735b0 x28: ffffff8008c16d70 
[ 1076.870374] x27: 0000000000080000 x26: ffffff800ae737f0 
[ 1076.875849] x25: 0000000000000001 x24: 0000000000000089 
[ 1076.881322] x23: 0000000000000089 x22: 0000000000000002 
[ 1076.886795] x21: 0000000000000001 x20: ffffffc03e115314 
[ 1076.892269] x19: ffffffc03e115308 x18: 000000000000000a 
[ 1076.897743] x17: 0000000000000000 x16: 0000000000000000 
[ 1076.903217] x15: 0000000000000089 x14: ffffffc037e3b504 
[ 1076.908691] x13: ffffffffffffffff x12: ffffffc03e13ea68 
[ 1076.914165] x11: 0000000000080000 x10: ffffffc03e13ea60 
[ 1076.919639] x9 : 0000000000000000 x8 : 0000000000000000 
[ 1076.925113] x7 : 0000000000000000 x6 : 0000000000000000 
[ 1076.930587] x5 : c6b4a61c2e2578ba x4 : ffffff8008bc2580 
[ 1076.936061] x3 : ffffff8008bb6000 x2 : c6b4a59c36e19e3a 
[ 1076.941535] x1 : ffffffc03ffc9580 x0 : ffffffc03e115314 
[ 1076.947009] Call trace:
[ 1076.949525] osq_lock+0x64/0x174
[ 1076.952848] __mutex_lock_slowpath+0x10/0x18
[ 1076.957245] mutex_lock+0x30/0x3c
[ 1076.960660] mtk_pcie_irq_domain_alloc+0x38/0xc8
[ 1076.965417] irq_domain_alloc_irqs_hierarchy+0x14/0x1c
[ 1076.970710] irq_domain_alloc_irqs_parent+0x14/0x24
[ 1076.975736] msi_domain_alloc+0x80/0x138
[ 1076.979775] irq_domain_alloc_irqs_hierarchy+0x14/0x1c
[ 1076.985068] __irq_domain_alloc_irqs+0x15c/0x254
[ 1076.989825] msi_domain_alloc_irqs+0x74/0x22c
[ 1076.994312] pci_msi_setup_msi_irqs+0x2c/0x3c
[ 1076.998799] __pci_enable_msix_range+0x29c/0x458
[ 1077.003554] pci_enable_msix_range+0x10/0x18
[ 1077.007959] gasket_interrupt_msix_init+0xac/0x170 [gasket]
[ 1077.013704] gasket_interrupt_init+0x108/0x168 [gasket]
[ 1077.019089] gasket_enable_device+0x28/0x1ac [gasket]
[ 1077.024299] apex_pci_probe+0x398/0x410 [apex]
[ 1077.028877] pci_device_probe+0xa0/0x120
[ 1077.032921] really_probe+0x134/0x284
[ 1077.036692] driver_probe_device+0xa0/0xe4
[ 1077.040910] __driver_attach+0x80/0xb8
[ 1077.044771] bus_for_each_dev+0x5c/0x94
[ 1077.048721] driver_attach+0x20/0x28
[ 1077.052403] bus_add_driver+0xe0/0x1e4
[ 1077.056263] driver_register+0xa8/0xf4
[ 1077.060127] __pci_register_driver+0x40/0x48
[ 1077.064529] apex_init+0x44/0x1000 [apex]
[ 1077.068659] do_one_initcall+0x78/0x140
[ 1077.072610] do_init_module+0x58/0x1c0
[ 1077.076470] load_module+0x19e8/0x1db4
[ 1077.080329] __se_sys_finit_module+0x7c/0x90
[ 1077.084727] __arm64_sys_finit_module+0x14/0x1c
[ 1077.089395] el0_svc_common+0x8c/0xf0
[ 1077.093166] el0_svc_handler+0x5c/0x64
[ 1077.097025] el0_svc+0x8/0xc
[ 1077.099992] Code: f865d845 8b050082 f9000422 d5033abf (f8256881) 
[ 1077.106271] ---[ end trace 7b0a3f70e5d265d8 ]---
[ 1077.111026] Kernel panic - not syncing: Fatal exception
[ 1077.116410] SMP: stopping secondary CPUs
[ 1077.120450] Kernel Offset: disabled
[ 1077.124041] CPU features: 0x0,20002000
[ 1077.127899] Memory Limit: none
[ 1077.131042] Rebooting in 1 seconds..

I have also tried one more approach by building the kernel from source code with PCI Options enabled(https://github.com/BPI-SINOVOIP/BPI-R64-bsp-4.19) (Config file attached with PCI Options enabled)mt7622_evb_bpi_defconfig (127.1 KB)

On doing this, the OS boot never completes and is stuck in scheduling the mutex lock. Logs of the bootup is attached.bootupLog_PCIOptionsEnabled.txt (57.0 KB)

One more approach that I have tried is by modifying the mt7622-bananapi-bpi-r64.dts as attachedmt7622-bananapi-bpi-r64.dts (10.4 KB)

here I have removed the pcie function and added pcie0 and pcie1 as suggested by [email protected]. With this approach I could boot the OS but the lspci command could not detect the mini PCIe Google Coral attached.

4

have updated 5.4-r64-main on my gdrive…

https://drive.google.com/open?id=16exG2_Zqz9Cu1VD2_k38IW6AoscN_ESt

@batul.rangwala what have you changed? I guess you have modified dts like my splitting-commit your dts looks like that…have you also changed dtsi? Imho you get compile error if not…make sure it it like on my repo

Maybe you need also irq-patch

5

@frank-w

I tried to compile the kernel using the Ref https://github.com/frank-w/BPI-R2-4.14/tree/5.4-r64-dsa. The kernel got compiled and using package option, the package was successfully created. And tried to replace existing /boot/bananapi/bpi-r64/linux/uImage and /boot/bananapi/bpi-r64/linux/dtb/bpi-r64.dtb.

However, the boot folder on my banana pi R64 doesnot contain uImage.

I am a newbie to the linux kernel, so you please let me know the steps to load a kernel image

6

You have folder-structure in /boot? this depends on the os-image you are using…The first Partition of sdcard (/dev/mmcblk1p1) should be mounted to /boot. If not (and /boot is empty) try to mount it…

mount /dev/mmcblk1p1 /boot

it is more secure using external computer to rename old uImage/dtb and add new…

if folders existing which files are there? Maybe your image is configured to use another file…maybe there is a uEnv.txt where a variable named kernel is set to a filename

If you don’t get it working please make new topic for kernel-replace with infos about image,uenv.txt and kernelversion. I recommend a debug-uart so that you not dig in the dark and see messages from uboot

8

@frank-w, I have modified the folders on SD card as suggested. When I try to boot the BananaPi R64 using that SD card, the following error appears.
(Attaching a log file too as the message is appearing on forum in misaligned formbootupLog.txt (2.1 KB) )

Partition Map for MMC device 1 – Partition Type: DOS

Part Start Sector Num Sectors UUID Type 1 204800 524288 1f9b70ec-01 0c 2 729088 14211072 1f9b70ec-02 83 mmc1 is available Interface: MMC Device 1: Vendor: Man 000003 Snr f7d58e01 Rev: 8.5 Prod: WP256€ Type: Removable Hard Disk Capacity: 244016.0 MB = 238.2 GB (499744768 x 512) Filesystem: FAT32 "BPI-BOOT " Boot from SD reading bananapi/bpi-r64/linux/uEnv.txt 1016 bytes read in 5 ms (198.2 KiB/s) Loaded environment from uEnv.txt Banana Pi bpi-r64 chip: mt7622 Service: linux reading bananapi/bpi-r64/linux/dtb/mt7622-bananapi-bpi-r64.dtb 24258 bytes read in 8 ms (2.9 MiB/s) reading bananapi/bpi-r64/linux/uImage_nodt 8716360 bytes read in 570 ms (14.6 MiB/s) reading bananapi/berryboot.img ** Unable to read file bananapi/berryboot.img ** bootm flag=0, states=70f

Booting kernel from Legacy Image at 44000000 …

Image Name: Linux Kernel 5.4.0 Image Type: AArch64 Linux Kernel Image (uncompressed) Data Size: 8716296 Bytes = 8.3 MiB Load Address: 40080000 Entry Point: 40080000 Verifying Checksum … OK Unsupported Architecture 0x16 ERROR: can’t get kernel image! reading bananapi/bpi-r64/linux/uImage_nodt FAT: Misaligned buffer address (4007ff28) 8716360 bytes read in 5123 ms (1.6 MiB/s) bootm flag=0, states=70f

Booting kernel from Legacy Image at 4007ff28 …

Image Name: Linux Kernel 5.4.0 Image Type: AArch64 Linux Kernel Image (uncompressed) Data Size: 8716296 Bytes = 8.3 MiB Load Address: 40080000 Entry Point: 40080000 Verifying Checksum … OK Unsupported Architecture 0x16 ERROR: can’t get kernel image! bootm flag=0, states=70f

Booting kernel from Legacy Image at 4007ff28 …

Image Name: Linux Kernel 5.4.0 Image Type: AArch64 Linux Kernel Image (uncompressed) Data Size: 8716296 Bytes = 8.3 MiB Load Address: 40080000 Entry Point: 40080000 Verifying Checksum … OK Unsupported Architecture 0x16 ERROR: can’t get kernel image! BPI-IoT>

9

if using old uboot (i guess so because new is in testing state and not available in any ready-to use image) you need a 32bit uImage (arm). On my dsa-tree the option for arm64 exists but is disabled by default…i guess you’ve changed it,right?

But please discuss this in new thread if it is not working after changing mkimage-type

btw. for output of any kind please use code formatting (select text and click on “<>” button) for better readability

10

@frank-w

Thanks for you input.

I could load the kernel image by commenting uimagearch=arm64 in build.conf .

Have few queries related to this setup.

  1. We wish to load a kernel module gasket.ko & apex.ko which is need by Google coral which we plan to interface on Minipcie slot.When I try to load that module, it gives “Invalid module format”. Running a $file apex.ko -> ELF 64bit LSB Relocatable. Any idea whether this module can be loaded on the compiled kernel 5.4 kernel(32 bit ) .

  2. Can you share a U-boot with 64bit functionality enabled if possible

Please note, I didn’t start a new thread though suggested by you, as my queries are related to your Git repository reference that I have been using

11

kernel itself is 64bit, but container (mkimage) for uboot have to be 32bit when using old uboot. New uboot is currently in testing (https://github.com/frank-w/u-boot/tree/2020-01-bpi-r64-v2) and without ethernet. you need new ATF to load it directly and you cannot load 32bit containers. imho you should wait till it is ready :slight_smile:

but you can use uboot-binary (2020-01 v2 1G) and ATF (mt7622_ATF_32_64_release.img) from my gdrive: https://drive.google.com/open?id=1-EJgBZrPFQbacLxFvLopyqKqEDCxwj8k

your module needs to be same arch (arm64/aarch64)…i don’t know how you have compiled it, but this stuff for new thread…

12

@frank,

I have started a new thread for installing kernel headers issue. Can you please help me with it new thread

13

Hi @frank-w

Based on our discussion with Google on the PCIe issue on 5.4 kernel, it seems that the BAR0 for the pcie bridge didn’t get memory range assigned, which is preventing the coral from accessing the host’s memory.

can you please suggest how can we fix this?

Below is the dmesg log:

[    1.604440] pci 0000:01:00.0: [1ac1:089a] type 00 class 0x0000ff [    1.610743] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref] [    1.618078] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x000fffff 64bit pref] [    1.626285] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:00.0 (capable of 4.000 Gb/s with 5 GT/s x1 link) [    1.641429] pci_bus 0000:01: fixups for bus [    1.645617] pci_bus 0000:01: bus scan returning with max=01 [    1.651195] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01 [    1.657816] pci_bus 0000:00: bus scan returning with max=01 [    1.663407] pci 0000:00:00.0: BAR 0: no space for [mem size 0x200000000 64bit pref] [    1.671071] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x200000000 64bit pref] [    1.679079] pci 0000:00:00.0: BAR 8: assigned [mem 0x20000000-0x201fffff] [    1.685872] pci 0000:00:00.0: PCI bridge to [bus 01] [    1.690846] pci 0000:00:00.0:   bridge window [mem 0x20000000-0x201fffff]

-Gaurav

14

Maybe ranges is wrong

wrong assumption

And it looks like start-adress is same for pcie0+1 (overlapping) and size too big

//pcie0
ranges = <0x82000000 0 0x20000000  0x0 0x20000000  0 0x8000000>;
//pcie1
ranges = <0x82000000 0 0x28000000  0x0 0x28000000  0 0x8000000>;

i’m no dts-expert,but i think ot should be more like this:

//pcie0
ranges = <0x82000000 0 0x2000000  0x0 0x20000000  0 0x8000000>;
//pcie1
ranges = <0x84000000 0 0x2000000  0x0 0x28000000  0 0x8000000>;

if i interprete values right:

  • pcie0 size #1 is 512Mb - removed one 0 - now 32mb
  • Pcie1-size #1 is 640mb - removed 8 - also 32mb
  • start-address #1 of both was same - moved start after end of pcie0

I guess second bar is also wrong (128mb). R64 has only 1GB :slight_smile:

but maybe only the 3rd param is offset and 4th the size for pcie-memory

this was change suggested for splitting pcie-controller:

-	ranges = <0x82000000 0 0x20000000 0x0 0x20000000 0 0x10000000>;
+	ranges = <0x82000000 0 0x20000000 0x0 0x20000000 0 0x8000000>;

so size was splitted into 2 equal parts of 128MB starting from 512MB

this is huge memory-block for device that has 1GB…maybe anything is overlapping

15 
Hi Frank,

I am facing similar issue. I have made modifications to the [mt7622.dtsi|attachment](upload://ljQRVgMu9DidPen3euRbbo4IV4F.dtsi) (25.0 KB) 
After making these modifications to the mt7622.dtsi file, it still shows error as 

 gasket: loading out-of-tree module taints kernel.
[  183.107996] apex 0000:01:00.0: assign IRQ: got 140
[  183.108043] pci 0000:00:00.0: enabling device (0000 -> 0002)
[  183.108058] pci 0000:00:00.0: enabling bus mastering
[  183.108082] apex 0000:01:00.0: can't enable device: BAR 0 [mem 0x00000000-0x00003fff 64bit pref] not claimed
[  183.108100] apex 0000:01:00.0: BAR 2: assigned [mem 0x20000000-0x200fffff 64bit pref]
[  183.108193] apex 0000:01:00.0: BAR 0: assigned [mem 0x20100000-0x20103fff 64bit pref]
[  183.108310] apex 0000:01:00.0: enabling device (0000 -> 0002)
[  183.108363] apex 0000:01:00.0: enabling bus mastering
**[  195.906661] apex 0000:01:00.0: RAM did not enable within timeout (12000 ms)**

Can you please suggest how can we fix this?

16

I guess my conclusions are wrong and ranges are correct…i also find no overlapping node. Imho we need some vendor info here

17

Thanks for your prompt response.

Can you please elaborate what kind of vendor info do we need ?

18

The “how to fix” :slight_smile:

I asked employee from mtk by whom i got the pcie-splitting code

19

@frank-w, which Qualcomm Atheros chip do you have? Have you try the ath10k-ct driver?

I have a Compex WLE900VX QCA9880 pcie wifi module, If I use the ath10k driver, I have the same error with you. If I use the ath10k-ct driver, the driver can startup correctly, but can’t connect to the QCA9880 hardware, it seems WLE900VX need usb interface in the minipcie, but the CN25 minipcie port isn’t have the usb interface.

20

I used ath10k driver and see wifi-interface…did not made further tests due to less time

21

Which wifi module do you use? and which board do you use? BPI-R64 or BPI-R2?