[bpi-r64] support pstore/ramoops

just share a information that new atf was released, and you can base on it to enable pstore/ramoops in both uboot and openwrt environment.

arm-trusted-firmware-mediatek: update to git HEAD

uboot-mediaktek: add support for PSTORE and check it on boot

mediatek: reserve memory for ramoops and enable PSTORE

thanks for the information

what is pstore/ramoops for?

i see mt7622-support is now splitted into multiple commits, is it already upstream?

ramoops is used to store crash log to dram before system reboot, and then user can dump log in uboot/linux in 2nd boot. if you have already used mtk atf from https://github.com/mtk-openwrt/arm-trusted-firmware.git and 2021.04 uboot. you can simply enable it in uboot/openwrt menuconfig, thanks.

File systems --> Miscellaneous filesystems
  <*>   Persistent store support  
  < >     DEFLATE (ZLIB) compression  
  < >     LZO compression (NEW) 
  < >     LZ4 compression (NEW) 
  < >     LZ4HC compression (NEW)  
  [ ]     842 compression (NEW)
  [ ]     zstd compression (NEW)  
  [*]     Log kernel console messages  
  [*]     Log user space messages 
  <*>     Log panic/oops to a RAM buffer   

Command line interface --> Misc commands

  [*] pstore 
  (0x42ff0000) Memory Address 
  (0x10000) Memory size
  (0x1000) Dump record size (NEW)
  (0x1000) Kernel console log size (NEW)
  (0x1000) FTrace log size (NEW) 
  (0x1000) User space message log size (NEW)  
  (0)   ECC size (NEW)

This looks useful… Can this be used with Debian, too?

Sure, this is a generic Linux feature, all needed is to set it up in device tree and make sure the bootchain doesn’t corrupt the RAM (ie. if you use recent enough ARM Trusted Firmware based bootchain it works). I guess @frank-w has made the changes needed for that for the SD card image for Debian as well, so it should just work. Try

echo c > /proc/sysrq-trigger

After the reboot you should see the previous kernel log in /sys/fs/pstore/.

I have not added pstore related changes. If you point me to needed changes i can add them

@moore summarized it in [bpi-r64] support pstore/ramoops

Let me explain the details:

  • You need to use ARM Trusted Firmware DDR initialization blob which was updated to support keeping DRAM content:
  • It’s nice to have some PSTORE support in U-Boot as well (but it’s not strictly needed for it to work in Linux, as long as you take care not to overwrite the RAM content in the area used for PSTORE (e.g. by just adding reserved memory node in DTS):
  • Last but not least, add ramoops node to DTS used by Linux (U-Boot will also do that for you if it has PSTORE support, however, for some boards with MT7622 supported by OpenWrt we use the vendor loader and hence cannot rely in U-Boot features to be present):

Oh, just realized this thread is about the R64… I only have a R2. Big sad face here…

Maybe it is possible with r2 too,but we have no preloader source and currently no information about it on mt7623

edit: added now pstore to u-boot and linux 5.17

build uboot and used it in my ATF branch mtksoc which is basicly on same state as the one you’ve mentioned

@dangowrt: tried to use it, but bl2 brings some strange errors and does not find fip-partition

NOTICE:  BL2: v2.4(release):v2.4-415-g7a2fc1857bde-dirty
NOTICE:  BL2: Built : 12:38:35, Jan 30 2022
ERROR:   Cannot find any pass-window
ERROR:   no DATLAT taps pass, DATLAT calibration fail!
ERROR:   DATLAT calibration fail, write back to  20!
ERROR:   EMI: Failed to detect rank size
ERROR:   EMI: complex R/W mem test failed: -2
ERROR:   Partition 'fip' not found
PANIC at PC : 0x000000000020478c

i remembered that i tried the mtksoc branch and it was not bootable so left the older tree as default

maybe @moore has an idea too?

The error looks like you forgot to set DDR3_FLYBY=1 to include the correct DDR calibration binary for boards with 2 RAM chips (and hence fly-by).

at least no error, but bl31 now hangs at

NOTICE:  BL2: v2.4(release):v2.4-416-g64e90e1a9628-dirty
NOTICE:  BL2: Built : 13:45:46, Jan 30 2022
NOTICE:  BL2: Booting BL31
NOTICE:  BL31: v2.4(release):v2.4-416-g64e90e1a9628-dirty
NOTICE:  BL31: Built : 13:45:48, Jan 30 2022

Is there anything special in uboot bootup?

i used u-boot.bin i build here: https://github.com/frank-w/u-boot/commits/2022-01-bpi This is how i build fip: https://github.com/frank-w/BPI-R64-ATF/blob/mtksoc/build.sh#L48

Maybe i can use a binary build by atf in uboot like i do for r2pro. There uboot fip is build with atf binaries. For r64 i need to copy uboot.bin to atf repo and build the resulting binary there (checked filesize of the fip.bin…it is 654kbyte, so uboot seems to be still included)

btw. patch in uboot cause this

arch/arm/dts/mt7622-rfb.dtb: Warning (ranges_format): /reserved-memory:ranges: empty "ranges" property but its #address-cells (2) differs from / (1)
arch/arm/dts/mt7622-rfb.dtb: Warning (ranges_format): /reserved-memory:ranges: empty "ranges" property but its #size-cells (2) differs from / (1)
  DTC     arch/arm/dts/mt7629-rfb.dtb
  DTC     arch/arm/dts/mt8183-pumpkin.dtb
arch/arm/dts/mt7622-bananapi-bpi-r64.dtb: Warning (ranges_format): /reserved-memory:ranges: empty "ranges" property but its #address-cells (2) differs from / (1)
arch/arm/dts/mt7622-bananapi-bpi-r64.dtb: Warning (ranges_format): /reserved-memory:ranges: empty "ranges" property but its #size-cells (2) differs from / (1)

i changed the address/size-cells to 1 each and the warning is gone

Kernel patch had a 0x too much which caused a warning

Edit: seems build-order has changed: openwrt includes atf in uboot now instead of atf including u-boot.bin. this is better as atf does not change so oftens as uboot.

Have not found the fiptool,but i guess it’s a binary…i want to avoid it if possible and try building uboot with bl31 like for r2pro

Hope this way works for r64 too (without bl32 and mkimage).

if including the blx into uboot i used the bl31.elf for r2pro, this file is for r64 500k in size, is this right? do i need any config-option in uboot for including the bl31? found none for r2pro, only for spl, we don’t need for r64, right? only CONFIG_FIT is maybe needed

edit: tried to include bl31 and build u-boot.itb

ln -sf files/bpi-r64/bl31.elf bl31.elf #the file from bl31 folder
make ${CFLAGS} u-boot.itb
sudo dd of=/dev/sdb if=u-boot.itb bs=512 seek=2048

but i get now:

NOTICE:  BL2: v2.4(release):v2.4-416-g64e90e1a9628-dirty
NOTICE:  BL2: Built : 13:45:46, Jan 30 2022
ERROR:   BL2: Failed to load image id 3 (-2)

Maybe @moore or @hackpascal can help

You could try to raise log level, maybe you get more info:

PLAT=mt7622 DDR3_FLYBY=1 LOG_LEVEL=40

got it…i accidentally built a 32bit uboot-binary for r64 (had forgotten to enable the arch=arm64 in build.conf)

but i see this error in uboot (seems to come from my env loading which loads to loadaddr=0x44000000):

ERROR: reserving fdt memory region failed (addr=0 size=43000000 flags=4)

maybe because uboot tries to allocate memory in region already blocked by kernel fdt?

and on running this:

echo c > /proc/sysrq-trigger

i get a stacktrace:

[email protected]:~# echo c > /proc/sysrq-trigger
[  236.402700] sysrq: Trigger a crash
[  236.406121] Kernel panic - not syncing: sysrq triggered crash
[  236.411863] CPU: 0 PID: 298 Comm: bash Tainted: G S                5.17.0-rc1-bpi-r64-r64 #1
[  236.420298] Hardware name: Bananapi BPI-R64 (DT)
[  236.424910] Call trace:
[  236.427350]  dump_backtrace+0xe0/0x114
[  236.431105]  show_stack+0x18/0x64
[  236.434419]  dump_stack_lvl+0x68/0x84
[  236.438081]  dump_stack+0x18/0x34
[  236.441393]  panic+0x130/0x2e8
[  236.444445]  sysrq_reset_seq_param_set+0x0/0x94
[  236.448977]  __handle_sysrq+0xc0/0x174
[  236.452724]  write_sysrq_trigger+0x9c/0xf8
[  236.456819]  proc_reg_write+0x8c/0xa4
[  236.460479]  vfs_write+0x90/0x108
[  236.463792]  ksys_write+0x70/0xdc
[  236.467105]  __arm64_sys_write+0x18/0x24
[  236.471024]  invoke_syscall+0x6c/0xf8
[  236.474685]  el0_svc_common.constprop.0+0xb0/0xdc
[  236.479387]  do_el0_svc+0x44/0x74
[  236.482700]  el0_svc+0x20/0x4c
[  236.485752]  el0t_64_sync_handler+0xcc/0x154
[  236.490018]  el0t_64_sync+0x1a0/0x1a4
[  236.493679] SMP: stopping secondary CPUs
[  236.497607] Kernel Offset: disabled
[  236.501088] CPU features: 0x0,00006002,00000842
[  236.505613] Memory Limit: none
[  236.508663] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---

source: u-boot and linux 5.17

based on dts change i guess i need the CONFIG_ARM_PSCI_CPUIDLE_DOMAIN in kernel, right (currently set to “n”)?

but still the crash…

[email protected]:~# zgrep 'PSTORE' /proc/config.gz 
CONFIG_PSTORE=y
CONFIG_PSTORE_DEFAULT_KMSG_BYTES=10240
CONFIG_PSTORE_DEFLATE_COMPRESS=y
# CONFIG_PSTORE_LZO_COMPRESS is not set
# CONFIG_PSTORE_LZ4_COMPRESS is not set
# CONFIG_PSTORE_LZ4HC_COMPRESS is not set
# CONFIG_PSTORE_842_COMPRESS is not set
# CONFIG_PSTORE_ZSTD_COMPRESS is not set
CONFIG_PSTORE_COMPRESS=y
CONFIG_PSTORE_DEFLATE_COMPRESS_DEFAULT=y
CONFIG_PSTORE_COMPRESS_DEFAULT="deflate"
CONFIG_PSTORE_CONSOLE=y
CONFIG_PSTORE_PMSG=y
CONFIG_PSTORE_RAM=y
# CONFIG_PSTORE_BLK is not set
[email protected]:~# zgrep 'IDLE' /proc/config.gz 
CONFIG_NO_HZ_IDLE=y
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_MULTIPLE_DRIVERS=y
# CONFIG_CPU_IDLE_GOV_LADDER is not set
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_CPU_IDLE_GOV_TEO is not set
CONFIG_DT_IDLE_STATES=y
# CONFIG_ARM_CPUIDLE is not set
CONFIG_ARM_PSCI_CPUIDLE=y
CONFIG_ARM_PSCI_CPUIDLE_DOMAIN=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_GENERIC_IDLE_POLL_SETUP=y
# CONFIG_IDLE_PAGE_TRACKING is not set
# CONFIG_NETFILTER_XT_TARGET_IDLETIMER is not set

or is this as intended (c=crash)?

sysrq: Trigger a crash

but i have nothing in the /sys/fs/pstore/ dir

[email protected]:~# mount -t pstore pstore /sys/fs/pstore
mount: /sys/fs/pstore: pstore already mounted on /sys/fs/pstore.
[email protected]:~# ls /sys/fs/pstore/
[email protected]:~#

looks like in uboot there is something in pstore

BPI-R64> pstore display
Compressed buffer, display not available
BPI-R64> 

but in linux still empty

[email protected]:~# dmesg | grep -i pstore
[    0.015542] pstore: Registered ramoops as persistent store backend
[    1.582087] pstore: Using crash dump compression: deflate
[email protected]:~# 
[email protected]:~# ls /sys/fs/pstore/
[email protected]:~#

after reboot (without power-loss) pstore in uboot is empty again

so i disabled compression in linux and i can see content in uboot…

BPI-R64> pstore display
**** Dump
Panic#1 Part1
<7>[    7.500652] mt7615e 0000:01:00.0: enabling bus mastering
<7>[    7.500835] mtk-pcie 1a143000.pcie: msi#0 address_hi 0x0 address_lo 0x410b50c0
...
<6>[   57.450520] sysrq: Trigger a crash
<0>[   57.453941] Kernel panic - not syncing: sysrq triggered crash
<7>[   57.459684] CPU: 1 PID: 292 Comm: bash Tainted: G S                5.17.0-rc1-bpi-r64-r64 #4
<7>[   57.468118] Hardware name: Bananapi BPI-R64 (DT)
<7>[   57.472729] Call trace:
<7>[   57.475168]  dump_backtrace+0xe0/0x114

so pstore seems to work across boot (wait after crash for reboot), so after display the pstore in uboot booting into same linux kernel again…but pstore there is empty again