Unable to boot anymore from SD (uboot env ?)

I’m currently no longer able to boot from SD. I’ve been able in the past to do the installation as usual, SD->NAND->EMMC, for reasons unknown (maybe uboot environment?). Attempting to boot from SD (all 4 switches in the UP position) just gives me this:

spi-nand: spi_nand spi_nand@1: Winbond SPI NAND was found.
spi-nand: spi_nand spi_nand@1: 128 MiB, block size: 128 KiB, page size: 2048, O4
jedec_spi_nor spi_nor@0: unrecognized JEDEC id bytes: 00, ef, aa
'spi-nand0' is now active device
* spi-nand0
  - device: spi_nand@1
  - parent: spi@1100a000
  - driver: spi_nand
  - type: NAND flash
  - block size:        0x20000 bytes
  - page size:         0x800 bytes
  - OOB size:          64 bytes
  - OOB available:     24 bytes
  - 0x000000000000-0x000008000000 : "spi-nand0"
          - 0x000000000000-0x000000080000 : "bl2"
          - 0x000000080000-0x000000380000 : "factory"
          - 0x000000380000-0x000000580000 : "fip"
          - 0x000000580000-0x000008000000 : "ubi"
Press ENTER to return to menu 

Uboot can see the card:

MT7986> mmcinfo
Device: mmc@11230000
Manufacturer ID: 2
OEM: 544d
Name: SA16G 
Bus Speed: 50000000
Mode: SD High Speed (50MHz)
Rd Block Len: 512
SD version 3.0
High Capacity: Yes
Capacity: 14.5 GiB
Bus Width: 4-bit
Erase Group Size: 512 Bytes

But thinks its partition table is bad:

MT7986> mmc part
## Unknown partition table type 0

Its possible that I may have inadvertenly selected

a. Reset all settings to factory defaults.

In the boot menu while navigating, and this may be the cause of my issue. I am able to successfully boot from NAND and eMMC, but it has the older, broken, image, which is not of very much use to me.

Any help appreciated.

Okay, I figured this out.

There seems to be issues when writing the (latest?) sdcard images. For whatever reason, using either dd or Etcher, linux complained that GPT is damaged. Using gdisk to “fix” the partition table allows me to boot from these SD cards. I dont know if that is a “leftover” from the sd card, which was previously used somewhere else…

However, I just did the following

  1. created a new GPT partition table on the device, with a cleared partition 1
  2. wrote the snapshot image with Etcher

On the machine where I just wrote the image, dmesg shows “Alternate GPT is invalid, using primary GPT”

Indeed, the R3 spews messages about invalid GPT partition, and doesn’t boot from this card

Take the sdcard, “correct” the GPT table with gdisk under Linux, reinsert into R3

Then, I can boot from SD.

I didn’t have this issue when I originally installed a few days ago, where I bascially did the same thing without issue.

I had the same issue with new Samsung Pro micro SD card which came with GPT partition table from the factory. Writing using dd didn’t clear the table at the end of the device, which then confused U-Boot.

Maybe this should go into the openwrt device page when it goes live.

Hi, I have the same issue but cannot follow your steps and solve the problem. Which OS image were you using?

However, I just did the following

  • created a new GPT partition table on the device, with a cleared partition 1
  • Take the sdcard, “correct” the GPT table with gdisk under Linux, reinsert into R3

How are you creating the GPT table and which options are you using to correct it with gdisk?

Could you further describe your situation? Thank you

You could try sgdisk backup/restore

sudo sgdisk --backup=bpi-r3_sgdisk.gpt /dev/sdb
sudo sgdisk --load-backup=bpi-r3_sgdisk.gpt /dev/sdb

Hi Frank-w and thanks for your reply. I am experiencing the following issue with the Ubuntu 22.04 server image you provided. How have you solved the problem?

I am leaving here some info about the problem in case someone runs into it.

When I write the image I am able to boot the OS straight away, but as soon as I power cycle I get this on the serial console

F0: 102B 0000
FA: 1040 0000
FA: 1040 0000 [0200]
F9: 103F 0000
F3: 1001 0000 [0200]
F3: 1001 0000
F6: 300C 0028
F5: 0000 0000
V0: 0000 0000 [0001]
00: 0000 0000
BP: 2400 0041 [0000]
G0: 1190 0000
** No partition table - mmc 0 **
Couldn't find partition mmc 0:5
## Error: "initrd" not defined
mmc - MMC sub system
[...]
sd available
jedec_spi_nor spi_nor@0: unrecognized JEDEC id bytes: 00, ef, aa
Failed to initialize SPI flash at 0:0 (error 0)
NAND available
sd nand
fit=bpi-r3.itb
** No partition table - mmc 0 **
Couldn't find partition mmc 0:5
Can't set block device
MT7986> 

Plugging the sd card into another PC and running fdisk or gdisk returns the following errors related to the partition table.

userk@dopamine:~/$ sudo fdisk -l /dev/sdd

The backup GPT table is corrupt, but the primary appears OK, so that will be used.
Disk /dev/sdd: 119.2 GiB, 127984992256 bytes, 249970688 sectors
Disk model: MassStorageClass
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: FCB519F7-173A-4AF3-B570-FD1701EFC7AC

Device      Start      End  Sectors  Size Type
/dev/sdd1      34     8191     8158    4M Linux filesystem
/dev/sdd2    8192     9215     1024  512K Linux filesystem
/dev/sdd3    9216    13311     4096    2M Linux filesystem
/dev/sdd4   13312    17407     4096    2M Linux filesystem
/dev/sdd5   17408   222207   204800  100M Linux filesystem
/dev/sdd6  222208 12805120 12582913    6G Linux filesystem

When I run gdisk

userk@dopamine:~/$ sudo gdisk /dev/sdd
GPT fdisk (gdisk) version 1.0.5

Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

Warning! One or more CRCs don't match. You should repair the disk!
Main header: OK
Backup header: ERROR
Main partition table: OK
Backup partition table: ERROR

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: damaged

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************

I even tried to print the partition table using the p option of gdisk and write the changes with w but nothing I get the same issue. When I backup the GPT partition table using the command you suggested returns:

userk@dopamine:~/$ sudo sgdisk --backup=bpi-r3_sgdisk.gpt /dev/sdd
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

Warning! One or more CRCs don't match. You should repair the disk!
Main header: OK
Backup header: ERROR
Main partition table: OK
Backup partition table: ERROR

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
The operation has completed successfully.

When I restore the GPT partition table from the file sgdisk returns the following:

userk@dopamine:~/$ sudo sgdisk --load-backup=bpi-r3_sgdisk.gpt /dev/sdd
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

Warning! One or more CRCs don't match. You should repair the disk!
Main header: OK
Backup header: ERROR
Main partition table: OK
Backup partition table: ERROR

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.

If I insert the microSD card the serial console returns a Kernel Panic error

[...]
[   10.478537] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,6)
[   10.486956] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 6.1.12-bpi-r3-main #6
[   10.493901] Hardware name: Bananapi BPI-R3 (DT)
[   10.498416] Call trace:
[   10.500851]  dump_backtrace+0xdc/0x130
[   10.504594]  show_stack+0x18/0x30
[   10.507896]  dump_stack_lvl+0x68/0x84
[   10.511549]  dump_stack+0x18/0x34
[   10.514852]  panic+0x174/0x338
[   10.517897]  mount_block_root+0x15c/0x20c
[   10.521897]  mount_root+0x1dc/0x218
[   10.525375]  prepare_namespace+0x130/0x170
[   10.529459]  kernel_init_freeable+0x2cc/0x2f4
[   10.533803]  kernel_init+0x24/0x130
[   10.537282]  ret_from_fork+0x10/0x20
[   10.540847] SMP: stopping secondary CPUs
[   10.544758] Kernel Offset: disabled
[   10.548232] CPU features: 0x00000,00800084,0000420b
[   10.553095] Memory Limit: none
[   10.556139] Rebooting in 1 seconds..

And from there every time I get

F0: 102B 0000
FA: 1040 0000
FA: 1040 0000 [0200]
F9: 103F 0000
F3: 1001 0000 [0200]
F3: 1001 0000
F6: 300C 0028
F5: 480A 0032
00: 1005 0000
FA: 1040 0000
FA: 1040 0000 [0200]
F9: 103F 0000
F3: 1001 0000 [0200]
F3: 1001 0000
F6: 300C 0028
01: 102A 0001
02: 1005 0000
BP: 2000 00C0 [0001]
EC: 0000 0000 [3000]
T0: 0000 0585 [010F]
System halt!

After one minute, I get the Kernel Panic error again.

Last log is when bl2 is not found on selected bootdevice (jumpers). The one before is missing bootargs (missing root= in bootargs).

Else i see no errors

I found the part of the log mentioning the missing the boot option.

[    1.941156] Alternate GPT is invalid, using primary GPT.
[    1.948490]  mmcblk0: p1 p2 p3 p4 p5 p6
[    2.162540] mtk-pcie-gen3 11280000.pcie: PCIe link down, current LTSSM state: detect.active (0x1000001)
[    2.171967] mtk-pcie-gen3: probe of 11280000.pcie failed with error -110
[    2.201430] mt7530 mdio-bus:1f: configuring for fixed/2500base-x link mode
[    2.208602] mt7530 mdio-bus:1f: Link is Up - 2.5Gbps/Full - flow control rx/tx
[    2.217545] mt7530 mdio-bus:1f wan (uninitialized): PHY [mt7530-0:00] driver [MediaTek MT7531 PHY] (irq=127)
[    2.237411] mt7530 mdio-bus:1f lan0 (uninitialized): PHY [mt7530-0:01] driver [MediaTek MT7531 PHY] (irq=128)
[    2.257090] mt7530 mdio-bus:1f lan1 (uninitialized): PHY [mt7530-0:02] driver [MediaTek MT7531 PHY] (irq=129)
[    2.276774] mt7530 mdio-bus:1f lan2 (uninitialized): PHY [mt7530-0:03] driver [MediaTek MT7531 PHY] (irq=130)
[    2.296434] mt7530 mdio-bus:1f lan3 (uninitialized): PHY [mt7530-0:04] driver [MediaTek MT7531 PHY] (irq=131)
[    2.307310] device eth0 entered promiscuous mode
[    2.311962] DSA: tree 0 setup
[    2.315325] xhci-mtk 11200000.usb: supply vbus not found, using dummy regulator
[    2.322716] xhci-mtk 11200000.usb: supply vusb33 not found, using dummy regulator
[    2.331062] xhci-mtk 11200000.usb: xHCI Host Controller
[    2.336332] xhci-mtk 11200000.usb: new USB bus registered, assigned bus number 1
[    2.346711] xhci-mtk 11200000.usb: hcc params 0x01403f99 hci version 0x110 quirks 0x0000000000210010
[    2.355871] xhci-mtk 11200000.usb: irq 132, io mem 0x11200000
[    2.361696] xhci-mtk 11200000.usb: xHCI Host Controller
[    2.366919] xhci-mtk 11200000.usb: new USB bus registered, assigned bus number 2
[    2.374307] xhci-mtk 11200000.usb: Host supports USB 3.2 Enhanced SuperSpeed
[    2.381799] hub 1-0:1.0: USB hub found
[    2.385573] hub 1-0:1.0: 2 ports detected
[    2.389866] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    2.398315] hub 2-0:1.0: USB hub found
[    2.402076] hub 2-0:1.0: 1 port detected
[    2.407105] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[    2.415798] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[    2.422875] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[    2.431479] platform regulatory.0: Falling back to sysfs fallback for: regulatory.db
[    2.452901] EXT4-fs (mmcblk0p6): INFO: recovery required on readonly filesystem
[    2.460210] EXT4-fs (mmcblk0p6): write access will be enabled during recovery
[    2.832515] usb 1-1: new high-speed USB device number 2 using xhci-mtk
[    3.013287] hub 1-1:1.0: USB hub found
[    3.017086] hub 1-1:1.0: 4 ports detected
[    3.164472] JBD2: Out of memory during recovery.
[   11.992778] random: crng init done
[   18.075717] JBD2: recovery failed
[   18.079024] EXT4-fs (mmcblk0p6): error loading journal
[   18.086319] VFS: Cannot open root device "mmcblk0p6" or unknown-block(179,6): error -5
[   18.094255] Please append a correct "root=" boot option; here are the available partitions:
[   18.102606] 1f00          125440 mtdblock0 
[   18.102611]  (driver?)
[   18.109125] 1f01            2048 mtdblock1 
[   18.109128]  (driver?)
[   18.115647] 1f02            3072 mtdblock2 
[   18.115650]  (driver?)
[   18.122165] 1f03             512 mtdblock3 
[   18.122168]  (driver?)
[   18.128686] b300       124985344 mmcblk0 
[   18.128689]  driver: mmcblk
[   18.135474]   b301            4079 mmcblk0p1 0fc33657-fb5e-4610-b086-bdd6c84e2a98
[   18.144430]   b302             512 mmcblk0p2 071e1c9e-c515-4076-b68c-87e24d3dda65
[   18.153380]   b303            2048 mmcblk0p3 9c4cd666-f93c-4cca-903a-c111bf834264
[   18.162325]   b304            2048 mmcblk0p4 34c4c6a4|                             |
[   18.171274]   b305          102400 mmcblk0p5 67dd2091|                             |
[   18.180223]   b306         6291456 mmcblk0p6 afed5d9e-1438-48c2-a539-f39bf7b35821
[   18.180227] 
[   18.189178] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,6)

The problem is that there is no way to provide any options since I don’t have a console to write commands into.

How do I specify or append the right boot option, should we put mmcblk0?

Last log is when bl2 is not found on the selected bootdevice (jumpers).

Don’t really know what bl2 is, am I doing something wrong? I double-checked the jumpers and they are in the right configuration.

Thanks for your support @frank-w, I appreciate it.

Bl2 is the first stage bootloader after bootrom on board (which shows your current output). Bl2 is part of atf and loads uboot fip (u-boot bundled with another part of atf).

Which image do you use? Your last trace had linux running so bl2,uboot and kernel was found…only your root in cmdline is missing…this is normally set in uboot (in my images/uboot is defined via environment)

Thanks for the info.

I am using bpi-r3_sdmmc_ubuntu2204.img from this Gdrive link.

it is my image, you have access with debug-uart? can you please post your env in uboot (call printenv on uboot-console)?

it should show variables similar to this: https://github.com/frank-w/u-boot/blob/2023-04-bpi/uEnv_r3.txt

key-parts are these:

newboot=run checkrd; run setbootconf; run buildargs;if printenv fit;then setenv kernel ${fit};if run loadkernel; then bootm ${kaddr}${bootconf}; fi;fi;
buildargs=setenv bootargs "board=${board} earlycon=uart8250,mmio32,0x11002000 ${bootopts} root=${root}"
root=/dev/mmcblk0p6 rootfstype=ext4 rootwait

so bootargs (and your cmdline) should be filled with “root=/dev/mmcblk0p6 rootfstype=ext4 rootwait” by calling buildargs by newboot (which is used by default)

Yes I do have access.

U-Boot 2023.01-bpi-00024-gbc45a33dad0a-dirty (Feb 18 2023 - 11:38:01 +0100)

CPU:   MediaTek MT7986
Model: mt7986-rfb
DRAM:  2 GiB
Core:  44 devices, 20 uclasses, devicetree: separate
MMC:   mmc@11230000: 0
Loading Environment from MMC... unable to read ssr
unable to read ssr
*** Warning - bad CRC, using default environment

In:    serial@11002000
Out:   serial@11002000
Err:   serial@11002000
Net:   
Warning: ethernet@15100000 (eth0) using random MAC address - 76:a3:1e:0b:4e:f8
eth0: ethernet@15100000
Hit any key to stop autoboot:  0 
MT7986>

Here is the output of the printenv command. I’m providing the output above for reference in case anyone encounters a similar issue.

I can see the section with root=/dev/mmcblk0p6 rootfstype=ext4 rootwait. Looks ok.

MT7986> printenv
askkernel=askenv kernelinput "enter kernel name:";
board=bpi-r3
boot0=run lskernel;run askkernel;if printenv kernelinput ;then setenv fit ${kernelinput}; run newboot; fi;
boot1=run loadenv; run newboot;
bootcmd=setenv bootdelay 3; run loadenv;bootmenu;
bootdelay=0
bootdevice=sd
bootenv=uEnv.txt
bootmenu_0=1. Enter kernel-name to boot from SD/EMMC.=run boot0
bootmenu_1=2. Boot from SD/EMMC.=run boot1
bootmenu_default=1
bootopts=debug=7
buildargs=setenv bootargs "board=${board} earlycon=uart8250,mmio32,0x11002000 ${bootopts} root=${root}"
checkenv=test -e ${device} ${partition} ${bootenv}
checkmmc=if mmc partconf 0; then echo "emmc available";mmcdev="emmc";else echo "sd available";mmcdev="sd";fi
checkrd=if printenv initrd;then run useinitrd;fi
checkspi=if sf probe; then echo "NOR available";spidev="nor";else echo "NAND available";spidev="nand";fi
console=ttyS0,115200n1 earlycon=uart8250,mmio32,0x11002000
device=mmc
ethaddr=76:a3:1e:0b:4e:f8
fdtcontroladdr=bf7fd1d0
fit=bpi-r3.itb
kaddr=0x46000000
loadaddr=0x46000000
loadenv=if run checkenv;then run loadenvfile;fi
loadenvfile=if fatload ${device} ${partition} ${loadaddr} ${bootenv};then env import -t ${loadaddr} ${filesize};else echo "fatload (${bootenvi
loadkernel=fatload $device $partition $kaddr ${kernel}
lskernel=ls ${device} ${partition};
newboot=run checkrd; run setbootconf; run buildargs;if printenv fit;then setenv kernel ${fit};if run loadkernel; then bootm ${kaddr}${bootcon;
partition=0:5
rdaddr=0x48000000
resetenv=env default -a;printenv;
root=/dev/mmcblk0p6 rootfstype=ext4 rootwait
setbootconf=run checkmmc checkspi;echo $mmcdev $spidev;setenv bootconf "#conf-$mmcdev-$spidev"
stderr=serial@11002000
stdin=serial@11002000
stdout=serial@11002000
useinitrd=setenv root "/dev/ram0 rw";setenv bootopts "${bootopts} initrd=${rdaddr},20M";fatload ${device} ${partition} ${rdaddr} ${initrd};
useusb=if usb info; then usb reset;else usb start;fi; setenv device usb;setenv partition 0:1;ls ${device} ${partition}
wremmc=if printenv bl2file;then mmc partconf 0 1 1 1;if fatload $device $partition $loadaddr $bl2file;then mmc erase 0x0 0x400;mmc write ${loi
wrspimnand=if printenv bl2file;then if fatload $device $partition $loadaddr $bl2file;then mtd erase spi-nand0 0x0 0x100000;mtd write spi-nandi

Environment size: 2971/524284 bytes
MT7986>

Pls try this:

run buildargs
printenv bootargs
bootmenu

MT7986> run buildargs

Returned no output

MT7986> printenv bootargs

Returned root=/dev/mmcblk0p6 rootfstype=ext4 rootwait

MT7986> bootmenu

Sent me to the bootmenu with three options but it selected one option really fast and tried to boot from SD.

I create a github gist with the output of the attempt to boot from SD

I tested writing the image to a different microSD card, and now I can power cycle the router without encountering any problems.

Just to help the next user, I used the following command to write the image using the bpi-tool software utility

sudo bpi-copy bpi-r3_sdmmc_ubuntu2204.img /dev/sdd

Thanks frank-w

you do not need the bpi-tools to do that…every linux distribution has dd-command

this unpacks the image on-the-fly and writes it to the card:

gunzip -c bpi-r3_sdmmc_bullseye.img.gz | sudo dd bs=1M status=progress conv=notrunc,fsync of=/dev/sdX

if your image is already unpacked the dd-command looks like this:

sudo dd bs=1M status=progress conv=notrunc,fsync if=sdmmc_bullseye.img of=/dev/sdX

I’d recommend to wipe all existing partition tables and filesystem identifiers before writing:

wipefs -a /dev/sdX

This can avoid problems, esp. with left-over GPT at the end of the device.