[BPI-R3] [OpenWRT] File system and partition hierarchy

[danoond (Dan Mikhailov)]

Hello! I have recently bought an BPI-R3 and have already installed OpenWRT 23.05.2 on the NAND and then on the eMMC. The next thing I wanted to do is to use all available size of the eMMC, because right after installation only a small part of it was occupied by partitions and their respecting file systems.

I have spent 3 or 4 days figuring out how all the partitions and file systems on eMMC work and why is it designed that way. Why is it so complicated to just resize the partition and expand the file system? How to do this the correct way, maybe i should create separate partition on the eMMC instead of expanding root? Is there any scripts that automatically manage the partitions/resizing/free space on bootup?

So many questions, but so little answers, OpenWRT documentation is not helping either as it looks pretty much outdated and is not being updated with relevant information. I would be very grateful if someone could give an explanation on how everuthing works… I am still relatively new to OpenWRT (previously had it installed on Rpi4 + USB SSD), so please don’t throw rocks at me

[dangowrt (Daniel Golle)]

@danoond

maybe i should create separate partition on the eMMC instead of expanding root?

Yes. Because the rootfs is an overlayfs re-created every time you update the device, only configuration files are kept anyway, so it doesn’t make sense to store big things there.

If you want to anyway increase the rootfs, lets say, to have some large software packages installed, you can still do so, but you will need to boot into the recovery initramfs system because you cannot resize a filesystem while it is mounted.

The easiest way to reboot into the recovery system is to issue

echo c > /proc/sysrq-trigger

which will simulate a crash and subsequent reboot, and result in the bootloader choosing to start the recovery image with initramfs instead of the production with the read/write overlay.

Now connect to the running recovery system using ssh [email protected]. There, first give the box wan access, so you can install some tools to comfortably change the partitioning, e.g. opkg update ; opkg install parted. Now use parted to resize the production partition to the size you like, 1GiB should be way more enough for any software you could possibly install in OpenWrt.

To make the system now go back to the production mode, you have to clear the log of the emulated crash:

rm /sys/fs/pstore/*

Now use sysupgrade to flash openwrt-23.05.2-mediatek-filogic-bananapi_bpi-r3-squashfs-sysupgrade.itb. OpenWrt should come up with lots of free space in the overlay.

Now that you set the desired size of the production partition you won’t ever have to do this again.

If you want to make use of the remaining space of the eMMC, e.g. to host container images, databases or stuff like that, I recommend you to install autopart and use uvol to create filesystem volumes for such purposes. Volumes created in that way are independent of sysupgrade mechanism, ie. data there will remain untouched.

[danoond (Dan Mikhailov)]

@dangowrt

but you will need to boot into the recovery initramfs system because you cannot resize a filesystem while it is mounted

If I understand correctly, another way I can manipulate the rootfs on eMMC is if I boot from NOR or NAND with OpenWRT installed on one of them?

@dangowrt

Now use sysupgrade to flash openwrt-23.05.2-mediatek-filogic-bananapi_bpi-r3-squashfs-sysupgrade.itb. OpenWrt should come up with lots of free space in the overlay.

Now that you set the desired size of the production partition you won’t ever have to do this again.

Is it necessary to sysupgrade in order to changes take effect? Can I simply resize.f2fs? I’ve seen people suggest doing so, what is the difference between those methods? Sysupgrade will return F2FS part of rootfs to it’s original size?

@dangowrt

If you want to make use of the remaining space of the eMMC, e.g. to host container images, databases or stuff like that, I recommend you to install autopart and use uvol to create filesystem volumes for such purposes. Volumes created in that way are independent of sysupgrade mechanism, ie. data there will remain untouched.

So just creating a new partition on eMMC’s free space and a filesystem is not an options? What are the risks of doing it this way?

I also don’t quite understand, how data is stored on the eMMC. Are the partitions physical, or they are virtual partitions (image, loop device)? I’ve read the forum for some time and have seen uLmage.FIT, loop device and virtual partition terms mentioned in BPI-R2 and R64 treads. Where can I read more about how OpenWRT manages storage?

And thank you very much for your help! I really want to fully understand the storage handling mechanisms, and your reply have shed some light on the topic for me

[dangowrt (Daniel Golle)]

@danoond

If I understand correctly, another way I can manipulate the rootfs on eMMC is if I boot from NOR or NAND with OpenWRT installed on one of them?

Yes, that is also an option if you set the dip switches correctly to still be able to access the eMMC.

@danoond

Is it necessary to sysupgrade in order to changes take effect? Can I simply resize.f2fs? I’ve seen people suggest doing so, what is the difference between those methods? Sysupgrade will return F2FS part of rootfs to it’s original size?

It’s the easiest way. The reason is simply that it will trigger re-creating the overlay filesystem covering the remaining space in the production partition after the firmware image.

You can also use resize.f2fs (after another reboot, so the FIT partition parser also takes note of the change).

@danoond

So just creating a new partition on eMMC’s free space and a filesystem is not an options? What are the risks of doing it this way?

No risk, you can just to that as well, it just involves manually creating that partition, creating the filesystem and configuring a mountpoint. autopart and uvol make all that happen automatically.

@danoond

I also don’t quite understand, how data is stored on the eMMC. Are the partitions physical, or they are virtual partitions (image, loop device)? I’ve read the forum for some time and have seen uLmage.FIT, loop device and virtual partition terms mentioned in BPI-R2 and R64 treads. Where can I read more about how OpenWRT manages storage?

In general OpenWrt uses a read-only squashfs as rootfs and a read/write overlay on top of that. How that works on a specific device depends on the read/write filesystem suitable for the backing storage (ie. ext4 on block device, JFFS2 on NOR flash, UBIFS on NAND flash) as well as on the requirements of the (stock) bootloader. As for the Bananapi routers we are building the bootloader from source as well, we were free to create a boot method allowing the same uImage.FIT being used on NOR, NAND and eMMC or SD card, while making sure the bootloader also validates the complete firmware image (incl. the squashfs) before starting, falling back to a recovery image in case of validation failing.

tl;dr: The content of the production partition dynamically split into rootfs (mmcblk0p65) and rootfs_data (mmcblk0p66) which aren’t actual GPT partitions, but rather represent the squashfs-part of the uImage.FIT (rootfs) as well as all the remaining space in the partition to be used as read/write overlay (rootfs_data).

This mechanism is now used on many recently added devices as it comes with a couple of advantages:

• instead of hardcoding sizes for kernel, rootfs and rootfs_data partitions, allocated space dynamically avoiding arbitrary limitation and/or gaps of unused/lost storage.
• the same image can be used accross storage types (NOR, NAND, eMMC/SD)
• the same image (*sysupgrade.itb) can be used to flash the device from within OpenWrt (using sysupgrade or Firmware Upgrade in LuCI) as well as when using the bootloader to load an image to be flashed via TFTP.
• the whole image (incl. the squashfs rootfs) is validated by the bootloader before starting the kernel, allowing for a meaningful dual-boot mechanism (previously U-Boot was only able to validate kernel and device tree blob – but without an intact rootfs the machine would run into a bootloop, in the worst case)
• in case of configurable hardware (such as BPi router boards allowing to select from several storage options) we can use device tree overlays embedded into that single image to be applied by the bootloader.

[univert]

I am trying to create two production partition and dual boot the emmc device on R3. I copied over the itb file onto these two production partition.

I am able to boot both of them by setting uboot environment variable part_default However It seems the rootfs(mmcblk0p65) and rootfs_data(mmcblk0p65) is not correct. They always point to the first production partition(mmcblk0p5) even though I am booting from the second production partition(mmcblk0p6) How to actually let the rootfs and rootfs_data to point to the production partition that uboot is choosing to boot from?

[dangowrt (Daniel Golle)]

As the FIT partition parser approach has been rejected upstream I’ve been working on a uImage.FIT block driver which can be told via a phandle in device tree where to look for the image.

block: add uImage.FIT subimage block driver

With this you would be able to tell the kernel which of the two production volumes should be used – but it is not part of OpenWrt yet and doing the switch from the current partition parser approach to the block driver approach will not be trivial for the BananaPi boards which offer multiple boot device options.

With the current partition parser approach all you could do is to change the partition type UUID in the GPT each time you change the production volume in use. Also for that you don’t have anything ready-made at this point, you will have to implement that youself in U-Boot.

[univert]

I am able to boot secondary production partition once I changed to the UUID type to cae9be83-b15f-49cc-863f-081b744a2d93 and the FIT parser recognize the FIT subparts as mmcblk0p65 mmcblk0p66 mmcblk0p67 mmcblk0p68 I boot the secondary production partition by changing uboot var bootargs to root=/dev/mmcblk0p67

However the rootfs_data is not not overlayed properly after I boot into the secondary production partition. It should overlay mmcblk0p68 over mmcblk0p67. However it still uses mmcblk0p66 as overlay.

root@OpenWrt:/# lsblk -i
NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mtdblock0        31:0    0 122.5M  0 disk
mtdblock1        31:1    0     2M  1 disk
mtdblock2        31:2    0     3M  0 disk
mtdblock3        31:3    0   512K  1 disk
mmcblk0         179:0    0   7.3G  0 disk
|-mmcblk0p1     179:1    0   512K  0 part
|-mmcblk0p2     179:2    0     2M  0 part
|-mmcblk0p3     179:3    0     4M  0 part
|-mmcblk0p4     179:4    0    32M  0 part
|-mmcblk0p5     179:5    0 180.1M  0 part
|-mmcblk0p6     179:6    0   246M  0 part
|-mmcblk0p65    259:0    0    30M  1 part
|-mmcblk0p66    259:1    0 145.2M  0 part /overlay
|-mmcblk0p67    259:2    0    30M  1 part /rom
|-mmcblk0p68    259:3    0 211.1M  0 part
`-mmcblk0p128   259:4    0     4M  0 part
mmcblk0boot0    179:8    0     4M  1 disk
mmcblk0boot1    179:16   0     4M  1 disk
ubiblock0_2     254:0    0    35M  0 disk
`-ubiblock0_2p1 259:5    0    30M  1 part
emphasized text

[dangowrt (Daniel Golle)]

True, that’s another way, but you will need to hack fstools which currently simply take the first partition with label rootfs_data it can find.