BPi-R3 + OpenWRT: what's the best way to use the rest of my space on my SD card?

Banana Pi BPI-R4/BPI-R3 Mini/BPI-R3/BPI-R64/BPI-R2
1

Running OpenWRT latest build (6/10) on a 128GB SD card. I’d like to be able to install software without worrying about overrunning the 100MB root partition. Extroot instructions don’t work and instructions for messing with overlay in order to do it seem to make incorrect assumptions about what paths are already in /overlay (i.e. ignore “upper” and “work.”). I tried resizing the production partition in gparted but it won’t do it because all the filesystems are read as “unknown” even after installing f2fs-tools and squashfs-tools-ng. What is the right way to do this? Thanks.

2

I think I have narrowed down the issue to this:

[   11.019758] UBIFS (ubi0:5): Mounting in unauthenticated mode
[   11.025492] UBIFS (ubi0:5): background thread "ubifs_bgt0_5" started, PID 697
[   11.517591] UBIFS (ubi0:5): UBIFS: mounted UBI device 0, volume 5, name "rootfs_data"
[   11.525414] UBIFS (ubi0:5): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
[   11.535306] UBIFS (ubi0:5): FS size: 91168768 bytes (86 MiB, 718 LEBs), max 729 LEBs, journal size 4571136 bytes (4 MiB, 36 LEBs)
[   11.546930] UBIFS (ubi0:5): reserved for root: 4306125 bytes (4205 KiB)
[   11.553526] UBIFS (ubi0:5): media format: w5/r0 (latest is w5/r0), UUID 864E1E15-9008-4117-9DE5-7AE7C7182ED7, small LPT model
[   11.572347] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
[   11.595934] block: unable to load configuration (fstab: Entry not found)
[   11.602667] block: attempting to load /tmp/ubifs_cfg/etc/config/fstab
[   11.609124] block: unable to load configuration (fstab: Entry not found)
[   11.615836] block: attempting to load /etc/config/fstab
[   11.621105] block: unable to load configuration (fstab: Entry not found)
[   11.627805] block: no usable configuration

But I have no idea how to fix it.

3

This is an ongoing problem, and an argument I have had with the OpenWrt devs. Unfortunately they have not seen fit to change the default partition size to something reasonable. This should be part of a pinned FAQ here because every new user runs into this issue.

You can’t use parted or gparted to do the resizing, because they change other things in the partition table and it won’t boot right. You have to use plain fdisk and do it manually.

Resizing the partition is tricky - it’s doable, and I’ll explain how, but it’s tricky. The better solution, if you have the skill, is to build your own image. You can select the size of the root partition when you build it yourself. Once you create your custom config file with the new partition size, you can save it and re-use it on subsequeny builds.

To do the actual resize, you’ll need either:

  • A second sdcard, along with a USB sdcard reader
  • To set up your device to boot from NAND flash

This is because you need to use the device itself to resize the partition, and the partition can’t be active when you do. So you need to boot off NAND, or boot off one sdcard while you resize the other. If you have the equipment, a second sdcard is easier. The instructions are going to assume that - adjust as required if you’re going the NAND route:

  1. Boot your R3 using a temp sd card
  2. Install the needed packages
    opkg install f2fs-tools f2fsck mkf2fs fdisk kmod-usb3 kmod-usb-storage kmod-usb-storage-uas parted
    We’re not actually using parted, but we need it because it has partprobe which we will use.
  3. Plug the sdcard you are going to use into your reader and connect it to the R3’s USB port. I’m going to assume it’s /dev/sda - adjust as required below if it’s not.
  4. Copy the sdcard OpenWrt image to the sdcard
  5. Reload the card’s partition table
    partprobe /dev/sda
  6. fdisk /dev/sda
  7. Print the partition table with the “p” command. Then switch to extended mode “x” and print the partition table in extended mode “p” before returning to normal “r”
  8. Copy and paste both partition table printouts to a text file.
  9. Delete partition 7 “d”
  10. Create a new partition 7 “n”, start it on the same sector as the old one (likely 131072), but have the end point further down. I use a 2GiB sized partition for mine, so if 131072 was your start the end would then be 4325375
  11. Set the type of partition 7 to what it was - use the “t” command, and set it to the type UUID it was before, you’ll see it in the expert printout you made - likely it’s “CAE9BE83-B15F-49CC-863F-081B744A2D93”
  12. Switch to expert mode “x”
  13. Set the UUID of partition 7 to what it was bedore “u” - likely “5452574F-2211-4433-5566-778899AABB07”
  14. Set the name of partition 7 to what it was before “n” - this will be “production”
  15. Print out the partition “p” in while still in expert mode and check to make sure it matches the old partition 7 in every way except the ending sector
  16. Return to normal mode “r”, then write the partition table “w”
  17. At this point I usually sync and remove the USB sdcard reader and then plug it back in because sometimes when the partition table is soft-read back after fdisk writes it the FIT table doesn’t read back correctly.
  18. Now you need to resize the virtual overlay partition, which should be /dev/sda66:
    resizef2fs /dev/sda66
  19. Sometimes the above doesn’t work unless you first mount and dismount the overlay:
    mkdir /mnt/tmp
    mount /dev/sda66 /mnt/tmp -t f2fs
    umount /mnt/tmp
  20. sync, remove the sdcard reader, halt the device, power down, and swap cards. Hopefully it is now large enough to be useful.

EDIT: I haven’t actually done this in some time, so this was from memory, so apologies in advance if I’ve missed anything

1 Like
6

Agreed on all points. I switched from OpenWRT to real Linux in large part because of the OpenWRT team’s insistence on assuming every device has less storage than my first PC 30 years ago. You will be better off taking the wifi configs generated by OpenWRT (they’re at /tmp/run/hostapd-*.conf) and installing hostapd on a real Linux distribution. It may seem daunting to set everything up yourself, but I promise it is not that hard, and there are so. many. benefits. for doing so, far beyond being able to use all the space on that big SD card.

7

for me the main draw of OpenWRT is LuCi. installing the Ubuntu image is tempting but I think not having the ease of management would get frustrating after a while.

8

I personally don’t go this far - I have found ways to work around OpenWrt’s self-imposed limitations OpenWrt puts in. OpenWrt’s other benefits outweigh even dumb decisions like this. Plus if users like us leave, they loose voices for change. But that being said, I don’t blame people for switching.

I realized I posted about resizing the partition without actually speaking to the title of the thread. You want to know how best to use the entire card.

You are using a 128GB card - I use 64GB cards myself, but the process I use will work for you too. In my post I suggested a 2GB OpenWrt overlay partition. I suggest an OpenWrt overlay partition this size because 1) it’s enough for any number of packages, and 2) so you can use the bulk of the space on the card for a persistent partition that stays the same even after you reimage with new OpenWrt releases. In general terms, here is my card’s layout:

  1. Space for the OpenWrt image
  2. Buffer space in case #1 changes in size
  3. Persistent “workbench” partition
  4. Swap space
  5. More buffer space in case I want to move to a different manufacturer’s SD card (because the exact size of two SD cards with the same capacity is often different)

Here is the actual partition layout of my SD card:

Disk /dev/mmcblk0: 58.27 GiB, 62562238464 bytes, 122191872 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5452574F-2211-4433-5566-778899AABB00

Device              Start       End   Sectors  Size Type
/dev/mmcblk0p1         34      8191      8158    4M Linux filesystem
/dev/mmcblk0p2       8192      9215      1024  512K Linux filesystem
/dev/mmcblk0p3       9216     13311      4096    2M Linux filesystem
/dev/mmcblk0p4      13312     21503      8192    4M EFI System
/dev/mmcblk0p5      24576     90111     65536   32M EFI System
/dev/mmcblk0p6      90112    131071     40960   20M EFI System
/dev/mmcblk0p7     131072   4325375   4194304    2G unknown
/dev/mmcblk0p8    4325376   8388607   4063232  1.9G Linux reserved
/dev/mmcblk0p9    8388608 109051903 100663296   48G Linux filesystem
/dev/mmcblk0p10 109051904 117440511   8388608    4G Linux swap
/dev/mmcblk0p11 117440512 122189823   4749312  2.3G Linux reserved

Here is the same device shown in fdisk’s expert mode:

Disk /dev/mmcblk0: 58.27 GiB, 62562238464 bytes, 122191872 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5452574F-2211-4433-5566-778899AABB00
First usable LBA: 34
Last usable LBA: 122191838
Alternative LBA: 122191871
Partition entries starting LBA: 2
Allocated partition entries: 128
Partition entries ending LBA: 33

Device              Start       End   Sectors Type-UUID                            UUID                                 Name       Attrs
/dev/mmcblk0p1         34      8191      8158 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5452574F-2211-4433-5566-778899AABB01 bl2        RequiredPartition LegacyBIOSBootable
/dev/mmcblk0p2       8192      9215      1024 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5452574F-2211-4433-5566-778899AABB02 ubootenv   RequiredPartition
/dev/mmcblk0p3       9216     13311      4096 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5452574F-2211-4433-5566-778899AABB03 factory    RequiredPartition
/dev/mmcblk0p4      13312     21503      8192 C12A7328-F81F-11D2-BA4B-00A0C93EC93B 5452574F-2211-4433-5566-778899AABB04 fip        RequiredPartition
/dev/mmcblk0p5      24576     90111     65536 C12A7328-F81F-11D2-BA4B-00A0C93EC93B 5452574F-2211-4433-5566-778899AABB05 recovery   RequiredPartition
/dev/mmcblk0p6      90112    131071     40960 C12A7328-F81F-11D2-BA4B-00A0C93EC93B 5452574F-2211-4433-5566-778899AABB06 install    RequiredPartition
/dev/mmcblk0p7     131072   4325375   4194304 CAE9BE83-B15F-49CC-863F-081B744A2D93 5452574F-2211-4433-5566-778899AABB07 production 
/dev/mmcblk0p8    4325376   8388607   4063232 8DA63339-0007-60C0-C436-083AC8230908 5452574F-2211-4433-5566-778899AABB08 dmz        
/dev/mmcblk0p9    8388608 109051903 100663296 0FC63DAF-8483-4772-8E79-3D69D8477DE4 5452574F-2211-4433-5566-778899AABB09 workbench  
/dev/mmcblk0p10 109051904 117440511   8388608 0657FD6D-A4AB-43C4-84E5-0933C84B4F4F 5452574F-2211-4433-5566-778899AABB0A sdswap     
/dev/mmcblk0p11 117440512 122189823   4749312 8DA63339-0007-60C0-C436-083AC8230908 5452574F-2211-4433-5566-778899AABB0B dmz

Partitions 1-7 come from the OpenWrt sdcard image. Partitions 8-11 I added myself. 8 is the buffer betweeen the OpenWrt partitions and my persistent workbench partition (I labelled it “dmz”), 9 is the workbench itself, 10 is swap space, and 11 is the end buffer.

This way when a new OpenWrt image comes out, I can copy it directly over my existing card, and know that even if it grows a little it won’t overwrite my persistent partition. This will hammer the partition table, but that’s ok, since I know where to rebuild it (I have a setup file with the partition layouts). So I just recreate partition 8 to start wherever partition 7 ends and then 8 will always end in the same spot. I can then remake partition 9 to start and end in exactly the same spot, and voila, all my persistent data is back again.

In my configuration process for a new build, I replace /root with a softlink to /mnt/workbench/root, which gives me a persistent home directory. Once I install the same packages I had before, recreate that /root link, and restore the configuration, in 5 minutes I have a new build/release that is functionally identical.

9

I think in the original version of your post you linked to image build instructions - do you still have those somewhere?

10

I didn’t, but here they are:

Here are the actual steps I use to build master into a snapshot:

git clone https://git.openwrt.org/openwrt/openwrt.git
cd openwrt
git pull
git checkout master
./scripts/feeds update -a
./scripts/feeds install -a
#wget https://downloads.openwrt.org/snapshots/targets/mediatek/filogic/config.buildinfo -O .config
#cp -a ~/devel/wrt/diffconfig_mt7986_alluserspace_v3 .config
cp -a ~/devel/wrt/diffconfig_mt7986 .config
wget https://va1der.ca/~public/openwrt/llvm/llvm-bpf-15.0.7.Linux-x86_64.tar.xz
tar -xaf llvm-bpf-15.0.7.Linux-x86_64.tar.xz
make defconfig
# To test the kernel fingerprint do the following (the first one will fail because the toolchain isn't built yet, this is expected)
#make target/linux/compile
#find build_dir/ -name .vermagic -exec cat {} \;
make -j $(nproc) toolchain/install
make -j $(nproc) download

# mt76 eeprom.c patch to allow driver to load the WiFi eeprom from a file - totally optional but useful
mkdir package/kernel/mt76/patches
wget https://va1der.ca/~public/openwrt/patches/mt76/999-mt76_eeprom_file_220302.diff -O package/kernel/mt76/patches/999-mt76_eeprom_file_220302.diff

make -j $(nproc) world

Here are the configuration files referenced above that I use for the R3:

The first one builds just a standard image. It is based on the stock snapshot configuration with the following changes:

  • Adds mkfs.f2fs, f2fs, ntfs3, and ntfs3.mount to the image
  • Changes LLVM setting to use the pre-supplied one rather than building it from source, saves about half the compile time
  • Makes the default wpad one that uses WolfSSL and Wolf’s hardware encryption acceleration library
  • Changes stripping to one that is friendly to libraries
  • Adds LuCI
  • Changes root partition size to 2048MiB

The second one is basically just the first one but also set to build ALL userspace packages (in case you want to self host all your packages) with a couple esoteric packages that cause problems deselected. This will take a long time to build, and there are often one or two packages with build problems at any particular time.

2 Likes
11

this rules, thank you. I really appreciate your going to the effort of documenting this so thoroughly. I hope someone at BPi is paying attention, because all this should be added to the wiki!

12

OpenWRT’s parted worked for me. Below are my steps. It’s a shame that p65 and p66 are only accessible from inside OpenWRT. The devs should make them regular partitions.

Thanks to VA1DER for all the important bits! I agree that his work should be on the OpenWRT wiki or at least link to these threads.

parted -s /dev/mmcblk0 resizepart 7 100%

# reboot into NAND (up, down, up, up)

# necessary if you get this error:
# [f2fs_do_mount:3651] Mount unclean image to replay log first
mkdir -p /mnt/tmp && mount /dev/mmcblk0p66 /mnt/tmp -t f2fs && umount /mnt/tmp

resize.f2fs /dev/mmcblk0p66

# reboot into SD card (up, up, up, up)
1 Like