How to create initrd/initramfs for use with tftp

Banana Pi Router design
41

I don’t know why it doesn’t work for you, therefore I just made a almost ready-to-go package for you to use. There are some #TODO’s in the Makefile you have to fill in such as your PATH to the compiler, SYSROOT, etc. SYSROOT is a path to the folder that your compiler uses to link the applications to, e.g. with the glibc/uclibc libraries. All binaries in that folder are compiled for the target platform. In my case it’s in: ./arm-bpir2-linux-gnueabihf/arm-bpir2-linux-gnueabihf/sysroot

My toolchain is built with crosstool-ng, but linaro should also work perfectly if you got other stuff working on the R2 already.

The package uses busybox-1.28.2, since I was using this version back then. You could try to replace it with latest greatest after you get the package to work. I’m just using the busybox defconfig, no additional settings (for now). I added some minimum required files in the overlay already. The compiled initramfs.cpio file should boot on your board.

In summary: Extract the package, adjust the variables in Makefile and run make -jXX where XX is your number of threads * 1.5

frank-w-initramfs.tar.gz (1.2 KB)

1 Like
42
Details 
frank@frank-N56VZ:~
[09:20:18]$ which arm-linux-gnueabihf-gcc
/usr/bin/arm-linux-gnueabihf-gcc
frank@frank-N56VZ:~
[09:22:49]$ arm-linux-gnueabihf-gcc -v
Using built-in specs.
COLLECT_GCC=arm-linux-gnueabihf-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc-cross/arm-linux-gnueabihf/7/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 7.3.0-27ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --disable-libquadmath-support --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-multiarch --enable-multilib --disable-sjlj-exceptions --with-arch=armv7-a --with-fpu=vfpv3-d16 --with-float=hard --with-mode=thumb --disable-werror --enable-multilib --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=arm-linux-gnueabihf --program-prefix=arm-linux-gnueabihf- --includedir=/usr/arm-linux-gnueabihf/include
Thread model: posix
gcc version 7.3.0 (Ubuntu/Linaro 7.3.0-27ubuntu1~18.04)

i guess sysroot is then /usr/arm-linux-gnueabihf/lib/ … i have no glibc/ulibc only libc, crosscompiler itself uses sysroot=/ so i try without setting it (only crosscompile-prefix)

Details 
frank@frank-N56VZ:~
[09:19:56]$ find / - name '*glibc*' 2>/dev/null | grep eabihf | grep libc
/usr/lib/gcc-cross/arm-linux-gnueabihf/5/libcc1.so
/usr/lib/gcc-cross/arm-linux-gnueabihf/5/plugin/libcc1plugin.so
/usr/lib/gcc-cross/arm-linux-gnueabihf/5/plugin/libcc1plugin.so.0
/usr/lib/gcc-cross/arm-linux-gnueabihf/5/plugin/libcc1plugin.so.0.0.0
/usr/lib/gcc-cross/arm-linux-gnueabihf/6/libcc1.so
/usr/lib/gcc-cross/arm-linux-gnueabihf/6/plugin/libcc1plugin.so
/usr/lib/gcc-cross/arm-linux-gnueabihf/6/plugin/libcc1plugin.so.0
/usr/lib/gcc-cross/arm-linux-gnueabihf/6/plugin/libcc1plugin.so.0.0.0
/usr/lib/gcc-cross/arm-linux-gnueabihf/7/libcilkrts.spec
/usr/lib/gcc-cross/arm-linux-gnueabihf/7/libcc1.so
/usr/lib/gcc-cross/arm-linux-gnueabihf/7/libcilkrts.a
/usr/lib/gcc-cross/arm-linux-gnueabihf/7/libcilkrts.so
/usr/lib/gcc-cross/arm-linux-gnueabihf/7/plugin/libcc1plugin.so
/usr/lib/gcc-cross/arm-linux-gnueabihf/7/plugin/libcc1plugin.so.0
/usr/lib/gcc-cross/arm-linux-gnueabihf/7/plugin/libcp1plugin.so.0
/usr/lib/gcc-cross/arm-linux-gnueabihf/7/plugin/libcp1plugin.so
/usr/lib/gcc-cross/arm-linux-gnueabihf/7/plugin/libcp1plugin.so.0.0.0
/usr/lib/gcc-cross/arm-linux-gnueabihf/7/plugin/libcc1plugin.so.0.0.0
/usr/lib/gcc-cross/arm-linux-gnueabihf/8/libcc1.so
/usr/lib/gcc-cross/arm-linux-gnueabihf/8/plugin/libcc1plugin.so
/usr/lib/gcc-cross/arm-linux-gnueabihf/8/plugin/libcc1plugin.so.0
/usr/lib/gcc-cross/arm-linux-gnueabihf/8/plugin/libcp1plugin.so.0
/usr/lib/gcc-cross/arm-linux-gnueabihf/8/plugin/libcp1plugin.so
/usr/lib/gcc-cross/arm-linux-gnueabihf/8/plugin/libcp1plugin.so.0.0.0
/usr/lib/gcc-cross/arm-linux-gnueabihf/8/plugin/libcc1plugin.so.0.0.0
/usr/arm-linux-gnueabihf/lib/libcidn.so.1
/usr/arm-linux-gnueabihf/lib/libcrypt.so.1
/usr/arm-linux-gnueabihf/lib/libc.so
/usr/arm-linux-gnueabihf/lib/libcidn-2.27.so
/usr/arm-linux-gnueabihf/lib/libc_nonshared.a
/usr/arm-linux-gnueabihf/lib/libc.a
/usr/arm-linux-gnueabihf/lib/libc-2.27.so
/usr/arm-linux-gnueabihf/lib/libc.so.6
/usr/arm-linux-gnueabihf/lib/libcilkrts.so.5
/usr/arm-linux-gnueabihf/lib/libcrypt.so
/usr/arm-linux-gnueabihf/lib/libcrypt.a
/usr/arm-linux-gnueabihf/lib/libcidn.so
/usr/arm-linux-gnueabihf/lib/libcrypt-2.27.so
/usr/arm-linux-gnueabihf/lib/libcilkrts.so.5.0.0
/usr/arm-linux-gnueabihf/include/bits/libc-header-start.h
/usr/arm-linux-gnueabihf/include/linux/libc-compat.h
/usr/arm-linux-gnueabihf/include/gnu/libc-version.h

seems crosscompiler is not used yet :frowning:

changed block in defconfig to this (and made make bpi_r2_defconfig again):

  49 CONFIG_CROSS_COMPILER_PREFIX="arm-linux-gnueabihf-"
  50 CONFIG_SYSROOT=""
  51 CONFIG_EXTRA_CFLAGS=""
  52 CONFIG_EXTRA_LDFLAGS=""
  53 CONFIG_EXTRA_LDLIBS=""

you package is only for pack the initramfs itself (with overlay) after buildroot-compile is successful, right

43

I didn’t set CROSS_COMPILE in the busybox config, for me it is definitely working exactly as written in the package.

Can you run the following command to check if this file is built for ARM:

file /usr/arm-linux-gnueabihf/lib/libc.so

If yes, try this configuration:

export CROSS_COMPILE:=arm-linux-gnueabihf-
#export PATH <- because you have it in your PATH already
SYSROOT=/usr/arm-linux-gnueabihf

You should be able to build it without any further changes to the package. If you check here, you see it should accept the CROSS_COMPILE environment variable.

44 
frank@frank-N56VZ:/media/data_ext/buildroot
[12:00:08]$ file /usr/arm-linux-gnueabihf/lib/libc.so
/usr/arm-linux-gnueabihf/lib/libc.so: ASCII text
frank@frank-N56VZ:/media/data_ext/buildroot
[12:00:38]$ cat /usr/arm-linux-gnueabihf/lib/libc.so
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf32-littlearm)
GROUP ( /usr/arm-linux-gnueabihf/lib/libc.so.6 /usr/arm-linux-gnueabihf/lib/libc_nonshared.a  AS_NEEDED ( /usr/arm-linux-gnueabihf/lib/ld-linux-armhf.so.3 ) )
frank@frank-N56VZ:/media/data_ext/buildroot
[12:00:45]$ file /usr/arm-linux-gnueabihf/lib/libc.so.6
/usr/arm-linux-gnueabihf/lib/libc.so.6: symbolic link to libc-2.27.so
frank@frank-N56VZ:/media/data_ext/buildroot
[12:01:38]$ file  /usr/arm-linux-gnueabihf/lib/ld-linux-armhf.so.3
/usr/arm-linux-gnueabihf/lib/ld-linux-armhf.so.3: symbolic link to ld-2.27.so
frank@frank-N56VZ:/media/data_ext/buildroot
[12:01:54]$ file  /usr/arm-linux-gnueabihf/lib/ld-2.27.so 
/usr/arm-linux-gnueabihf/lib/ld-2.27.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=fe0c7d2a3475029fdc1126c1d62390410fd2d791, stripped

ahh…you have a separate makefile for build buildroot…i thought i have to change it in your defconfig

genfs.sh seems not to run make…do i have to put makefile into a empty folder an run make for building buildroot and run genfs to pack the rootfs+overlay?

45

Perfect.

No, make runs genfs.sh

I am not using buildroot, just busybox for now. Buildroot has (much) more configuration.

Just extract the tar.gz into it’s own folder, adjust the variables in top of the Makefile (as above), and run make.

46 
frank@frank-N56VZ:/media/data_ext/buildroot/frank-w-initramfs
[12:27:25]$ file output/rootfs/bin/busybox 
output/rootfs/bin/busybox: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=92167abc8b0c51637ce0d2901a8ed7c1be206777, stripped

seems to work so far…also a initramfs.cpio is there…ok, i tried to start with buildroot because hostapd will be nice :wink: but i start with busybox…i have to include my modules in overlay-fs? direct under lib/modules or like debian/ubuntu in lib/modules/$(uname-r)?

47

I would install the modules directly to output/rootfs, before genfs.sh is executed. I already added some comments in the Makefile for you.

See chapter 2.3

48

how can i access current dir in make file?

then i can do something like this:

12 linux-modules:
13     # TODO: build linux modules and install to ./output/rootfs
14     kerneldir=../../bpi-r2-kernel/github
15     INSTALL_MOD_PATH=$(pwd)/output/rootfs/lib/modules
16     make -C $(kerneldir) modules_install
17 .PHONY: busybox
49

use ${PWD}

remove /lib/modules from INSTALL_MOD_PATH.

put the INSTALL_MOD_PATH on the same line as make (as parameter of make)

50 
14     kerneldir=../../bpi-r2-kernel/github
15     #INSTALL_MOD_PATH=${pwd}/output/rootfs/lib/modules
16     make -C $(kerneldir) modules_install INSTALL_MOD_PATH=${pwd}/output/rootfs/

mhm, seems kerneldir is empty…not my value i set above

51

Try { } brackets instead of ( )

52

also empty…${pwd} is also empty

frank@frank-N56VZ:/media/data_ext/buildroot/frank-w-initramfs
[16:34:04]$ make linux-modules
# TODO: build linux modules and install to ./output/rootfs
kerneldir=../../bpi-r2-kernel/github
#INSTALL_MOD_PATH=/output/rootfs/lib/modules
make -C  modules_install INSTALL_MOD_PATH=/output/rootfs/
make[1]: *** modules_install: No such file or directory.  Stop.
Makefile:13: recipe for target 'linux-modules' failed
make: *** [linux-modules] Error 2

kerneldir should be set outside the rule…i moved that line to top and this gets evaluated…but not the targetdir (${pwd})

make -C ${kerneldir} modules_install INSTALL_MOD_PATH=$(shell pwd)/output/rootfs/

seems to be the right way

ls output/rootfs/lib/modules/
4.19.0-rc1-bpi-r2-hdmiv5

can i call genfs.sh after install the modules? it seems it’s only working from the makefile, but this downloads busybox again

i have modified make file this way:

12 all: busybox initramfs
13 #output/initramfs.cpio
14
15 linux-modules:
16     # TODO: build linux modules and install to ./output/rootfs
17     #INSTALL_MOD_PATH=${pwd}/output/rootfs/lib/modules
18     make -C ${kerneldir} modules_install INSTALL_MOD_PATH=$(shell pwd)/output/rootfs/
19 .PHONY: busybox
20 busybox:
21     wget https://busybox.net/downloads/${BUSYBOX}.tar.bz2
22     tar xjf ${BUSYBOX}.tar.bz2
23     $(MAKE) -C ${BUSYBOX} defconfig
24     $(MAKE) -C ${BUSYBOX} CONFIG_PREFIX=$(realpath .)/output/rootfs install
25     rsync -ua rootfs-overlay/* output/rootfs
26
27 output/initramfs.cpio: busybox #TODO-enable-linux-modules
28     fakeroot /bin/sh genfs.sh $(SYSROOT) output/rootfs
29
30 initramfs:
31     fakeroot /bin/sh genfs.sh $(SYSROOT) output/rootfs

and run “make initramfs”

ls -lh output/initramfs.cpio 
-rw-r--r-- 1 frank frank 142M Oct  8 16:59 output/initramfs.cpio

how large can be the cpio-file to get loaded by kernel? mhm…tftp seems to stop while loading :frowning: (after 2000 #-chars)

kernel with initramfs is 64 MB (cpio gzipped by makefile)

i load kernel via

tftp 0x80200000 ${bootfile};bootm

using dnsmasq as tftp-server and tested also a 45MB-Kernel…same result

seems that dnsmasq-option tftp-no-blocksize results in loading more data (many more #-chars)…but also hang after a while…kernel doesn’t got loaded, i think i need to choose which modules are included and which don’t

details 
476K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/ata
240K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/dma
332K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/net/dsa
7.6M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/net/wireless/mediatek/mt76
7.6M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/net/wireless/mediatek
7.6M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/net/wireless
288K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/net/phy
932K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/net/ppp
612K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/net/ipvlan
268K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/net/slip
11M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/net
172K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/i2c
128K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/iio/adc
132K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/iio
152K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/input/keyboard
156K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/input
400K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/spi
788K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/thermal
120K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers/pwm
14M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/drivers
224K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/lib
1.2M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/crypto
20M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/fs/xfs
672K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/fs/nfs/filelayout
776K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/fs/nfs/flexfilelayout
18M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/fs/nfs
9.2M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/fs/cifs
160K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/fs/fat
304K	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/fs/nfs_common
6.8M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/fs/nfsd
1.5M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/fs/fuse
3.7M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/fs/lockd
2.1M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/fs/ntfs
62M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel/fs
76M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/kernel
76M	output/rootfs/lib/modules/4.19.0-rc1-bpi-r2-hdmiv5/

also 37MB-Kernel does not get loaded ;(

53

Just edit the Makefile to not download busybox and use your extracted sources, or extract an already downloaded tar.gz. I added the download to make it easier to get started.

Run du -shx * inside output/rootfs to check why the cpio file is so big. You could also run it through gzip, but I didn’t test this.

I don’t know exactly what the size limits are, but I would keep it somewhere around 10~15MB max. I could find this page though.

54

lib is currently 46 MB…have my kernel below 30MB and set ramdisk to 32MB, but still hangs during download

I can try without modules,but this makes it useless for my usecase

I guess it is a tftp-problem (dnamasq or uboot)

55

Try to decrease your lib folder by removing libs you don’t use right now. Is that 30M kernel with the (compressed) initramfs.cpio file included? Otherwise a 30M kernel is having lots of stuff included you don’t need. You are building for a dedicated platform so you only have to enable what you are really using. Yes, try without modules first to check if it even boots.

Are you sure it isn’t trying to boot the file after download is complete, but you don’t see any output?

56

The 29 mb-kernel is of course with gzipped ramfs :slight_smile: you see filelisting above (uncompressed) bis blocks are the net-modules (mostly iptables). Dropped nfs at last try (near 20mb)

57

So my initramfs is also 17.5M uncompressed. This one was booting, right?

58

I linked your initramfs in my kernel and thus works.but i don’t know resulting size (initrd was also gzipped)

removed my modules and have the basic ramfs (24M unpacked cpio), build into kernel (12.75 MB) and load it successfull via tftp (no login needed)

[17:03:34]$ grep initram .config
CONFIG_INITRAMFS_SOURCE="../initramfs.cpio"

just to confirm that it is a size based issue

[   15.641524] Run /init as init process                                        
                                                                                
Processing /etc/profile... Done                                                 
                                                                                
/ # ls                                                                          
bin      dev      include  lib      proc     sbin     usr                       
boot     etc      init     linuxrc  run      sys                                
/ #
59

I may have found another way.

What I did:

a) installed initramfs-tools

b) I ran update-initramfs -c -t -k "5.15.173-bpi-r2-main"

c) I extracted new initrd image: unmkinitramfs -v /boot/bananapi/bpi-r2/linux/initrd.img-5.15.173-bpi-r2-main /tmp/extracted_initrd

d) from x86_64 system, I took initramfs hooks for ZFS, because I need it, and because package for armhf doesn’t exist (shell scripts) and copied them to extracted directory tree

e) I used bellow to recompress the initramfs image back to cpio format:

cd /tmp/extracted_initrd
find . -print0 | cpio --null --create --format=newc >../initramfs.cpio

f) I then took resulting initramfs.cpio to the kernel source tree, updated initrd configuration to point to this file, and started rebuild. I will post once done, whether or not it was successful. But to me, it seems most natural thing towards ubuntu I have on BPI-R2.

60

It boots. And seems to be working. Although - it is not, not really, not yet. I got stuck during the boot with init, which tried to execute script I didn’t uploaded to the initrd image. Also, I was missing few libraries, so I have to repackage. But so far, according to what I saw on the console, the system requested an IP from local DHCP, set a hostname, created /dev/, /sys/ and /proc, and initiated /sbin/init, which tried to mount non-existent SD card (forgot my own fstab there…), and froze there… When I came back a bit later, it was in kernel panic screen, generic message “cant mount fs” - but I am close. At least I hope I may really be :wink: