Good the hear that all of this has already been figured out.
I was wondering though, it does not seem to use extlinux.conf, or does it? Which other way do you use to boot kernel from u-boot?
That’s why I want to install uboot on NoR and rootfs on m.2. This is possible by adding the appropriate BOOT_TARGET_DEVICES when building uboot, right?
Where is kernel located? Uboot cannot access nvme
Yes uboot can access nvme in general,but not on r3 as pcie driver is missing for mt7986
You can, but you need to setup a boot partition on emmc, where U-Boot can load the kernel Image from.
But then it also wouldn’t hurt to just use emmc for ATF and U-Boot also, instead of NOR. That only needs 1MB.
Bit short on time this week, will have more on the weekend to explain things and upload code. Everything works (including LTE, WiFi, Ethernet with VLANs/VXLAN/whatever, also have a custom ZigBee HAT with Zigbee2MQTT working as well as RTC shenanigans), but there are issues with the stock NixOS modules (for example the official hostapd module only allows one hostapd instance). The code on that branch is also not up-to-date, I have some newer stuff locally based on 6.3 which has quite a few less patches. I’ve also been trying to upstream some of my kernel changes.
I’m actually quite interested in just using kexec to boot these, i.e. TF-A -> Linux without u-boot (or some minimal shim, maybe coreboot? not sure) and then kexec to the main environment. This would fix the issue that u-boot needs drivers for everything.
Yes, it does use extlinux.conf. Custom u-boot.
Here is my fork of atf that can boot kernel without u-boot (also can use bootpartition instead of fip partition) i’ll update it later today to latest changes.
I have not tested kexec on R3 (yet). It could reveal bugs in drivers as it starts kernel without hard-reset.
I’ve pulled the relevant parts from my NixOS flake that describes how I bootstrapped NixOS onto my BPi-R3 with an SD Card and put them into a repo.
The u-boot there has a couple of patches, they’re rather ugly, and you should probably take a look at them before you use them. The memory allocations to get distro bootcmd/extlinux.conf working might be brittle; I’m not exactly sure why I’ve had to add those details there, nor why relocation was so brittle to always fail with my kernels, I thought they should be coming from the device tree.
The kernel was patched up before 6.3 released, and should be revisited, though not all the requisite patches made it into 6.3, but a good number of them did. The config is a file because I couldn’t get a working system with structuredExtraConfig, I needed to set some options to N but that was disallowed as a dependency was only allowing M or Y that couldn’t be changed until later in the config process. I can’t recall if this was only an issue because I didn’t want 12hr+ compile times as I’m compiling on an SBC, or if I needed to disable some options to get the BPiR3 functional.
Once built with nix build -L '.#nixosConfigurations.bpir3.config.system.build.sdImage', you should get an image in ./result that’s good for flashing onto a SD card.
2 Likes
I’m trying to find where you add stuff like:
#include <config_distro_bootcmd.h>
Trying learn which way best to implement distro-boot. But I cannot find it in your commits. Do you use a custom script/env as well?
what do you use to build the system? and how long does it take?
not really as you need the pcie/nvme driver in ATF then 
Best solution so far: Use emmc for boot partition. Updating kernel image file once a week, lets say 20MB, and using the buildin wear leveling. It will take 8 years to have all 8GB written to only once. At this rate, the emmc will fail somewhere during the next ice age. 
Nix doesn’t do cross-compilation, so any aarch64 system with enough memory for the nix command to and still have enough memory for the compiler to operate. 1GB might work, but you probably are better off with 2GB or more.
I build on a RPi4 with 4GB and a A2 rated 64GB SanDisk Extreme. The kernel build takes an hour or two and isn’t bottle-necked on the SD card too heavily, everything else spends a ton of time in disk-wait. Re-building the root-fs and SD image takes about 20-30 minutes for me.
If you don’t have another aarch64 system available to act as a bootstrap, you can install the other Linux distros available and install the nix command on them, then build your nix image.
I’ve also produced a NixOS SD image you can use to get started if you prefer that. Keep in mind that nixos-rebuild will never update ATF or u-boot for you however, this is normal behaviour for it on SBC devices.
Thanks for the answer. I first tried binfmt on my x86_64 laptop. It was a bad idea, after 4 hours the kernel never built. The second idea was to use nix-on-droid on my pixel 4a. It took ~35 minutes to build the kernel. It seems to be the easiest and most affordable solution.
Thanks for it, I am going to try it this weekend. One question, I was wondering if it wouldn’t be better to run it from the emmc, although @lorenz said that 8GB might be tight. What is your thought on that? Are you going to run it from SD card only?
1 Like
8GB is doable if you are using a remote system for building. My setup ends up weighing about 3.1GB, which is fairly heavy as it’s pulling in a lot of unnecessary things.
I’m conflicted about running from eMMC, it’s not large enough that the speed of it over an SD card provides an advantage that offsets the extra effort to keep the nix store small, and provisioning it is cumbersome with the required pivot through NAND or NOR(more on this later). The size of the store can be made into a non-issue if the eMMC is for boot only and everything else is on the NVMe drive, however that increases provisioning complexity further. If I end up with a non-bootable system, I’d have to pull it off the wall, crack the case, and hookup to the UART, or spend time getting some u-boot scripting solid for a working automatic fallback method.
The SD card on the other hand I can access in-situ and either re-image or edit the extlinux.conf to switch the boot generation; it’s not elegant, but it is easy and the next nixos-rebuild will clean that up. My goal is to treat the SD card as a stateless golden-image, though that isn’t without challenges; for example, I deal with MAC addresses with custom patches to u-boot. Some other information I’d like to store on one of the flashes, but I’m having trouble with that right now, so the SD card isn’t completely stateless yet.
As for the pivot through NAND or NOR, I can’t do that from my nix install currently as I cannot write to the NAND or NOR without the data corrupting significantly. I haven’t attempted to look into this yet, but seeing as the factory provisioned images on those devices booted OpenWRT successfully I expect it’s a software issue.
For me, I probably am going to run it from the SD card, with impermanence and the NVMe mounted up to handle persistence, large files and things like logs…
Thanks for that thoughtful and detailed answer, that’s exactly what I needed.
For me, I probably am going to run it from the SD card, with impermanence and the NVMe mounted up to handle persistence, large files and things like logs…
That will be my plan as well.
If you end up doing that, and by any chance you will have it opensourced somewhere please let me know. I will be eager to steal to learn from your configuration 
Hi,
I decided to write a blogpost based on the information gathered in this thread, and the great work that @Nakato did. I hope that it will be useful for people like me who want to run NixOS on yet unsupported board, as well as for more experienced linux users who don’t know much about nix ecosystem yet.
You can find it here: https://github.com/ghostbuster91/blogposts/blob/main/router2023/main.md