Yes, it does use extlinux.conf. Custom u-boot.
Here is my fork of atf that can boot kernel without u-boot (also can use bootpartition instead of fip partition) iāll update it later today to latest changes.
I have not tested kexec on R3 (yet). It could reveal bugs in drivers as it starts kernel without hard-reset.
Iāve pulled the relevant parts from my NixOS flake that describes how I bootstrapped NixOS onto my BPi-R3 with an SD Card and put them into a repo.
The u-boot there has a couple of patches, theyāre rather ugly, and you should probably take a look at them before you use them. The memory allocations to get distro bootcmd/extlinux.conf working might be brittle; Iām not exactly sure why Iāve had to add those details there, nor why relocation was so brittle to always fail with my kernels, I thought they should be coming from the device tree.
The kernel was patched up before 6.3 released, and should be revisited, though not all the requisite patches made it into 6.3, but a good number of them did. The config is a file because I couldnāt get a working system with structuredExtraConfig, I needed to set some options to N but that was disallowed as a dependency was only allowing M or Y that couldnāt be changed until later in the config process. I canāt recall if this was only an issue because I didnāt want 12hr+ compile times as Iām compiling on an SBC, or if I needed to disable some options to get the BPiR3 functional.
Once built with nix build -L '.#nixosConfigurations.bpir3.config.system.build.sdImage', you should get an image in ./result thatās good for flashing onto a SD card.
2 Likes
Iām trying to find where you add stuff like:
#include <config_distro_bootcmd.h>
Trying learn which way best to implement distro-boot. But I cannot find it in your commits. Do you use a custom script/env as well?
what do you use to build the system? and how long does it take?
not really as you need the pcie/nvme driver in ATF then 
Best solution so far: Use emmc for boot partition. Updating kernel image file once a week, lets say 20MB, and using the buildin wear leveling. It will take 8 years to have all 8GB written to only once. At this rate, the emmc will fail somewhere during the next ice age. 
Nix doesnāt do cross-compilation, so any aarch64 system with enough memory for the nix command to and still have enough memory for the compiler to operate. 1GB might work, but you probably are better off with 2GB or more.
I build on a RPi4 with 4GB and a A2 rated 64GB SanDisk Extreme. The kernel build takes an hour or two and isnāt bottle-necked on the SD card too heavily, everything else spends a ton of time in disk-wait. Re-building the root-fs and SD image takes about 20-30 minutes for me.
If you donāt have another aarch64 system available to act as a bootstrap, you can install the other Linux distros available and install the nix command on them, then build your nix image.
Iāve also produced a NixOS SD image you can use to get started if you prefer that. Keep in mind that nixos-rebuild will never update ATF or u-boot for you however, this is normal behaviour for it on SBC devices.
Thanks for the answer. I first tried binfmt on my x86_64 laptop. It was a bad idea, after 4 hours the kernel never built. The second idea was to use nix-on-droid on my pixel 4a. It took ~35 minutes to build the kernel. It seems to be the easiest and most affordable solution.
Thanks for it, I am going to try it this weekend. One question, I was wondering if it wouldnāt be better to run it from the emmc, although @lorenz said that 8GB might be tight. What is your thought on that? Are you going to run it from SD card only?
1 Like
8GB is doable if you are using a remote system for building. My setup ends up weighing about 3.1GB, which is fairly heavy as itās pulling in a lot of unnecessary things.
Iām conflicted about running from eMMC, itās not large enough that the speed of it over an SD card provides an advantage that offsets the extra effort to keep the nix store small, and provisioning it is cumbersome with the required pivot through NAND or NOR(more on this later). The size of the store can be made into a non-issue if the eMMC is for boot only and everything else is on the NVMe drive, however that increases provisioning complexity further. If I end up with a non-bootable system, Iād have to pull it off the wall, crack the case, and hookup to the UART, or spend time getting some u-boot scripting solid for a working automatic fallback method.
The SD card on the other hand I can access in-situ and either re-image or edit the extlinux.conf to switch the boot generation; itās not elegant, but it is easy and the next nixos-rebuild will clean that up. My goal is to treat the SD card as a stateless golden-image, though that isnāt without challenges; for example, I deal with MAC addresses with custom patches to u-boot. Some other information Iād like to store on one of the flashes, but Iām having trouble with that right now, so the SD card isnāt completely stateless yet.
As for the pivot through NAND or NOR, I canāt do that from my nix install currently as I cannot write to the NAND or NOR without the data corrupting significantly. I havenāt attempted to look into this yet, but seeing as the factory provisioned images on those devices booted OpenWRT successfully I expect itās a software issue.
For me, I probably am going to run it from the SD card, with impermanence and the NVMe mounted up to handle persistence, large files and things like logsā¦
Thanks for that thoughtful and detailed answer, thatās exactly what I needed.
For me, I probably am going to run it from the SD card, with impermanence and the NVMe mounted up to handle persistence, large files and things like logsā¦
That will be my plan as well.
If you end up doing that, and by any chance you will have it opensourced somewhere please let me know. I will be eager to steal to learn from your configuration 
Hi,
I decided to write a blogpost based on the information gathered in this thread, and the great work that @Nakato did. I hope that it will be useful for people like me who want to run NixOS on yet unsupported board, as well as for more experienced linux users who donāt know much about nix ecosystem yet.
You can find it here: https://github.com/ghostbuster91/blogposts/blob/main/router2023/main.md