Booting from sd card. When the radios went down I was able to ssh into it over ethernet, but after rebooting to see if that would fix the issue, I am no longer even able to ssh in wired.
Since it wont come up again after the reboot maybe the image on the sd card is somehow being corrupted because the sd card is dying? I’m really not sure though because whatever is occurring is only happening during early morning hours and it seems weird to me that a damaged sd card would only malfunction during a specific time frame, and that reflashing that same card would then WORK after. Very confused atm, gonna run out and get a different sd card because the only other one I have is fast and large and I’ve heard theres issues with those.
Can you do an lspci when the radios are not available?
If they don’t show up there, look at mediatek: filogic: keep MT7988 PCIe PEXTP clocks enabled by Folkerist · Pull Request #25751 · openwrt/openwrt · GitHub
I did that earlier and they did not show up, neither did my nvme drive as well. I’ll take a look at that right now, thank you!
How do I go about testing these patches? Do I need to download something and then use the firmware builder?
Leaving instructions here (Claude helped me) for anyone who also runs into this issue and wants to try this solution.
Your router has four PCIe ports, each with its own reference clock. While booting, the kernel switches off clocks it thinks nothing is using. On the 8X that sometimes happens while the PCIe links are still starting up, so a random mix of the Wi-Fi card and NVMe drives goes missing on each boot. You can confirm it’s your problem by running this on the router:
dmesg | grep -i “link down”
If you see lines like mtk-pcie-gen3 ... PCIe link down, this is your bug.
- Check your current boot options:
fw_printenv -n bootargs
It should print something like console=ttyS0,115200n1 pci=pcie_bus_perf root=/dev/fit0 rootwait (NAND installs also have ubi.block=0,fit).
- Save a copy so you can undo it later:
fw_printenv -n bootargs > /root/bootargs.orig
- Add the option:
fw_setenv bootargs “$(fw_printenv -n bootargs) clk_ignore_unused”
- Reboot, then check that the option is active:
cat /proc/cmdline
The output should include clk_ignore_unused. Reboot a few times and check that the Wi-Fi and NVMe show up every time.
I’ve been using that patch for a while. Actually there’s also another clock to fix for USB discovery as well.
Using the patch for a few days it seems the crashes are back with a vengeance. Now multiple times daily seemingly randomly throughout the day. Its not the wifi radios going down this time, all internet access is lost even over ethernet. Rebooting fixes it. Any suggestions from anyone for what to do now that the previous patch hasnt solved it?
Imho it is the wrong thread discussing random issues.
Have you changed power supply? Sounds like such issue.
This only began after updating to the mainline snapshot a few weeks ago. Previously I was on the Wozi build but I noticed SQM didn’t work there so I tried switching to the mainline build. For the first week or so it seemed to work fine and then I had the crashing issues I previously posted about. I figured that since I’m running into these issues, others may too so I figured I’d ask here so it could be fixed. If it is not relevant enough, I can create a new thread if you want. I have not changed the power supply or anything else. The only thing I’ve done is the clock fix as well as bootconf_extra mt7988a-bananapi-bpi-r4-pro-8x-wan-phy.
activating the phy can increase power-usage, but let us discuss this in separate thread
I’m using consistenly main - with the clock fix for BE14 - and its perfectly stable to me.
As @frank-w said, if it is random, look at power supply or shaky SFPs (if you have them).
Not using SFP’s and power supply is the stock one it came with, and it doesnt come loose. Also the issue isnt power cutting out, I just lose internet. The router is still on and the radios are working. Gonna make a new thread.