The mux implementation is no real device with registers and such…it is pnly a dumb switch which reacts on moddef0 gpio and sets another gpio which switches the serdes lanes in hardware. So you cannot read much except gpio states.
frank-w — confirmed with hard data, this is exactly it.
Decoded the devicetree reset-gpio mapping:
ethernet-phy@24(MDIO 0x18, the healthy mxl_lan5) → GPIO local pin 0x53=83 → global gpio-595ethernet-phy@28(MDIO 0x1c, our broken eth1) → GPIO local pin 0x52=82 → global gpio-594
Cross-referencing with the debugfs GPIO dump:
gpio-594 (PHY reset) out lo ACTIVE LOW ← eth1's AS21xxx — RESET ASSERTED
gpio-595 (PHY reset) out hi ACTIVE LOW ← mxl_lan5's AS21xxx — reset released, working fine
So eth1’s PHY chip is sitting in hardware reset. That explains everything — zeroed SerDes regs, intermittent IPC (probably just enough leaking through for firmware version to answer from some latched/cached state), no amount of OS-level reboot or aeon_normal_retrain fixing it, since the chip itself isn’t even out of reset.
What I still don’t understand: I did a full power-off (10 min, module removed) and gpio-594 still comes back asserted after boot. Is this GPIO driven by:
- A boot-time default in the pinctrl/DT setup that’s wrong for this specific board revision, or
- Some sequencing in the mux/reset driver that’s supposed to release it after detecting the module (via moddef0) but isn’t doing so anymore because of some other state I broke earlier (the failed
unbind)?
If it’s (2), is there a way to manually drive gpio-594 high from userspace to test (e.g. via /sys/class/gpio or gpioset) to confirm the chip comes alive once released, without risking anything? Or is manually toggling this GPIO something that could conflict with the mux_poll logic and cause more harm than good?
Afaik you cannot manually drive gpio when mapped to the mux code,but you can disable the mux node in dts or code to not bind it and then drive the gpio manually
Thanks — that explains why manual gpioset failed silently for me.
Two follow-up questions before I go down this path:
1. What’s the exact DT node name I’d need to disable — ethernet-mux@1 (under soc/ethernet@15100000/mux-bus/)? I want to make sure disabling it only affects the stuck eth1/mdio-bus:1c chip and not the healthy mxl_lan5 one, since both seem to route through mux-related code in mtk_eth_soc.c.
2. Is this something I could test via a device tree overlay at runtime (if configfs/overlay support is enabled in this build), or does it require a static .dts edit + full kernel rebuild? I’d rather confirm the fix works before committing to a full rebuild cycle if there’s a faster way to test it.
Once the node is disabled and unbound, is manually driving gpio-594 via gpioset (or /sys/class/gpio if exported) expected to be enough on its own, or does something else in the reset sequence (timing, order relative to the moddef0 read) need to happen too for the chip to actually come out of reset cleanly?
Just disable the mux-bus (i’m not sure if status=disabled set by overlay is recognized) and all subnodes or patch out the registration code in driver looks at at this subnode (near probe).
In later kernelversions (6.19+ afaik phylink needs to be reinitialized),afair 6.18 worked the way like vendor driver does (just switching and updating phy modes).
I guess the phy modes can be a problem when using usxgmii for sfp and 10base-r for phy. But i did manual switch too and updated mac mode via register writes…need to search for exact commands
Please use the following command if you need to test the AN8831 10G PHY on the eth1/eth2-MXL side:
Configure mux to eth1 10G PHY
regs m 0x1001F300 0x0 12 3 #GPIO3
regs m 0x1001F000 0x1 3 1 #Dir output
regs m 0x1001F100 0x0 3 1 #GPIO output low
Configure mux to eth2-mxl 10G PHY
regs m 0x1001F360 0x0 24 3 #GPIO54
regs m 0x1001F010 0x1 22 1 #Dir output
regs m 0x1001F110 0x1 22 1 #GPIO output high
Regs is a tool from mtk-sdk which i also added in my latest branches to cross compile separately. I guess you can use also devmem2 or similar. Needs kerneloptions to be set (kmem which are set iny defconfig iny kernel repo)
noticed that only the gpio was changed here, not the phy mode in mac…so basicly you should be able to do also with gpio mapped to mux code as it is directly register write and not using gpio driver binding which checks if it is busy
Thanks for the GPIO/comment mapping — confirms exactly what we did.
Quick update on what happened when I tried it: the regs m sequence for eth1 worked exactly as described — dmesg showed the PHY waking up cleanly (Firmware Version: 1.9.2, PHY driver attach, requesting link mode inband/2500base-x), and /sys/kernel/debug/sfp2/state confirmed Main state: link_up with Signalling rate: 3125 kBd.
But ethtool eth1 stayed at Link detected: no, and critically — dmesg never showed the switched to inband/2500base-x link mode confirmation line that we saw during our earlier successful test (before the mux got stuck). That matches exactly what you flagged: we only released the GPIO reset, we didn’t touch the MAC-side phy mode in mtk_soc_eth/phylink, so the driver’s internal state never got told the interface actually changed.
Also worth noting for anyone else trying this: a physical reinsert of the module afterwards caused mux_poll to silently override our manual GPIO write and flip the port back to copper mode (Port: Twisted Pair, 100/1000/10000baseT) — so the raw register write doesn’t stick across a hotplug event, the driver’s own polling logic re-asserts its (wrong) cached state.
Did you ever dig up those exact register writes for updating the MAC-side interface mode? That seems to be the missing half — GPIO mux + PHY reset alone gets the physical layer talking, but the MAC never gets told to actually switch to 2500base-x/USXGMII to match, so nothing above the PHY sees a link.
Which base do you use? Openwrt from wozi? The original mtk-sdk patches for phylink were buggy (and causing rcu stalls from time to time). So i had reworked them by deleting+creating phylink on mac side for each switch which fixes the stalls and also should fix the hang on old phy-mode settings. Possibly you need the usxgmii (in-band-maned) changes i’ve done with daniel/Christian.
Base: Wozi’s OpenWrt build (bpi-r4-deploy, builder-pro-8x-wired.sh), kernel 6.12.94, MTK feed pinned at a specific commit (git01/mtk-openwrt-feeds, checked out at 822c2f0603614e47ec8496571043431494fd2841 per the builder script).
That explains a lot — “hang on old phy-mode settings” matches exactly what we’re seeing: the driver requests 2500base-x, dmesg shows the request, but it never confirms switched to inband/2500base-x link mode, and ethtool stays at Link detected: no with the old advertised modes. That’s a “hang” in exactly the sense you describe.
Is your reworked phylink fix (delete+recreate phylink per switch) available as a standalone patch or branch I could apply on top of Wozi’s 6.12 tree, or is it tightly coupled to your own 6.18-main branch structure? And regarding the usxgmii in-band-managed changes with Daniel/Christian — is that also something isolated enough to backport to 6.12, or does it depend on other phylink core changes that only landed in later kernels?
Happy to test either as a patch against mtk-openwrt-feeds the same way we did for the sfp.c quirk, if you can point me to the right commit (s)/fles to pull.
Maybe you could bootup my kernel (7.0) from uboot to test the behavior first? Just to be sure you are not backporting what hangs too in your case.
Mux fixes are here (alao one additional rework from @rmandrad ):
Pcs:
Hi Wozi,
Wanted to report that I have nearly everything working, the only thing is the m.2 drives.
BPI-R4 Pro 8X running OpenWrt 25.12-SNAPSHOT r0+32995-7b8ce1e095.
Both Wi-Fi controllers enumerate correctly and one NVMe controller (11290000.pcie) enumerates correctly.
The second M-Key slot never appears because platform device 11280000.pcie is never instantiated.
/proc/device-tree/model reports Bananapi BPI-R4-PRO-8X
/sys/bus/platform/devices contains:
11290000.pcie 11300000.pcie 11310000.pcie
but not
11280000.pcie
As a result only one NVMe SSD is detected. Right now both slots are populated with 1TB nvme drives, I also have 3x 5G Modem Chips, and the 1 Wifi Module that goes into the two slots for the WiFi on the board. I have everything fully working except for the 1 nvme drive.
Pcie2 is a bit special as it needs the xsphy which is shared with back usb. Maybe xsphy driver is missing or some probe error for usb or pci.
Interesting configuration. From your post, it’s not clear to me exactly which release you’re using. Is it one from my x8 deploy GH repository? I also don’t know exactly how you’re using the CNxx M.2 slots or what’s in them.