[BPI-R4] and SFP

[jpsollie (Janpieter Sollie)]

tested: BPI-r4 is a dumb AP. PC1 and PC2 both have a 10gbe adapter.

Tests are done via NFS file transfer by mounting PC2/share on PC1 and dd if=share1/file1 of=/dev/null

The BPI-R4 in between has 2 RJ45 2.5G SFP adapters

1. from PC1 @ sfp-lan (eth2) to PC2 @ sfp-wan (eth1): 100mbit
2. from PC1 @ lan1 to PC2 @ sfp-wan (eth1): 100mbit
3. from PC1 - PC2 (direct link): 5gbit (limited by SATA SSD)
4. from PC1 @ lan1 to PC2 @ lan2: 1gbit

so the only conclusion here could be that eth1 and eth2 are running at 100mbit speed, even though they claim to be running at 2500mbit. How could I invesitgate more?

[frank-w]

You should test only 1 connection first (pc1 or pc2 to r4),e.g. acessing a file on bpi-r4 nvme,or better using iperf(3) to avoid bandwidth limitation through protocol overhead. I’m not sure how much overhead is added for nfs and how performant it is configured in your case (packet size,latency,…).then adding complexity/protocols step by step. E.g. when using encryption like on sftp/scp maybe this is the bottleneck.

[jpsollie (Janpieter Sollie)]

looks like you’re right: when running this on BPI-R4 itself:

mount -t nfs4 PC1/ /tmp/desktopjp
date; dd if=/tmp/desktopjp/largefile.iso of=/dev/null; date
umount /tmp/desktopjp

the results are unpredictable at best (even when swtiching between sfp-lan and lan1), but nowhere at a limited size of close to 100mbit (more like 800-2.2gbit, it depends). so it’s definitely not the SFP adapter. Could it be a bridging problem when a non-dsa port is involved?

[frank-w]

It could be an issue when dsa port involved. So just leave the bridge and try your original setup without.

[jpsollie (Janpieter Sollie)]

I’m not exactly sure what you mean … In the testing setup: the bridge interface is the only intrface with an IP.

PC1 is connected via SFP-LAN and LAN1 to a internal bridge on the PC itself and to the bridge of the BPI-R4. it used to be a bonding link BPI-R4<=>PC1 and bondPC1 was connected to br-lan, but I broke it and performed testing with sfp-lan and lan1 connected to the internal bridge.

I performed testing with sfp-lan (xor) lan1 up and did the command unmounting and remounting every time to work around the file caching of linux. on both interfaces, the speed was way above 100mbit

So, I’m not sure I get what you’d like me to try…

edit: or did you mean: both PCs on a lan1 / lan2 interface? that is 1gbit. I tested, based on your suggestion, how fast the BPI would copy files from PC1 to internal /dev/null, so traffic speed is measured without actually switching between ethernet ports. Traffic speed from PC1 to PC2 when both are connected to DSA is 1gbit

[frank-w]

@jpsollie

PC1 is connected via SFP-LAN and LAN1 to a internal bridge on the PC itself and to the bridge of the BPI-R4. it used to be a bonding link BPI-R4<=>PC1 and bondPC1 was connected to br-lan, but I broke it and performed testing with sfp-lan and lan1 connected to the internal bridge.

Test only on sfp-lan without bridge and bonding. Same on the other side (sfp-wan - pc2).

You can give the sfp an ip adress on the mac (eth1/2)

[Polaris_of_Stars]

Hi to all.

I have xPON stick HSGQ which is the same as ODI DFP-34X-2IY3 (APC) but V8, not V6 curcuit board plugged to Banana Banana BPI-R3. Banana works at OpenWrt 24.10.1 r28597-0425664679 / LuCI (HEAD detached at 2ac26e56) branch 25.103.51521~2ac26e5 with kernel 6.6.86.

As I know the one of different between V6 and V8 is that V8 eeprom can be changed only by coding board. But I can be wrong.

As I understand at the time there is no necessary quirk record for this stick. So, the stick is accessible at 1Gbit mode - 1000baseX/Full but with auto negotiation strictly OFF It is enough for me at the time. Of course 2,5 Gbit would be prefferable. But there is ONE big incompatability - tre stick is accessible without fiber cable plugged. But just after patch cord is plugged the stick is disapeared. And ethertool show: : RX_LOS implemented, inverted Option
: TX_FAULT implemented Option

So, I founded that byte at 0x41 of 0x50 eeprom has to be changed as well as 0x5f checksum byte. Now 0x41 = 0x1C, but have to be 0x1A (I can not check it yet). I tried to use i2csfp byte 0x50 0x41 0x1A -p some password, as well as i2csfp byte 0x50 0x5f 0xFF -p some password But it didn’t help. And after rebooting the stick eeprom seems not changed. Temporarily solution i2csfp sfp-1 eepromfix -V OEM -N SFP-2G5 -p 0xffffffff helped. But this have to be applied before the fiber plugged and after rebooting these settings is not here as follow unplug fiber, apply command and plug fiber again is necessary.

Could please some helps me ?

[jpsollie (Janpieter Sollie)]

@frank-w : I’m sorry I did not reply to you earlier, but the issue seems to be the transceiver; one gave me 2.5GbE speeds, the other one just 100mbit. Right now, I replaced both with a “I claim to be” aquantia phy … as far as I understand in other posts, those should have necessary settings for what I want to try.

However, can someone tell me if this IS the case at all? The picture says so, and the dmesg off course doesn’t say anything, but also I do not see anything of the module in sfp.c kernel code. Could someone please tell me if the picture is what it claims to be? @ericwoud : I tried your beautiful i2csfp utility, but could not read the common i2c pages

        Identifier                                : 0x03 (SFP)
        Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
        Connector                                 : 0x07 (LC)
        Transceiver codes                         : 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
        Transceiver type                          : 10G Ethernet: 10G Base-SR
        Encoding                                  : 0x06 (64B/66B)
        BR Nominal                                : 10300MBd
        Rate identifier                           : 0x00 (unspecified)
        Length (SMF)                              : 0km
        Length (OM2)                              : 80m
        Length (OM1)                              : 20m
        Length (Copper or Active cable)           : 0m
        Length (OM3)                              : 300m
        Laser wavelength                          : 850nm
        Vendor name                               : OEM
        Vendor OUI                                : 00:1b:21
        Vendor PN                                 : ZK-10G-TX
        Vendor rev                                : 1
        Option values                             : 0x00 0x1a
        Option                                    : TX_DISABLE implemented
        BR margin max                             : 0%
        BR margin min                             : 0%
      Vendor SN                                 : 2505010443
        Date code                                 : 250412
        Optical diagnostics support               : Yes
        Laser bias current                        : 6.000 mA
        Laser output power                        : 0.5000 mW / -3.01 dBm
        Receiver signal average optical power     : 0.4000 mW / -3.98 dBm
        Module temperature                        : 70.36 degrees C / 158.65 degrees F
        Module voltage                            : 3.3400 V
        Alarm/warning flags implemented           : Yes
        Laser bias current high alarm             : Off
        Laser bias current low alarm              : Off
        Laser bias current high warning           : Off
        Laser bias current low warning            : Off
        Laser output power high alarm             : Off
        Laser output power low alarm              : Off
        Laser output power high warning           : Off
        Laser output power low warning            : Off
        Module temperature high alarm             : Off
        Module temperature low alarm              : Off
        Module temperature high warning           : Off
        Module temperature low warning            : Off
        Module voltage high alarm                 : Off
        Module voltage low alarm                  : Off
        Module voltage high warning               : Off
        Module voltage low warning                : Off
        Laser rx power high alarm                 : Off
        Laser rx power low alarm                  : Off
        Laser rx power high warning               : Off
        Laser rx power low warning                : Off
        Laser bias current high alarm threshold   : 15.000 mA
        Laser bias current low alarm threshold    : 1.000 mA
        Laser bias current high warning threshold : 13.000 mA
        Laser bias current low warning threshold  : 2.000 mA
        Laser output power high alarm threshold   : 1.9952 mW / 3.00 dBm
        Laser output power low alarm threshold    : 0.1584 mW / -8.00 dBm
        Laser output power high warning threshold : 1.5848 mW / 2.00 dBm
        Laser output power low warning threshold  : 0.1778 mW / -7.50 dBm
        Module temperature high alarm threshold   : 95.00 degrees C / 203.00 degrees F
        Module temperature low alarm threshold    : -50.00 degrees C / -58.00 degrees F
        Module temperature high warning threshold : 90.00 degrees C / 194.00 degrees F
        Module temperature low warning threshold  : -45.00 degrees C / -49.00 degrees F
        Module voltage high alarm threshold       : 3.6000 V
        Module voltage low alarm threshold        : 3.0000 V
        Module voltage high warning threshold     : 3.5000 V
        Module voltage low warning threshold      : 3.1000 V
        Laser rx power high alarm threshold       : 1.1220 mW / 0.50 dBm
        Laser rx power low alarm threshold        : 0.0199 mW / -17.01 dBm
        Laser rx power high warning threshold     : 1.0000 mW / 0.00 dBm
        Laser rx power low warning threshold      : 0.0223 mW / -16.52 dBm

*EDIT: dumped wrong trainsceiver, so SN had to be corrected

[ericwoud (Eric W.)]

Make sure you do not have this problem first:

[BPI-R4] nvme+i2c

Beware of NVME M2 key M. In the electrical circuit there is an I2C bus. [image] I got in trouble for this. My disk is Miсron 2200 512GB. [51xpbTapoFL.AC_SL1200] As soon as I install it, the entire I2C bus goes down. Not a single device is detected. I had to cover these 2 pins with Scotch tape to solve this. [5190v7R+GxL.AC_SL1200_1]

As for the AQR113C:

It really depends how the mdio-i2c bridge is implemented on your sfp module, if implemented at all… I think the aqr113c management interface is only available via mdio, so if you are lucky, it has an extra IC that has rollball on it. So you can try a fixup for using rollball, with or without the 4 seconds delay.

The 2.5G luleey has rollball, so the 10G may also have it.

Otherwise, if the phy registers are not accessible, it is stuck in 10G speed with rate adaptation. The eeprom should provide the correct info. This means that the mac and mac-side of the phy are always running at full speed, even when slower copper is connected. It also works, but is not very energy efficient.

Anyway, if the 113C driver supports switching interfaces, still need to look it up.

[jpsollie (Janpieter Sollie)]

The main purpose here would indeed be power saving features, the aqr113C supports eee, so it would be nice from a thermal / power perspective. If the i2c pages on the 2.5gbe module could be read, is that enough to say 'I do not have this m.2 problem?

[frank-w]

Yes,when having the m.2 issue,the complete i2c bus including the mux goes down. So you cannot read the sfp vendor/product information.

[jpsollie (Janpieter Sollie)]

I meant: I have a device in the m.2 slot, When using it with the 2 2.5gbe transceivers, I can read the i2c sfp pages, When replacing those with the 10gbe transceivers, the output of Eric’s i2csfp tool is:

./i2csfp sfp1 eepromdump
0x50:
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f   0123456789abcdef
00: 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00   ............g...
10: 08 02 00 1e 4f 45 4d 20 20 20 20 20 20 20 20 20   ....OEM
20: 20 20 20 20 00 00 1b 21 5a 4b 2d 31 30 47 2d 54       ...!ZK-10G-T
30: 58 20 20 20 20 20 20 20 31 20 20 20 03 52 00 89   X       1   .R..
40: 00 1a 00 00 32 35 30 35 30 31 30 34 34 33 20 20   ....2505010443
50: 20 20 20 20 32 35 30 34 31 32 20 20 68 f0 03 9b       250412  h...
60: 9f 00 11 6d 82 26 b8 38 2a 2f 53 b4 10 06 7d e3   ...m.&.8*/S...}.
70: 18 2f eb 00 00 00 00 00 00 00 00 00 37 9e f6 d5   ./..........7...
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0x51:
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f   0123456789abcdef
00: 5f 00 ce 00 5a 00 d3 00 8c a0 75 30 88 b8 79 18   _...Z.....u0..y.
10: 1d 4c 01 f4 19 64 03 e8 4d f0 06 30 3d e8 06 f2   .L...d..M..0=...
20: 2b d4 00 c7 27 10 00 df 00 00 00 00 00 00 00 00   +...'...........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00   ....?...........
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 f1   ................
60: 4d 2f 82 78 00 00 00 00 0f a0 00 00 00 00 82 00   M/.x............
70: 05 00 00 00 05 00 00 00 00 00 00 ff ff ff ff 00   ................
0x51 PAGE 0x00:
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f   0123456789abcdef
80: 43 4f 55 49 41 38 4e 43 41 41 31 30 2d 32 34 31   COUIA8NCAA10-241
90: 35 2d 30 33 56 30 33 20 01 00 46 00 00 00 00 c6   5-03V03 ..F.....
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 aa aa   ................
c0: 53 46 50 2d 31 30 47 2d 53 52 20 20 20 20 20 20   SFP-10G-SR
d0: 20 20 20 20 33 32 00 00 00 00 00 00 00 00 00 35       32.........5
e0: 1e 28 2e 2e 31 34 29 36 00 00 00 00 00 00 00 00   .(..14)6........
f0: 00 00 00 00 00 66 00 00 ff ff ff ff 00 00 00 00   .....f..........
0x51 PAGE 0x01:
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f   0123456789abcdef
80: 43 4f 55 49 41 38 4e 43 41 41 31 30 2d 32 34 31   COUIA8NCAA10-241
90: 35 2d 30 33 56 30 33 20 01 00 46 00 00 00 00 c6   5-03V03 ..F.....
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 aa aa   ................
c0: 53 46 50 2d 31 30 47 2d 53 52 20 20 20 20 20 20   SFP-10G-SR
d0: 20 20 20 20 33 32 00 00 00 00 00 00 00 00 00 35       32.........5
e0: 1e 28 2e 2e 31 34 29 36 00 00 00 00 00 00 00 00   .(..14)6........
f0: 00 00 00 00 00 66 00 00 ff ff ff ff 00 00 00 00   .....f..........
0x51 PAGE 0x02:
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f   0123456789abcdef
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
0x51 PAGE 0x03:
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f   0123456789abcdef
80: 00 00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
90: ff ff 00 00 ff ff ff ff ff ff ff ff ff ff ff ff   ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
f0: ff ff ff ff ff 56 31 32 ff ff 04 00 63 73 77 77   .....V12....csw

edit2 the command was wrong, my mistake

[ericwoud (Eric W.)]

@jpsollie

63 73 77 77 csww

That looks like the write protect password, same as the 2.5G module.

So as far as I can see, this is rollball, so add a fixup to sfp.c to set the rollball protocol.

The rollball protocol supports C45 communication only, no C22. So hopefully the AQR113C driver only uses c45. For the realtek phy, we needed to add a new c45-only driver instance.

As with others, the extcc byte (offset 0x24 on address 0x50) is 0x00, so the eeprom does not report which interface should be used. So this needs to be set in the fixup, or edit the eeprom, using the password (automatically).

If your kernel is new enough, phylink.c can handle the fact that the mtk pcs does not support inband negotiation at 2500basex.

[jpsollie (Janpieter Sollie)]

I tried to apply a quirk (as you described), but it doesn’t seem to work like we’d expect:

--- a/drivers/net/phy/sfp.c
+++ b/drivers/net/phy/sfp.c
@@ -515,6 +515,7 @@
SFP_QUIRK_F("OEM", "SFP-GE-T", sfp_fixup_ignore_tx_fault),

SFP_QUIRK_F("OEM", "SFP-10G-T", sfp_fixup_rollball_cc),
+SFP_QUIRK_F("OEM", "ZK-10G-TX", sfp_fixup_rollball_cc),
SFP_QUIRK_M("OEM", "SFP-2.5G-T", sfp_quirk_oem_2_5g),
SFP_QUIRK_M("OEM", "SFP-2.5G", sfp_quirk_oem_2_5g),
SFP_QUIRK_M("OEM", "SFP-2.5G-BX10-D", sfp_quirk_2500basex), 

But when booting the kernel, somehow, the module seems to fail validation:

[   18.881557] mtk_soc_eth 15100000.ethernet sfp-wan: configuring for inband/10gbase-r link mode
[   18.913436] bondjp: (slave sfp-wan): making interface the new active one
[   18.920948] bondjp: (slave sfp-wan): Enslaving as an active interface with an up link
[   18.930753] mt7530-mmio 15020000.switch wan: configuring for phy/internal link mode
[   18.939839] mt7530-mmio 15020000.switch wan: configuring for phy/internal link mode
[   18.947861] bondjp: (slave wan): Enslaving as a backup interface with an up link
[   22.349999] mt7530-mmio 15020000.switch lan3: Link is Up - 1Gbps/Full - flow control off
[   23.060038] mt7530-mmio 15020000.switch wan: Link is Up - 1Gbps/Full - flow control rx/tx
[   32.268141] mtk_soc_eth 15100000.ethernet sfp-wan: validation with support 00,00000000,00000000,00000000 failed: -EINVAL
[   32.279115] sfp sfp1: sfp_add_phy failed: -EINVAL

Does this “validation” thing tell you anything?

I tried (once again) your i2csfp tool, but it was in vain:

root@APBureau4:~# i2csfp sfp1 eepromfix
[ 1141.132693] sfp sfp1: module removed
Checksum 0x00-0x3e matched 89
Checksum 0x40-0x5e matched 9b
RollBall Password used: 0x63737777
Checksum 0x00-0x3e matched 89
Checksum 0x40-0x5e matched 9b
root@APBureau4:~# i2csfp sfp1 restore
[ 1154.964505] sfp sfp1: Host maximum power 3.0W
[ 1155.287121] sfp sfp1: module OEM              ZK-10G-TX        rev 1    sn 2505010443       dc 250412  
[ 1155.336988] hwmon hwmon1: temp1_input not attached to any thermal zone
[ 1163.222307] mtk_soc_eth 15100000.ethernet sfp-wan: validation with support 00,00000000,00000000,00000000 failed: -EINVAL
[ 1163.233253] sfp sfp1: sfp_add_phy failed: -EINVAL

so, based on this mysterious validation error, can you help me out?

[ericwoud (Eric W.)]

@jpsollie

Does this “validation” thing tell you anything?

Yes, it cannot find a valid interface mode to apply, as I mentioned, extcc is zero… Try

static void sfp_fixup_oem_10gt(struct sfp *sfp)
{
        sfp_fixup_rollball_cc(sfp);
        sfp_fixup_10gbaset_30m(sfp);
}

SFP_QUIRK_F("OEM", "ZK-10G-TX", sfp_fixup_oem_10gt),

As far as I know, sfp_fixup_rollball_wait4s(), is only for the FS modules, but you could also try it, but I would gamble for sfp_fixup_rollball_cc()

echo "file drivers/net/phy/* +p" > /sys/kernel/debug/dynamic_debug/control

Will give you more info on the dmesg output.

i2csfp

USE WITH CAUTION!!!

I think better not to use it, you are already making a new fixup. There is nothing to use it for anymore, as we already know it is rollball.

Is CONFIG_AQUANTIA_PHY set?

After setting link up on the sfp, it could be interesting to dump the i2c at address 0x56. There usually is a copy if the phy-id in the early bytes I believe 0x04-0x07

[jpsollie (Janpieter Sollie)]

@ericwoud

echo "file drivers/net/phy/* +p" > /sys/kernel/debug/dynamic_debug/control

[   36.568303] sfp sfp1: sfp_add_phy failed: -ETIMEDOUT
[   36.573379] mt7530-mmio 15020000.switch wan: configuring for phy/internal link mode
[   36.581873] bondjp: (slave wan): Enslaving as a backup interface with an up link
[   36.589463] br-lan: port 2(lan3) entered blocking state
[   36.594685] br-lan: port 2(lan3) entered forwarding state
[   36.600602] bondjp: entered promiscuous mode
[   36.604876] mtk_soc_eth 15100000.ethernet sfp-wan: entered promiscuous mode
[   36.612056] 8021q: adding VLAN 0 to HW filter on device bondjp
[   36.618090] br-lan: port 3(bondjp) entered blocking state
[   36.623485] br-lan: port 3(bondjp) entered disabled state
[   36.628907] bondjp: entered allmulticast mode
[   36.633259] mtk_soc_eth 15100000.ethernet sfp-wan: entered allmulticast mode
[   36.640496] br-lan: port 3(bondjp) entered blocking state
[   36.645890] br-lan: port 3(bondjp) entered forwarding state
[   36.653963] mtk_soc_eth 15100000.ethernet sfp-lan: configuring for inband/10gbase-r link mode
[   36.686377] br-lan: port 4(sfp-lan) entered blocking state
[   36.691881] br-lan: port 4(sfp-lan) entered disabled state
[   44.604079] mtk_soc_eth 15100000.ethernet sfp-lan: validation with support 00,00000000,00000000,00000000 failed: -EINVAL
[   44.615030] sfp sfp2: sfp_add_phy failed: -EINVAL

this means I have to try the 4s() hack?

[ericwoud (Eric W.)]

You can try, otherwise it is time to do some debugging…

Did you add anything in the fixup with setting extended_cc? It is where phylink gets the info which interface it can use on the phy and validate it with what the mac is capable of.

Don’t see much extra info. Try inserting the module after adding to the dynamic debugging.

Is dynamic debugging enabled at all? How about that kernel config of aquantia?

Edir:

You could also try another sfp port, not the one on the dsa-switch, just to rule out some possible issues.

Also disable bonding until your module works as expected.

[jpsollie (Janpieter Sollie)]

@ericwoud

You could also try another sfp port, not the one on the dsa-switch, just to rule out some possible issues.

Also disable bonding until your module works as expected.

Yeah, well, the 2.5GBE issue is broken beyond repair: it seemed to be 2 different SFP modules. Let’s not worry about those, they had no EEE compatible chip anyway.

So, your long-requested dynamic debug output:

i2csfp sfp1 restore
[  145.645484] sfp sfp1: Host maximum power 3.0W
[  145.649918] sfp sfp1: tx disable 1 -> 1
[  145.653764] sfp sfp1: SM: enter empty:detached:down event insert
[  145.659777] sfp sfp1: SM: exit probe:detached:down
[  145.664851] sfp sfp1: SM: enter probe:detached:down event dev_attach
[  145.671207] sfp sfp1: SM: exit probe:down:down
[  145.675642] sfp sfp1: SM: enter probe:down:down event dev_up
[  145.681312] sfp sfp1: SM: exit probe:up:down
root@APBureau4:~# [  145.966584] sfp sfp1: SM: enter probe:up:down event timeout
[  145.983234] sfp sfp1: module OEM              ZK-10G-TX        rev 1    sn 2505010443       dc 250412
[  145.992638] sfp sfp1: tx disable 1 -> 0
[  145.996491] sfp sfp1: SM: exit present:up:wait
[  146.016614] sfp sfp1: los 0 -> 1
[  146.019837] sfp sfp1: SM: enter present:up:wait event los_high
[  146.025659] sfp sfp1: SM: exit present:up:wait
[  146.039197] hwmon hwmon2: temp1_input not attached to any thermal zone
[  150.006588] sfp sfp1: SM: enter present:up:wait event timeout
[  150.013172] mdio_bus i2c:sfp1: probed
[  157.446587] mtk_soc_eth 15100000.ethernet sfp-wan: validation with support 00,00000000,00000000,00000000 failed: -EINVAL
[  157.457532] sfp sfp1: sfp_add_phy failed: -EINVAL
[  157.462230] sfp sfp1: SM: exit present:up:fail
[  157.467131] sfp sfp1: los 1 -> 0
[  157.470360] sfp sfp1: SM: enter present:up:fail event los_low
[  157.476099] sfp sfp1: SM: exit present:up:fail

EDIT: aquantia:

modinfo aquantia
filename:       /lib/modules/6.12.33/aquantia.ko
license:        GPL v2
depends:        crc-itu-t
intree:         Y
name:           aquantia
vermagic:       6.12.33 SMP mod_unload aarch64
root@APBureau4:~# 

I’m using both ports to experiment with, so whatever issues bonding may present, they won’t show up on the not bonded uplink

for your extended_cc: How should I put function calls? first rollball, then extended_cc and then gbaset_30m? or some other way?

[jpsollie (Janpieter Sollie)]

@ericwoud Do you have any other hints for bringing this i2c interface to life? I tried with the extended_cc first and with the 30m call first, but it didn’t help. Obviously it is probing something it should not. Do you have other dynamic debug flags I could try to figure out why the timeout happens and where? it may also be a hw issue, but not sure why

EDIT I may have found something (but it will give you a headache): When compiling with i2c core, algorithm and bus debug, it seems to work:

[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd090][ 0.0...

while when turning off i2c debug output, it fails:

[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd090][ 0.0...

There’s clearly some timeout we’re hitting, right? This is the patch I currently applied:

--- a/drivers/net/phy/sfp.c     2025-06-13 13:14:05.404485796 +0200
+++ b/drivers/net/phy/sfp.c     2025-06-13 13:21:18.598841574 +0200
@@ -446,6 +446,13 @@
        sfp_quirk_disable_autoneg(id, modes, interfaces);
 }
 
+
+static void sfp_fixup_oem_10gt(struct sfp *sfp)
+{
+       sfp_fixup_rollball_cc(sfp);
+       sfp_fixup_10gbaset_30m(sfp);
+}
+
 static void sfp_quirk_ubnt_uf_instant(const struct sfp_eeprom_id *id,
                                      unsigned long *modes,
                                      unsigned long *interfaces)
@@ -515,6 +522,7 @@
        SFP_QUIRK_F("OEM", "SFP-GE-T", sfp_fixup_ignore_tx_fault),
 
        SFP_QUIRK_F("OEM", "SFP-10G-T", sfp_fixup_rollball_cc),
+       SFP_QUIRK_F("OEM", "ZK-10G-TX", sfp_fixup_oem_10gt),
        SFP_QUIRK_M("OEM", "SFP-2.5G-T", sfp_quirk_oem_2_5g),
        SFP_QUIRK_M("OEM", "SFP-2.5G", sfp_quirk_oem_2_5g),
        SFP_QUIRK_M("OEM", "SFP-2.5G-BX10-D", sfp_quirk_2500basex),

would adding a “sleep” command in sfp_fixup_oem_10gt() work?

[jpsollie (Janpieter Sollie)]

one final answer, then I hope some light will shine: I tried with this patch:

--- a/drivers/net/phy/sfp.c     2025-06-13 13:14:05.404485796 +0200
+++ b/drivers/net/phy/sfp.c     2025-06-13 13:21:18.598841574 +0200
@@ -446,6 +446,13 @@
        sfp_quirk_disable_autoneg(id, modes, interfaces);
 }
 
+
+static void sfp_fixup_oem_10gt(struct sfp *sfp) {
+       sfp_fixup_10gbaset_30m(sfp);
+       sfp_fixup_rollball_cc(sfp);
+       sfp->module_t_wait = msecs_to_jiffies(10000);
+}
+
 static void sfp_quirk_ubnt_uf_instant(const struct sfp_eeprom_id *id,
                                      unsigned long *modes,
                                      unsigned long *interfaces)
@@ -515,6 +522,7 @@
        SFP_QUIRK_F("OEM", "SFP-GE-T", sfp_fixup_ignore_tx_fault),
 
        SFP_QUIRK_F("OEM", "SFP-10G-T", sfp_fixup_rollball_cc),
+       SFP_QUIRK_F("OEM", "ZK-10G-TX", sfp_fixup_oem_10gt),
        SFP_QUIRK_M("OEM", "SFP-2.5G-T", sfp_quirk_oem_2_5g),
        SFP_QUIRK_M("OEM", "SFP-2.5G", sfp_quirk_oem_2_5g),
        SFP_QUIRK_M("OEM", "SFP-2.5G-BX10-D", sfp_quirk_2500basex),

and I turned on “i2c algorithm debugging” and “i2c bus debugging”. It was not enough:

i2csfp sfp1 restore
[  155.393151] sfp sfp1: Host maximum power 3.0W
[  155.397583] sfp sfp1: tx disable 1 -> 1
[  155.401428] sfp sfp1: SM: enter empty:detached:down event insert
[  155.407437] sfp sfp1: SM: exit probe:detached:down
[  155.412509] sfp sfp1: SM: enter probe:detached:down event dev_attach
[  155.418864] sfp sfp1: SM: exit probe:down:down
[  155.423299] sfp sfp1: SM: enter probe:down:down event dev_up
[  155.428968] sfp sfp1: SM: exit probe:up:down
root@APBureau4:~# [  155.716366] sfp sfp1: SM: enter probe:up:down event timeout
[  155.733000] sfp sfp1: module OEM              ZK-10G-TX        rev 1    sn 2505010443       dc 250412  
[  155.742401] sfp sfp1: tx disable 1 -> 0
[  155.746253] sfp sfp1: SM: exit present:up:wait
[  155.756413] sfp sfp1: los 0 -> 1
[  155.759643] sfp sfp1: SM: enter present:up:wait event los_high
[  155.765466] sfp sfp1: SM: exit present:up:wait
[  155.779015] hwmon hwmon2: temp1_input not attached to any thermal zone
[  166.246368] sfp sfp1: SM: enter present:up:wait event timeout
[  166.252950] mdio_bus i2c:sfp1: probed
[  173.669344] mtk_soc_eth 15100000.ethernet sfp-wan: validation with support 00,00000000,00000000,00000000 failed: -EINVAL
[  173.680350] sfp sfp1: sfp_add_phy failed: -EINVAL
[  173.685051] sfp sfp1: SM: exit present:up:fail
[  176.691632] sfp sfp1: los 1 -> 0
[  176.694877] sfp sfp1: SM: enter present:up:fail event los_low
[  176.700628] sfp sfp1: SM: exit present:up:fail

so I hope @ericwoud you know why slowing down this way didn’t work, but slowing down i2c as a whole by turning on debug output worked

EDIT just to be sure: the link is REALLY up, I could even view the eee settings of the aqr113c:

root@APBureau4:~# ethtool --show-eee sfp-lan
EEE settings for sfp-lan:
        EEE status: disabled
        Tx LPI: 0 (us)
        Supported EEE link modes:  1000baseT/Full
                                   10000baseT/Full
                                   1000baseKX/Full
                                   10000baseKX4/Full
                                   10000baseKR/Full
                                   2500baseT/Full
                                   5000baseT/Full
        Advertised EEE link modes:  Not reported
        Link partner advertised EEE link modes:  100baseT/Full
                                                 1000baseT/Full
                                                 10000baseT/Full