SFP OEM SFP-2.5G-T kernel PHY?

Banana Pi Router design
19

Inserted in SFP1 slot at boot:

[root@bpir3 ~]# dmesg | grep sfp
[    0.663101] sfp sfp-1: Host maximum power 1.0W
[    0.668432] sfp sfp-2: Host maximum power 1.0W
[    1.013634] sfp sfp-1: module OEM              SFP-2.5G-T       rev 1.0  sn SK2301110007     dc 230110  
[root@bpir3 ~]# ip link set eth1 up
[  110.910137] mtk_soc_eth 15100000.ethernet eth1: configuring for inband/2500base-x link mode

[  136.867078] RTL8221B-VB-CG 2.5Gbps PHY: probe of i2c:sfp-1:11 failed with error -95
[  136.874799] mtk_soc_eth 15100000.ethernet eth1: 11111 validation with support 00,00000000,00000000,00000000 failed: -EINVAL
[  136.886040] sfp sfp-1: sfp_add_phy failed: -EINVAL

[  407.112709] sfp sfp-1: module removed

Moved to SFP2 slot:

[  423.803562] sfp sfp-2: module OEM              SFP-2.5G-T       rev 1.0  sn SK2301110007     dc 230110  
[  423.859855] hwmon hwmon0: temp1_input not attached to any thermal zone

[  449.827118] RTL8221B-VB-CG 2.5Gbps PHY: probe of i2c:sfp-2:11 failed with error -95
[  449.834836] mt7530-mdio mdio-bus:1f lan4: 11111 validation with support 00,00000000,00000000,00000000 failed: -EINVAL
[  449.845560] sfp sfp-2: sfp_add_phy failed: -EINVAL

lan4 is managed by systemd-networkd and it sets it link up.

I’ve added 11111 to the validation with support message, to find out which one. It is the one from phylink_sfp_config_phy()

20

I figure the probe of the device is failing because .probe is not defined for this device.

	}, {
		PHY_ID_MATCH_EXACT(0x001cc849),
		.name           = "RTL8221B-VB-CG 2.5Gbps PHY",
		.get_features   = rtl822x_get_features,
		.config_aneg    = rtl822x_config_aneg,
		.read_status    = rtl822x_read_status,
		.suspend        = genphy_suspend,
		.resume         = rtlgen_resume,
		.read_page      = rtl821x_read_page,
		.write_page     = rtl821x_write_page,
	}, {

I could add:

		.probe		= rtl822x_probe,

and

static int rtl822x_probe(struct phy_device *phydev)
{
	return 0;
}

As I have no clue about the registers.

Then see if this changes anything…

21

tried to set the page but seems not working (confirms my error-value 0xff)

root@bpi-r3:~# i2cset -y 1 0x56 0x3e 0x0000 w
root@bpi-r3:~# i2cget -y 1 0x56 0x3e
0xff

if interested these are the other 2 i2c-device dumps:

root@bpi-r3:~# i2cdump -y 1 0x50                                                
No size specified (using byte-data access)                                      
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef         
00: 03 04 07 00 01 00 00 00 00 02 00 05 19 00 00 00    ???.?....?.??...         
10: 1e 14 00 00 4f 45 4d 20 20 20 20 20 20 20 20 20    ??..OEM                  
20: 20 20 20 20 00 00 00 00 53 46 50 2d 32 2e 35 47        ....SFP-2.5G         
30: 2d 54 20 20 20 20 20 20 31 2e 30 20 03 52 00 19    -T      1.0 ?R.?         
40: 00 1a 00 00 53 4b 32 33 30 31 31 31 30 30 31 30    .?..SK2301110010         
50: 20 20 20 20 32 33 30 31 31 30 20 20 68 f0 01 e1        230110  h???         
60: 00 00 11 fd 9b 82 05 cf 1d 67 70 da 2b 1a 48 b0    ..???????gp?+?H?         
70: d8 89 05 00 00 00 00 00 00 00 00 00 ed 86 84 a9    ???.........????         
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................         
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................         
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................         
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................         
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................         
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................         
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................         
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
root@bpi-r3:~# i2cdump -y 1 0x51                                                
No size specified (using byte-data access)                                      
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef         
00: 5f 00 ce 00 5a 00 d3 00 8c a0 75 30 88 b8 79 18    _.?.Z.?.??u0??y?         
10: 1d 4c 01 f4 19 64 03 e8 4d f0 06 30 3d e8 06 f2    ?L???d??M??0=???         
20: 2b d4 00 c7 27 10 00 df 00 00 00 00 00 00 00 00    +?.?'?.?........         
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................         
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00    ....??......?...         
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 f1    ?...?...?......?         
60: 1b b0 83 98 00 00 00 00 00 00 ff ff ff ff 82 ff    ????..........?.         
70: 05 40 ff ff 05 40 ff ff 00 ff ff ff ff ff ff 00    ?@..?@..........         
80: 43 4e 53 38 54 55 54 41 41 43 33 30 2d 31 34 31    CNS8TUTAAC30-141         
90: 30 2d 30 34 56 30 34 20 49 fb 46 00 00 00 00 29    0-04V04 I?F....)         
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................         
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 aa aa    ..............??         
c0: 47 4c 43 2d 54 20 20 20 20 20 20 20 20 20 20 20    GLC-T                    
d0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 97                   ?         
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................         
f0: 00 00 00 00 00 00 00 00 00 40 00 40 00 00 00 00    .........@.@....
22

I do not know exactly why mine is at page 0 and yours is not.

Anyway, the PHY in the module is a small microprocessor addressed at 0x50 and 0x51. It uses the rollbal protocol. It talks to whatever MMC hardware is on the bus and does all the magic.

You need to set protocol to rollball anyway, regardless of which chip is at 0x56.Setting protocol to rollball makes the mdio-i2c talk to address 0x51.Findng the phy_id number will go through this address. It will identify what is at 0x56.

/* SFP modules appear to always have their PHY configured for bus address
 * 0x56 (which with mdio-i2c, translates to a PHY address of 22).
 * RollBall SFPs access phy via SFP Enhanced Digital Diagnostic Interface
 * via address 0x51 (mdio-i2c will use RollBall protocol on this address).
 */
#define SFP_PHY_ADDR		22
#define SFP_PHY_ADDR_ROLLBALL	17

	switch (sfp->mdio_protocol) {
	case MDIO_I2C_NONE:
		break;

	case MDIO_I2C_MARVELL_C22:
		err = sfp_sm_probe_phy(sfp, SFP_PHY_ADDR, false);
		break;

	case MDIO_I2C_C45:
		err = sfp_sm_probe_phy(sfp, SFP_PHY_ADDR, true);
		break;

	case MDIO_I2C_ROLLBALL:
		err = sfp_sm_probe_phy(sfp, SFP_PHY_ADDR_ROLLBALL, true);
		break;

So apply the patch from post #18 on netdev/net-next.

You will then see:

[  449.827118] RTL8221B-VB-CG 2.5Gbps PHY: probe of i2c:sfp-2:11 failed with error -95

The chip is identified. Or see some information about another chip if yours is different. It takes 25 seconds after eth1/lan4 link is brought up.

PS: I still have not found any datasheet, but I have found realtek code of rtl8226, it was later renamed to rtl8221.:

xdr608x-rtl8221-fix/drivers/net/ethernet/mediatek/rtl822x/nic_rtl8226b.c at xdr608x-rtl8221-fix · ericwoud/xdr608x-rtl8221-fix · GitHub

Edit:

OpenWrt has interesting code here:

git.openwrt.org Git - openwrt/openwrt.git/blob - target/linux/realtek/files-5.4/drivers/net/phy/rtl83xx-phy.c

And here:

openwrt/target/linux/realtek/files-5.15/drivers/net/phy/rtl83xx-phy.c at main · openwrt/openwrt · GitHub

Maybe can use rtl8226_get_eee() and rtl8226_set_eee() from this code…

Edit 2:

Here are the most recent rtl822x patches, starting from 721

openwrt/target/linux/generic/pending-6.1 at main · openwrt/openwrt · GitHub

23

On non SFP system it looks like this:

image

And the SOC is the STA.

But on our SFP module only MAC1, the STA is at 0x51 (17). We are also on the same mdio-bus with an i2c adapter and talk to the STA with rollball protocol. (but we can snoop around the entire bus)

In our case, we have only 1 MMD at 0x56 (22).

With this construction, the SFP module can function on many systems as the hardware is set up independantly.

Before the patch at post #18, sfp->mdio_protocol = MDIO_I2C_NONE and there is no further communication done over mdio-i2c bridge. The MMD is setup by the STA at 0x51 and the MAC is setup with fixed values.

Also reading the phy_id of the MMD goes through the STA at 0x51, this got me confused at first. In fact, all communication to the MMD should go this way. The protocol of the mdio-i2c bridge is set to rollball and the only device that speaks it is the STA at 0x51.

24

Adding:

		.probe		= rtl822x_probe,

Did not help, but comparing:

	}, {
		PHY_ID_MATCH_EXACT(0x001cc840),
		.name		= "RTL8226B_RTL8221B 2.5Gbps PHY",
		.get_features	= rtl822x_get_features,
		.config_aneg	= rtl822x_config_aneg,
		.read_status	= rtl822x_read_status,
		.suspend	= genphy_suspend,
		.resume		= rtlgen_resume,
		.read_page	= rtl821x_read_page,
		.write_page	= rtl821x_write_page,
		.read_mmd	= rtl822x_read_mmd,
		.write_mmd	= rtl822x_write_mmd,
	}, {

With:

	}, {
		PHY_ID_MATCH_EXACT(0x001cc849),
		.name           = "RTL8221B-VB-CG 2.5Gbps PHY",
		.get_features   = rtl822x_get_features,
		.config_aneg    = rtl822x_config_aneg,
		.read_status    = rtl822x_read_status,
		.suspend        = genphy_suspend,
		.resume         = rtlgen_resume,
		.read_page      = rtl821x_read_page,
		.write_page     = rtl821x_write_page,
	}, {

And considering:

int __phy_read_mmd(struct phy_device *phydev, int devad, u32 regnum)
{
	int val;

	if (regnum > (u16)~0 || devad > 32)
		return -EINVAL;

	if (phydev->drv && phydev->drv->read_mmd) {
		val = phydev->drv->read_mmd(phydev, devad, regnum);
	} else if (phydev->is_c45) {
		val = __mdiobus_c45_read(phydev->mdio.bus, phydev->mdio.addr,
					 devad, regnum);
	} else {
		struct mii_bus *bus = phydev->mdio.bus;
		int phy_addr = phydev->mdio.addr;

		mmd_phy_indirect(bus, phy_addr, devad, regnum);

		/* Read the content of the MMD's selected register */
		val = __mdiobus_read(bus, phy_addr, MII_MMD_DATA);
	}
	return val;
}
EXPORT_SYMBOL(__phy_read_mmd);

The else if and else paths both have -EOPNOTSUPP (-95) as error value.

So it would seem that we are missing:

		.read_mmd	= rtl822x_read_mmd,
		.write_mmd	= rtl822x_write_mmd,

So I’ll try that next time…

Edit:

Nope, that did not fix it either… But I did find it: The realtek driver uses __phy_read().

__phy_read() points to __mdiobus_read(), which checks bus->read. If bus->read == NULL it returns -EOPNOTSUPP.

But on a bus with rollball protocol, only bus->read_c45 is a valid pointer and bus->read = NULL.

Guess this is the first time the realtek phy driver is used together with rollball protocol.

I have asked Russell how this should be fixed… See what he says.

Edit 2:

If I look at the code… I think the realtek.c phy driver needs to use phy_read_mmd() at least for the rtl822x family. This is the same as the aquantia and marvell driver.

25

So I’ve set up the realtek driver as a generic C45 device like so:

}, {
	PHY_ID_MATCH_EXACT(0x001cc849),
	.name           = "RTL8221B-VB-CG 2.5Gbps PHY",
	.get_features   = genphy_c45_pma_read_abilities,
	.config_aneg    = genphy_c45_config_aneg,
	.read_status    = genphy_c45_read_status,
	.suspend        = genphy_c45_pma_suspend,
	.resume         = genphy_c45_pma_resume,
}, {

I get everything set up nicely, no errors in the kernel log. Almost everything seems to work also at first glance. The usb rtl8156 on the other side of the copper even says that it has link up. I can read speed advertisements, also of link partner. I can connect at 2500 and at 100, and even see this speed also on the SFP. But somehow it does not report link up… That still needs some work.

1 Like
26

Do you see the phy-id on i2c address 0x51 too? As my 0x56 is inaccessable and rollball uses 0x51 it maybe works with my sfp too

27

Can really forget what you see in i2c, except for i2cdetect -y 0

Yours is for sure also a rtl8221 and yes it will work the same on yours too.

For now, need to find out why it does not report link up, for the rest it looks ok for now.

I have temporarily:

static int rtl822x_config_aneg(struct phy_device *phydev)
{
	return 0;
 // Could later add a condition on the presence of SFP and/or autoneg.
}
.......
}, {
	PHY_ID_MATCH_EXACT(0x001cc849),
	.name           = "RTL8221B-VB-CG 2.5Gbps PHY",
	.get_features   = genphy_c45_pma_read_abilities,
	.config_aneg    = rtl822x_config_aneg,
	.read_status    = genphy_c45_read_status,
	.suspend        = genphy_c45_pma_suspend,
	.resume         = genphy_c45_pma_resume,
}, {

So in fact, we are using the rtl driver only for reading

  • get_features()
  • read_status()

(If not looking at the suspend and resume.)

First find out about the link detection…

28

Deleted some stuff here which is not the way to go…

29 
[  113.518966] RTL8221B-VB-CG 2.5Gbps PHY i2c:sfp-1:11: EEEEE: 11111 phydev->link = 1
[  114.878970] RTL8221B-VB-CG 2.5Gbps PHY i2c:sfp-1:11: EEEEE: 11111 phydev->link = 1
[  116.238971] RTL8221B-VB-CG 2.5Gbps PHY i2c:sfp-1:11: EEEEE: 11111 phydev->link = 1
[  116.573704] r8152 2-1:1.0 enu1: carrier off
[  117.568969] RTL8221B-VB-CG 2.5Gbps PHY i2c:sfp-1:11: EEEEE: 11111 phydev->link = 0
[  118.928965] RTL8221B-VB-CG 2.5Gbps PHY i2c:sfp-1:11: EEEEE: 11111 phydev->link = 0
[  120.288968] RTL8221B-VB-CG 2.5Gbps PHY i2c:sfp-1:11: EEEEE: 11111 phydev->link = 0
[  121.648968] RTL8221B-VB-CG 2.5Gbps PHY i2c:sfp-1:11: EEEEE: 11111 phydev->link = 0
[  122.710835] r8152 2-1:1.0 enu1: carrier on
[  122.958964] RTL8221B-VB-CG 2.5Gbps PHY i2c:sfp-1:11: EEEEE: 11111 phydev->link = 1
[  124.318970] RTL8221B-VB-CG 2.5Gbps PHY i2c:sfp-1:11: EEEEE: 11111 phydev->link = 1

The phydev->link of the RTL8221B-VB-CG 2.5Gbps PHY does actually go up and down. It is however not propagated to eth1 somehow.

So the realtek driver itself is functioning correctly now, only using the genphy_c45_read_status() and genphy_c45_pma_read_abilities() functions. Now we need to find out why eth1 still reports it is down…

30 
r8152 2-1:1.0 enu1: carrier off
r8152 2-1:1.0 enu1: carrier on

Looks like it is (only device is named enu1…predictable net interface name) or is this your debug above?

31

enu1 is the USB rtl8152 on the other side of the copper.

enu1 does see link up, but eth1 not.

32

Change the code from:

static int rtl822x_read_status(struct phy_device *phydev)
{
	int ret;

	if (phydev->autoneg == AUTONEG_ENABLE) {
		int lpadv = phy_read_paged(phydev, 0xa5d, 0x13);

		if (lpadv < 0)
			return lpadv;

		linkmode_mod_bit(ETHTOOL_LINK_MODE_10000baseT_Full_BIT,
			phydev->lp_advertising, lpadv & RTL_LPADV_10000FULL);
		linkmode_mod_bit(ETHTOOL_LINK_MODE_5000baseT_Full_BIT,
			phydev->lp_advertising, lpadv & RTL_LPADV_5000FULL);
		linkmode_mod_bit(ETHTOOL_LINK_MODE_2500baseT_Full_BIT,
			phydev->lp_advertising, lpadv & RTL_LPADV_2500FULL);
	}

	ret = genphy_read_status(phydev);
	if (ret < 0)
		return ret;

	return rtlgen_get_speed(phydev);
}

Into this:

static int rtl822x_read_status(struct phy_device *phydev)
{
	return genphy_c45_read_status(phydev);
}

genphy_read_status() is the C22 version of genphy_c45_read_status()

Except that C22 does not support anything above 1000Mbps. This is why there is extra code, to fix the missing part, namely 2500, 5000 and 10000. You also see extra code about these speeds in rtl822x_config_aneg() and rtl822x_get_features()

Because we have switched to C45 all that code can be dropped and we are left with genphy_c45_xxxxx() calls only. As far as the realtek.c driver, this all seems to function properly. As far as can test, without eth1 link coming up.

When it is functional (at least at 2500Gbps), check also if there needs to be something done about this:

/* get actual speed to cover the downshift case */
static int rtlgen_get_speed(struct phy_device *phydev)

Edit:

eth1 would be the source from phylink.c. So need to take a look at

phylink_resolve()

To see what is happening with the phylink link status up/down.

Edit 2:

The “Down Shift” seems to be a mechanism that links at the second highest speed ability, if there are too many errors…Which is different from the “Rate Adaptor”

33

phylink_mac_pcs_get_state() returns state->link = 0

So now it is the mac that has link state down…

Edit:

Guess this is a auto negation issue between MAC (,PCS) and PHY also…

34

May find it interesting, here is my first patch set, but I need to test a lot more…

It basically works… I had it on SFP2 if that matters… Apply to netdev/net-next

Now need to test this a lot more… But I do not have the non-SFP rtl822x hardware…

The realtek patch as .patch file looks horrible… The functions are entirily rewritten, but it is totally mangled in the diff. with the old code.

35

No autoneg hassle anymore needed?

If all works so far just send it to netdev to let more users test the patch :slight_smile:

Btw. Many Thanks for taking care of this sfp and digging into code so deep :+1:

I guess you need to change the magic values to constants in rtl822x_read_status before upstreaming

36

It doesn’t always seem to initialize correctly, but after re-inserting the module, it seems that is does work nicely. I thought it was networkmanager interfering with my usb network adapter, but it apparently is not.

Anyway, it is a start in the right direction and need to test and debug some more… But it proves it can basically work…

So definitely not upstreaming for a while until it is stable…

37

Maybe send russel an update for this so he can look over it?

38

I have… Also cleaned up the code a bit more, and the readable version of the changed part of realtek.c is shown as the first file here:

1 Like