[BPI-R4] Failed attempt: add NVMe support on uboot

frank-w · March 1, 2024, 2:35pm

i’m not at home this weekend so cannot test, but pin numbers also differ, but not same offset…

have you checked with pinmux dump if the pinmux is applied?

we have defined these pins in dts

pcie3_pins: pcie3-pins {
	mux {
		function = "pcie";
		groups = "pcie_1l_1_pereset", "pcie_clk_req_n3",
			 "pcie_wake_n3_0";
	};
};

referring these pins/functions in pinctrl-driver (drivers/pinctrl/mediatek/pinctrl-mt7988.c)

static const int mt7988_pcie_1l_1_pereset_pins[] = { 20 };
static const int mt7988_pcie_1l_1_pereset_funcs[] = { 1 };

static const int mt7988_pcie_clk_req_n3_pins[] = { 10 };
static const int mt7988_pcie_clk_req_n3_funcs[] = { 1 };

static const int mt7988_pcie_wake_n3_0_pins[] = { 9 };
static const int mt7988_pcie_wake_n3_0_funcs[] = { 1 };

static const struct mtk_function_desc mt7988_functions[] = {
...
	{"pcie", mt7988_pcie_groups, ARRAY_SIZE(mt7988_pcie_groups)},
...
static const char *const mt7988_pcie_groups[] = { "pcie_wake_n0_0",
	"pcie_clk_req_n0_0", "pcie_wake_n3_0", "pcie_clk_req_n3",
	"pcie_p0_phy_i2c", "pcie_p1_phy_i2c", "pcie_p3_phy_i2",
	"pcie_p2_phy_i2c", "ckm_phy_i2c", "pcie_wake_n0_1", "pcie_wake_n3_1",
	"pcie_2l_0_pereset", "pcie_1l_1_pereset", "pcie_clk_req_n2_1",
	"pcie_2l_1_perese", "pcie_1l_0_pereset", "pcie_wake_n1_0",
	"cie_clk_req_n1", "pcie_wake_n2_0", "pcie_wake_n2_1", };

cie_clk_req_n1 misses “p”, but unrelated to our controller

linux prints these

    type: MUX_GROUP controller pinctrl_moore group: pcie_1l_1_pereset (40) function: pcie (7)
    type: MUX_GROUP controller pinctrl_moore group: pcie_clk_req_n3 (16) function: pcie (7)
    type: MUX_GROUP controller pinctrl_moore group: pcie_wake_n3_0 (15) function: pcie (7)

in linux pinmux definitions are same, but strange that displayed function is different

maybe you can set debugs in mtk_pinconf_group_set and mtk_pinconf_set in pinctrl-mtk-common.c

also wonder why such few clocks are shown in clk dump in uboot…

@moore have you an idea whats wrong here or is it right and the output is confusing us?

YuyuMarco · March 2, 2024, 5:34am

Adding debugger to mtk_pinconf_group_set and mtk_pinconf_set has no output, so I add some debugger to its mux version (mtk_pinmux_group_set and mtk_pinmux_group_set)

Here is the output when booting:

Here is the output for pci enum:

I believe the groups are pcie_1l_1_pereset, pcie_clk_req_n3, and pcie_wake_n3_0 and they all have only one pin.
Notice I edit pinctrl-mt7988.c which make the pin# and function id the same with Linux.

To be clear, in the output for loop with pin#=1, the pin#=1 actually means pin count in the group is 1.

Here is the pinmux dump:

pinctrl@1001f000:
UART2_RXD : Aux Func.1
UART2_TXD : Aux Func.1
UART2_CTS : Aux Func.1
UART2_RTS : Aux Func.1
GPIO_A    : Aux Func.0
SMI_0_MDC : Aux Func.1
SMI_0_MDI : Aux Func.1
PCIE30_2L : Aux Func.1
PCIE30_2L : Aux Func.1
PCIE30_1L : Aux Func.1
PCIE30_1L : Aux Func.1
GPIO_P    : Aux Func.1
WATCHDOG  : Aux Func.1
GPIO_RESE : Aux Func.0
GPIO_WPS  : Aux Func.0
PMIC_I2C_ : Aux Func.7
PMIC_I2C_ : Aux Func.7
I2C_1_SCL : Aux Func.1
I2C_1_SDA : Aux Func.1
PCIE30_2L : Aux Func.1
PCIE30_1L : Aux Func.1
PWMD1     : Aux Func.5
SPI0_WP   : Aux Func.1
SPI0_HOLD : Aux Func.1
SPI0_CSB  : Aux Func.1
SPI0_MISO : Aux Func.1
SPI0_MOSI : Aux Func.1
SPI0_CLK  : Aux Func.1
SPI1_CSB  : Aux Func.5
SPI1_MISO : Aux Func.5
SPI1_MOSI : Aux Func.5
SPI1_CLK  : Aux Func.5
SPI2_CLK  : Aux Func.5
SPI2_MOSI : Aux Func.5
SPI2_MISO : Aux Func.5
SPI2_CSB  : Aux Func.5
SPI2_HOLD : Aux Func.5
SPI2_WP   : Aux Func.5
EMMC_RSTB : Aux Func.1
EMMC_DSL  : Aux Func.1
EMMC_CK   : Aux Func.7
EMMC_CMD  : Aux Func.1
EMMC_DATA : Aux Func.1
EMMC_DATA : Aux Func.1
EMMC_DATA : Aux Func.1
EMMC_DATA : Aux Func.1
EMMC_DATA : Aux Func.1
EMMC_DATA : Aux Func.1
EMMC_DATA : Aux Func.1
EMMC_DATA : Aux Func.1
PCM_FS_I2 : Aux Func.1
PCM_CLK_I : Aux Func.1
PCM_DRX_I : Aux Func.1
PCM_DTX_I : Aux Func.1
PCM_MCK_I : Aux Func.1
UART0_RXD : Aux Func.1
UART0_TXD : Aux Func.1
PWMD0     : Aux Func.1
JTAG_JTDI : Aux Func.0
JTAG_JTDO : Aux Func.0
JTAG_JTMS : Aux Func.0
JTAG_JTCL : Aux Func.0
JTAG_JTRS : Aux Func.0
USB_DRV_V : Aux Func.1
LED_A     : Aux Func.0
LED_B     : Aux Func.0
LED_C     : Aux Func.0
LED_D     : Aux Func.0
LED_E     : Aux Func.0
GPIO_B    : Aux Func.1
GPIO_C    : Aux Func.1
I2C_2_SCL : Aux Func.1
I2C_2_SDA : Aux Func.1
PCIE30_2L : Aux Func.1
PCIE30_1L : Aux Func.1
PCIE30_2L : Aux Func.1
PCIE30_2L : Aux Func.1
PCIE30_1L : Aux Func.1
PCIE30_1L : Aux Func.1
USB_DRV_V : Aux Func.1
UART1_RXD : Aux Func.1
UART1_TXD : Aux Func.1
UART1_CTS : Aux Func.1
UART1_RTS : Aux Func.1

frank-w · March 2, 2024, 6:22am

This thread is about uboot,not linux/openwrt pls do not hijack this thread with non-uboot posts

vexorg · March 2, 2024, 4:56pm

Both of the problems seem to share a common issue: the device tree. I found this thread discussing the M-key PCIe issues while researching the B-key PCIe slot.

frank-w · March 2, 2024, 5:08pm

We are talking about getting nvme slot (pcie3) working in uboot…this is working in our linux kernel, so please stay on topic…nobody talkes about key-b slot (pcie2) or the 2 mpci slots (0+1)

totkeks · March 16, 2024, 6:26pm

Just wanted to check in again, has anyone got any news on this topic? @YuyuMarco maybe? I haven’t played with the source or the board for a while now, but got time again to test out new debug methods or new source code.

Looking at the other thread about the second pcie port ([BPI-R4] how to activate key B PCIe2 @ 11280000? [solved] - #53 by vexorg), it sounds like the clock configuration for the mediatek chip is a bit special sometimes.

frank-w · March 17, 2024, 8:47am

This is only for p2 (the key-b slot)…but i try to get p3 (key m) working…but it seems either pinctrl or clock is missing something

i also tried adding fix for p2, but still same issue (as expected as i only enabled p3 on r4).

--- a/drivers/clk/mediatek/clk-mt7988.c
+++ b/drivers/clk/mediatek/clk-mt7988.c
@@ -802,7 +802,7 @@ static const struct mtk_gate infracfg_mtk_gates[] = {
        GATE_INFRA0(CK_INFRA_PCIE_PERI_26M_CK_P1,
                    "infra_pcie_peri_ck_26m_ck_p1", CK_INFRA_F26M_O0, 8),
        GATE_INFRA0(CK_INFRA_PCIE_PERI_26M_CK_P2,
-                   "infra_pcie_peri_ck_26m_ck_p2", CK_INFRA_F26M_O0, 9),
+                   "infra_pcie_peri_ck_26m_ck_p2", CK_INFRA_PCIE_PERI_26M_CK_P3, 9),
        GATE_INFRA0(CK_INFRA_PCIE_PERI_26M_CK_P3,
                    "infra_pcie_peri_ck_26m_ck_p3", CK_INFRA_F26M_O0, 10),
 };

compared clk with r3mini where pcie works, looks similar

BPI-R3M> pci enum
drivers/pci/pcie_mediatek_gen3.c:mtk_pcie_startup_port[261] detected a card
set trans table 0: 0x20000000 0x20000000, 0x10000000
BPI-R3M> clk dump
 Rate               Usecnt      Name
------------------------------------------
 12000000             0        |-- dummy12m
BPI-R3M> nvme scan
BPI-R3M> nvme info                                                              
Device 0: Vendor: 0x1c5c Rev: 80002C00 Prod: ND94N163610404F0R                  
            Type: Hard Disk                                                     
            Capacity: 244198.3 MB = 238.4 GB (500118192 x 512)                  
BPI-R3M>

it looks like clk command is not able to print enabled clocks somehow…

mtk-clock driver seems to use UCLASS_CLK , so it is strange that clocks are not listed

github.com

u-boot/u-boot/blob/master/cmd/clk.c#L70


      
          }
          
          static int soc_clk_dump(void)
          {
          	struct udevice *dev;
          	const struct clk_ops *ops;
          
          	printf(" Rate               Usecnt      Name\n");
          	printf("------------------------------------------\n");
          
          	uclass_foreach_dev_probe(UCLASS_CLK, dev)
          		show_clks(dev, -1, 0);
          
          	uclass_foreach_dev_probe(UCLASS_CLK, dev) {
          		ops = dev_get_driver_ops(dev);
          		if (ops && ops->dump) {
          			printf("\n%s %s:\n", dev->driver->name, dev->name);
          			ops->dump(dev);
          		}
          	}

@hackpascal can you help here?

hackpascal · March 25, 2024, 9:44am

@sam33 can help you with this issue

frank-w · March 25, 2024, 12:16pm

Sam is already in mail loop with daniel,john and steven. I only posted state and tried to figure out why clocks are not displayed in uboot. Thought you maybe have an idea about this.

hackpascal · March 26, 2024, 12:34am

hmm…Sam is fully in charge of the clk drivers. According to his description, the clk driver in u-boot is designed in a non-standard way that all clocks can’t be shown by the clk command.

frank-w · March 26, 2024, 7:13am

Which description do you mean? I did not see this info in files or commits yet.

hackpascal · March 26, 2024, 7:20am

internal discussion

b33cb3975b · April 3, 2024, 6:53am

Hoping @sam33 / @hackpascal have managed to shed some light into their unnatural implementation choices, @frank-w - nvme for the rootfs is the last item I’m personally hoping can be enabled for the bpi-r4; everything else seems to be working in my arrangement (with everything aside from the cpu fan fixes already having been mainlined).

In the mean time, @hackpascal / @sam33 , any details you can share here (beyond just giggling, @hackpascal ) would be helpful for others attempting to debug their way through this implementation process

@dangowrt , @frank-w , you guys are champions (@YuyuMarco too!)

frank-w · April 3, 2024, 5:12pm

With big help of @sam33 i got it working when pcie2 is disabled. It seems that the not-detected slots breaking the pci core. So not yet final but working so far which is a huge step forward.

Merged it to my “main” tree,so the changes are also available by releases.

Last release also contains r3mini network fix (allow now multiple access).

totkeks · April 4, 2024, 12:03am

Does this mean when there is no other card used in the mini pcie slots? Or disabled via the device tree of the board?

b33cb3975b · April 4, 2024, 1:12am

Heck yes Awesome, @frank-w, I appreciate it I’ll pull/cherry-pick and give it a try - thanks a ton, @sam33

I’m assuming the menu interface for provisioning the nvme volume/configuring uboot env is still needing some implementation - once I’ve gotten a build that’s working + manually installed, I’ll take a crack at replicating these steps with the process that ends up working for me/adjusting the existing menu interface items to include nvme as an option: sdmmc, snand, emmc

frank-w · April 4, 2024, 6:17am

Pcie2 is the keyb slot and empty in my case…have not tried without mpcie card but these are after the key-m so i guess this should not break

Have not done anything with menu for nvme yet,but if you use my precompiled you can add menuitems with uEnv.txt

btw. steps that have to be done (my nvme is currently empty, so i’m stuck here atm…need to create partitiontable first ):

BPI-R4> if pci enum; then echo success;fi                                       
drivers/pci/pcie_mediatek_gen3.c:mtk_pcie_startup_port[262] detected a card     
set trans table 0: 0x28200000 0x28200000, 0x7e00000                             
drivers/pci/pcie_mediatek_gen3.c:mtk_pcie_startup_port[262] detected a card     
set trans table 0: 0x30200000 0x30200000, 0x7e00000                             
drivers/pci/pcie_mediatek_gen3.c:mtk_pcie_startup_port[262] detected a card     
set trans table 0: 0x38200000 0x38200000, 0x7e00000                             
success                                                                         
BPI-R4> if nvme scan; then echo success;fi                                      
success                                                                         
BPI-R4> if nvme info; then echo success;fi                                      
Device 0: Vendor: 0x1cc1 Rev: VC0S036H Prod: 2M232LAJ8HNT                       
            Type: Hard Disk                                                     
            Capacity: 476940.0 MB = 465.7 GB (976773168 x 512)                  
success                                                                         
BPI-R4> ls nvme 0:1                                                             
** No partition table - nvme 0 **                                               
Couldn't find partition nvme 0:1

after creating partitions i see my testfile

BPI-R4> if nvme info;then echo success;fi                                       
Device 0: Vendor: 0x1cc1 Rev: VC0S036H Prod: 2M232LAJ8HNT                       
            Type: Hard Disk                                                     
            Capacity: 476940.0 MB = 465.7 GB (976773168 x 512)                  
success                                                                         
BPI-R4> ls nvme 0:1                                                             
        0   bpi-boot-vfat                                                       
                                                                                
1 file(s), 0 dir(s)                                                             
                                                                                
BPI-R4> ls nvme 0:2                                                             
<DIR>       4096 .                                                              
<DIR>       4096 ..                                                             
<DIR>      16384 lost+found                                                     
               0 bpi-root-ext                                                   
BPI-R4>
BPI-R4> if fatload nvme 0:1 $loadaddr bpi-boot-vfat;then echo load success;fi
0 bytes read in 0 ms
load success
BPI-R4>

if the loaded file (e.g. from environment - fit var) you can continue with the boot-chain (bootm $loadaddr#bootconf). maybe you can copy useusb as usenvme and change related parts (that would my way).

seems nvme info returns also “success” if no device is found…mhm, imho we need to access partition to check if nvme works via script

have it now defined like this:

usenvme=pci enum; nvme scan;nvme info; if ls nvme 0:1;then setenv device nvme;setenv partition 0:1;else echo nvme partition not found;fi

seems to work

but it would be interesting, if nvme also works if there is nothing in mpcie slots

frank-w · April 6, 2024, 6:48pm

added nvme boot in my uboot 2024-04-bpi tree for r3 and r4 (currently tested only on r3mini, but should work across all devices)

totkeks · April 6, 2024, 7:07pm

Cool, thanks for the update. Will try on my R4 the next days. Currently working on a windows build setup using Powershell and Docker to build things my way. Though I also got your build.sh working in WSL.

What are the chances of this patch making it to mainline U-Boot, since what I understood was, that this was more a workaround or hack than a final fix to the issue?

frank-w · April 6, 2024, 7:33pm

John will try to upstream nvme driver next week,not sure about mt7988 support as there is the problem with the key-b slot i currently have disabled. So from driver view it will be upstreamed.

The builtin environment is my specific way to use this driver and this is not upstreamable…maybe @dangowrt add nvme boot in a similar way. If you do not want the ask for filename you have to patch it out…at this point i want it configurable

If you have it working in docker/wsl please share way how to setup…maybe anybody else will use this too