[BPI-R4] Failed attempt: add NVMe support on uboot

Banana Pi Router design BPI-R4/BPI-R4 Pro(MT7988)
60

Hi frank-w,

I was looking in U-boot for MT7988 NVME boot and I notice that is not working. I checked this thread and I’ve applied the patch to my U-boot and I got this:

MT7988> pci enum
detected a card at 0x11290000
no card detected at 0x11300000
no card detected at 0x11310000
MT7988> pci
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
00.00.00   0x14c3     0x7988     Bridge device           0x04
01.00.00   0x1ed0     0x2283     Mass storage controller 0x08
no card detected at 0x11300000
MT7988> nvme scan
MT7988> nvme info
Device 0: Vendor: 0x1ed0 Rev: APF1M3R1 Prod: 702172310010561     
            Type: Hard Disk
            Capacity: 244198.3 MB = 238.4 GB (500118192 x 512)
MT7988> nvme detail
Blk device 0: Optional Admin Command Support:
        Namespace Management/Attachment: no
        Firmware Commit/Image download: yes
        Format NVM: yes
        Security Send/Receive: yes
Blk device 0: Optional NVM Command Support:
        Reservation: yes
        Save/Select field in the Set/Get features: no
        Write Zeroes: yes
        Dataset Management: yes
        Write Uncorrectable: no
Blk device 0: Format NVM Attributes:
        Support Cryptographic Erase: No
        Support erase a particular namespace: Yes
        Support format a particular namespace: No
Blk device 0: LBA Format Support:
        LBA Format 0 Support: (current)
                Metadata Size: 0
                LBA Data Size: 512
                Relative Performance: Better
Blk device 0: End-to-End DataProtect Capabilities:
        As last eight bytes: No
        As first eight bytes: No
        Support Type3: No
        Support Type2: No
        Support Type1: No
Blk device 0: Metadata capabilities:
        As part of a separate buffer: No
        As part of an extended data LBA: No
MT7988> nvme device 

IDE device 0: Vendor: 0x1ed0 Rev: APF1M3R1 Prod: 702172310010561     
            Type: Hard Disk
            Capacity: 244198.3 MB = 238.4 GB (500118192 x 512)
MT7988> nvme part  

Partition Map for nvme device 0  --   Partition Type: EFI

Part    Start LBA       End LBA         Name
        Attributes
        Type GUID
        Partition GUID
  1     0x00008000      0x1dcf2fff      ""
        attrs:  0x0000000000000000
        type:   0fc63daf-8483-4772-8e79-3d69d8477de4
                (linux)
        guid:   110b369b-ed3e-4244-9e89-d59fa1864e8b
MT7988> ls nvme 0:1 
<DIR>       4096 .
<DIR>       4096 ..
<DIR>      16384 lost+found

So I was able to see the FS on NVME in U-boot.

I’m using a BananaPI R4 brought 1-2 weeks ago.

U-boot version is:

commit fe2ce09a0753634543c32cafe85eb87a625f76ca (HEAD -> master, origin/master, origin/HEAD)
Merge: 16324b43db 1fd754cebd
Author: Tom Rini <[email protected]>
Date:   Tue Jun 18 08:34:56 2024 -0600
61

looks like you got it working, and yes, it is not yet merged as we have to do OF_UPSTREAM first which is very much work (requires complete clock driver change and some other driver patches to be compatible with upstream DTS)…

1 Like
62

I hope you will be able to push it to the mainline. Now I will be able to use NVME SSD to build my Booting and root FS on it and give it a try :smiley:

63

I was able to patch the latest version of U-boot:

commit fd46ea0e701920eb205c2bce9d527bf0dec10b59 (HEAD -> master, origin/master, origin/HEAD)
Merge: 18908395ed 7d22317a18
Author: Tom Rini <[email protected]>
Date:   Fri Jul 5 16:36:43 2024 -0600

And confirm that is still working.

64

Hi @frank-w

Seems that U-boot 202407 (not 202407-rc5) has already pushed PCIe 3.0 for Mediatek MT7988 driver in mainline. But still missing DTS patch for PCIe. I was able to build 202407 with partial patch from original patch, just to add missing devices in DTS and is working.

1 Like
65

Yes,dts patches are rejected because boards are not of_upstream yet,but good to know that dts patches still working…christian made a bit different handling of the optional phy…only tested it with of_upstream conversion

It is merged for 2024.10…not in 2024.07 release

1 Like
66

Hi all,

I was playing around with latest version of U-Boot 2025.01. I’ve noticed that the PCIe 3.0 driver is already in U-Boot but DTS for PCIe is not yet there. So I took the DTS PCIe part from 202410 from Frank-W and I’ve ported to the new U-Boot version.

Then, success!

## Starting application at 0x41E00000 ...


U-Boot 2025.01-rc3-EasyOS+ (Dec 03 2024 - 02:11:51 +0100)

CPU:   MediaTek MT7988
Model: mt7988-rfb
DRAM:  4 GiB
Core:  54 devices, 22 uclasses, devicetree: separate
MMC:   mmc@11230000: 0
Loading Environment from nowhere... OK
In:    serial@11000000
Out:   serial@11000000
Err:   serial@11000000
Net:   
Warning: ethernet@15100000 (eth0) using random MAC address - aa:88:d9:66:02:88
eth0: ethernet@15100000
LED 'blue:status' not found (err=-19)
LED 'green:status' not found (err=-19)
Starting EasyOS ..

EEEEEEEEEEEEEEEEEEEEEE                                                             OOOOOOOOO        SSSSSSSSSSSSSSS 
E::::::::::::::::::::E                                                           OO:::::::::OO    SS:::::::::::::::S
E::::::::::::::::::::E                                                         OO:::::::::::::OO S:::::SSSSSS::::::S
EE::::::EEEEEEEEE::::E                                                        O:::::::OOO:::::::OS:::::S     SSSSSSS
  E:::::E       EEEEEE  aaaaaaaaaaaaa      ssssssssssyyyyyyy           yyyyyyyO::::::O   O::::::OS:::::S            
  E:::::E               a::::::::::::a   ss::::::::::sy:::::y         y:::::y O:::::O     O:::::OS:::::S            
  E::::::EEEEEEEEEE     aaaaaaaaa:::::ass:::::::::::::sy:::::y       y:::::y  O:::::O     O:::::O S::::SSSS         
  E:::::::::::::::E              a::::as::::::ssss:::::sy:::::y     y:::::y   O:::::O     O:::::O  SS::::::SSSSS    
  E:::::::::::::::E       aaaaaaa:::::a s:::::s  ssssss  y:::::y   y:::::y    O:::::O     O:::::O    SSS::::::::SS  
  E::::::EEEEEEEEEE     aa::::::::::::a   s::::::s        y:::::y y:::::y     O:::::O     O:::::O       SSSSSS::::S 
  E:::::E              a::::aaaa::::::a      s::::::s      y:::::y:::::y      O:::::O     O:::::O            S:::::S
  E:::::E       EEEEEEa::::a    a:::::assssss   s:::::s     y:::::::::y       O::::::O   O::::::O            S:::::S
EE::::::EEEEEEEE:::::Ea::::a    a:::::as:::::ssss::::::s     y:::::::y        O:::::::OOO:::::::OSSSSSSS     S:::::S
E::::::::::::::::::::Ea:::::aaaa::::::as::::::::::::::s       y:::::y          OO:::::::::::::OO S::::::SSSSSS:::::S
E::::::::::::::::::::E a::::::::::aa:::as:::::::::::ss       y:::::y             OO:::::::::OO   S:::::::::::::::SS 
EEEEEEEEEEEEEEEEEEEEEE  aaaaaaaaaa  aaaa sssssssssss        y:::::y                OOOOOOOOO      SSSSSSSSSSSSSSS   
                                                           y:::::y                                                  
                                                          y:::::y                                                   
                                                         y:::::y                                                    
                                                        y:::::y                                                     
                                                       yyyyyyy                                                      
Scanning NVME..
Scanning USB..
starting USB...
Bus xhci@11200000: xhci-mtk xhci@11200000: hcd: 0x0000000011200000, ippc: 0x0000000011203e00
xhci-mtk xhci@11200000: ports disabled mask: u3p-0x0, u2p-0x0
xhci-mtk xhci@11200000: u2p:1, u3p:1
Register 200010f NbrPorts 2
Starting the controller
USB XHCI 1.10
scanning bus xhci@11200000 for devices... 4 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
EasyOS(BPi-R4)> pci   
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
00.00.00   0x14c3     0x7988     Bridge device           0x04
01.00.00   0x1ed0     0x2283     Mass storage controller 0x08
EasyOS(BPi-R4)> nvme info
Device 0: Vendor: 0x1ed0 Rev: APF1M3R1 Prod: 702172310010561     
            Type: Hard Disk
            Capacity: 244198.3 MB = 238.4 GB (500118192 x 512)
EasyOS(BPi-R4)> ls nvme 0:1
            ./
            ../
            lost+found/

0 file(s), 3 dir(s)

To implement PCIe 3.0 DTS I’ve created 2 additional .dtsi files: mt7988-pcie-enable.dtsi and mt7988-pcie.dtsi. Then I’ve just included in mt7988.dtsi mt7988-pcie.dtsi like this:

#include <dt-bindings/interrupt-controller/irq.h>
#include <dt-bindings/interrupt-controller/arm-gic.h>
#include <dt-bindings/clock/mt7988-clk.h>
#include <dt-bindings/reset/mt7988-reset.h>
#include <dt-bindings/gpio/gpio.h>
#include <dt-bindings/pinctrl/mt65xx.h>
#include <dt-bindings/phy/phy.h>
#include "mt7988-pcie.dtsi"

/ {
..

And in mt7988-rfb.dts mt7988-sd-rfb.dts like this:

/dts-v1/;
#include "mt7988.dtsi"
#include "mt7988-pcie-enable.dtsi"
#include <dt-bindings/gpio/gpio.h>

I’m attaching the files in case somebody wants to try it.

mt7988-pcie.dtsi (4.8 KB)

mt7988-pcie-enable.dtsi (216 Bytes)

mt7988.dtsi.patch (337 Bytes)

mt7988-rfb.dts.patch (272 Bytes)

mt7988-sd-rfb.dts.patch (278 Bytes)

Files mt7988-pcie-enable.dtsi and mt7988-pcie.dtsi just copy them to arch/arm/dts. The other 3 patches you can use them with patch -p1 -i ... I’ve preferred to keep them in a non-patch format to be able to modify fast.

Or you can use my U-boot 202501 fork in my git hub where I’ve applied these patches: GitHub - EasyNetDev/u-boot at mt7988

1 Like
67

You could leave the status=disabled for the 3 pcie and drop the enable file…to make things easier.

1 Like
68

Indeed I could do it. But I went on this approach to not enable globally the PCIe just for the specific .dtb

69

if you do it for you only, why not :wink:

currently there are only rfb and the bpi-r4…and imho all here having the r4 and no other mt7988 based board, right?

for mainlining of course the “disable first and enable in board” aproach is the way to go

1 Like
70

Yes, was trying to leave close to mainline and in the same time to try to touch as little as possible the original dtsi and dts file.

In this way I can apply the 2 files easier to next U-boot release and make the changes than have a full patch over the 3 main dts/dtsi files.

71

Only wanted to show an easier way till it is not yet supported in mainline uboot (dts changes are stopped till OF_UPSTREAM changes for which i currently upstream linux dts and we need fdtoverlay command built in uboot and merging of dtso to basedt to have the dtbs for linking into uboot).

1 Like
72

Thanks for the tip. In the end I was building a custom .dts for the board and I’m compiling it and used in U-boot.

73

Hello Gentlemen, kudos on your work so far!

I’ve been scouring the internet for anything similar, and came across this active forum post, so thought I’d throw myself out there early rather than regret it later.

I’m working on a product using a mt8395 (same as mt8195 from software perspective), for which we would like to boot into an OS on a NVMe drive. If it would take too many engineering hours, we may swap over to using a UFS drive, but the hardware engineer and I prefer NVMe. Right now I’m doing a little prototyping using the eval board, and at least confirmed the kernel level drivers work well with NVMe out-of-the-box. But, since I’m not seeing native U-boot support for NVMe (at least in MediaTek’s demo branch) I got curious about what’s already out there to address it.

Would you two (or any others) happen to have any suggestions, knowledge and/or advice? I’m a complete novice when it comes to U-boot (several coworkers historically take care of all the U-boot development, but I’ve been looking for a chance to get into it for a while), but I’m sure it’s not going to be simple. Still, I’m hoping I can port the existing upstream PCIe driver and/or your own work to the mt8195 fairly painlessly.

Are either of you aware of a better starting point and/or any major stumbling blocks I should consider before attempting to do so? Or conversely, if there is a better starting point/other project or resource out there which would be a better starting point or additional reference for my use case? Do you think it would be better to port the related files into MediaTek’s branch, or are the changes extensive enough that it would be better to just setup the eval board for upstream U-boot? I saw at least one of the Github links in this forum is dead, (perhaps because it was ported into mainline?), do you guys have all the source contained within one location? If you have any general guidance before I get too deep, please do share!

Thank you in advance and apologies if this is off topic! Good luck on your further developments!

74

please create an own topic as this is not related to bpi-r4, but i’m not sure anyone can help you here as mt8195 is completely different chipset where afaik bpi has no board with

75

hi there, just for me to be sure to understand correctly…this will allow to boot directly on the nvme like for the vanilla openwrt instead of the risky sd card?

76

You cannot boot directly from nvme. You can only boot from spi (nand/nor) or mmc (sd/emmc) and then in uboot continue from nvme (linux kernel,initrd,rootfs).

1 Like
77

Only if someone ever finished this idea. It’s just an idea now and would need someone to build the uboot image for it and the right configuration.

Then technically, yes, it would be possible to boot “straight” (boots from NAND or NOR the uboot, which then activates the nvme and chains to that) from NVME.

78

My uboot has most parts prepared

But for my testing i added a prompt for asking the kernel file,but this could also fixed value or something configurable with env var.

Not sure if openwrt has something similar.

79

if i switch to vanilla openwrt would that be possible then? i am just wondering if i go with an sd card or if i push a bit further to make it more robust with an nvme.