So I have embarked on an odd project, just to see if it can be done. I am trying to get my AMD RX580 to work with the Banana Pi R4 but am running into issues with trying to map a big enough BAR space. I have been following Jeff Geerling’s attempts to do similar things on the raspberry pi for inspiration. Increase the BAR memory address space for PCIe devices on the Raspberry Pi Compute Module 4 · GitHub
It looks like I need 512MB to get rid of any BAR issues, and the only way to get that to fit is to move up in memory. I tried 0x16 instead of 0x28 set in the mt7988a.dtsi file:
ranges = <0x81000000 0x00 0x28000000 0x00
0x28000000 0x00 0x00200000>,
<0x82000000 0x00 0x28200000 0x00
0x28200000 0x00 0x07e00000>;
[ 3.384469] pci 0002:00:00.0: BAR 9: no space for [mem size 0x18000000 64bit pref]
[ 3.392027] pci 0002:00:00.0: BAR 9: failed to assign [mem size 0x18000000 64bit pref]
[ 3.399935] pci 0002:00:00.0: BAR 8: assigned [mem 0x28200000-0x282fffff]
[ 3.406714] pci 0002:00:00.0: BAR 0: assigned [mem 0x28300000-0x28307fff 64bit]
[ 3.414018] pci 0002:00:00.0: BAR 7: assigned [io 0x200000-0x200fff]
[ 3.420450] pci 0002:01:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[ 3.428008] pci 0002:01:00.0: BAR 0: failed to assign [mem size 0x10000000 64bit pref]
[ 3.435914] pci 0002:01:00.0: BAR 2: no space for [mem size 0x00200000 64bit pref]
[ 3.443471] pci 0002:01:00.0: BAR 2: failed to assign [mem size 0x00200000 64bit pref]
[ 3.451374] pci 0002:01:00.0: BAR 5: assigned [mem 0x28200000-0x2823ffff]
[ 3.458155] pci 0002:01:00.0: BAR 6: assigned [mem 0x28240000-0x2825ffff pref]
[ 3.465366] pci 0002:01:00.1: BAR 0: assigned [mem 0x28260000-0x28263fff 64bit]
[ 3.472675] pci 0002:01:00.0: BAR 4: assigned [io 0x200000-0x2000ff]
[ 3.479106] pci 0002:00:00.0: PCI bridge to [bus 01]
[ 3.484065] pci 0002:00:00.0: bridge window [io 0x200000-0x200fff]
[ 3.490495] pci 0002:00:00.0: bridge window [mem 0x28200000-0x282fffff]
TO
ranges = <0x81000000 0x00 0x28000000 0x00
0x28000000 0x00 0x00200000>,
<0x82000000 0x00 0x16000000 0x00
0x16000000 0x00 0x20000000>;
[ 3.374405] pci 0002:00:00.0: BAR 9: assigned [mem 0x18000000-0x2fffffff 64bit pref]
[ 3.382136] pci 0002:00:00.0: BAR 8: assigned [mem 0x16000000-0x160fffff]
[ 3.388915] pci 0002:00:00.0: BAR 0: assigned [mem 0x16100000-0x16107fff 64bit]
[ 3.396219] pci 0002:00:00.0: BAR 7: assigned [io 0x200000-0x200fff]
[ 3.402651] pci 0002:01:00.0: BAR 0: assigned [mem 0x20000000-0x2fffffff 64bit pref]
[ 3.410392] pci 0002:01:00.0: BAR 2: assigned [mem 0x18000000-0x181fffff 64bit pref]
[ 3.418136] pci 0002:01:00.0: BAR 5: assigned [mem 0x16000000-0x1603ffff]
[ 3.424917] pci 0002:01:00.0: BAR 6: assigned [mem 0x16040000-0x1605ffff pref]
[ 3.432125] pci 0002:01:00.1: BAR 0: assigned [mem 0x16060000-0x16063fff 64bit]
[ 3.439434] pci 0002:01:00.0: BAR 4: assigned [io 0x200000-0x2000ff]
[ 3.445867] pci 0002:00:00.0: PCI bridge to [bus 01]
[ 3.450821] pci 0002:00:00.0: bridge window [io 0x200000-0x200fff]
[ 3.457252] pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]
[ 3.464030] pci 0002:00:00.0: bridge window [mem 0x18000000-0x2fffffff 64bit pref]
This does solve the BAR issue, but it seems to cause another issue. I can no longer talk to the ROM BAR when attempting modprobe amdgpu.
Before moving to 0x16:
root@OpenWrt:~# modprobe amdgpu
[ 223.070487] [drm] amdgpu kernel modesetting enabled.
[ 223.075582] amdgpu 0002:01:00.0: assign IRQ: got 115
[ 223.080689] amdgpu 0002:01:00.0: enabling device (0000 -> 0003)
[ 223.086629] [drm] initializing kernel modesetting (POLARIS10 0x1002:0x67DF 0x1458:0x22FC 0xE7).
[ 223.095346] [drm] register mmio base: 0x28200000
[ 223.099950] [drm] register mmio size: 262144
[ 223.104323] [drm] add ip block number 0 <vi_common>
[ 223.109190] [drm] add ip block number 1 <gmc_v8_0>
[ 223.113984] [drm] add ip block number 2 <tonga_ih>
[ 223.118763] [drm] add ip block number 3 <gfx_v8_0>
[ 223.123545] [drm] add ip block number 4 <sdma_v3_0>
[ 223.128411] [drm] add ip block number 5 <powerplay>
[ 223.133280] [drm] add ip block number 6 <dm>
[ 223.137538] [drm] add ip block number 7 <uvd_v6_0>
[ 223.142319] [drm] add ip block number 8 <vce_v3_0>
[ 223.355252] amdgpu 0002:01:00.0: amdgpu: Fetched VBIOS from ROM BAR
[ 223.361514] amdgpu: ATOM BIOS: xxx-xxx-xxx
[ 223.365630] [drm] UVD is enabled in VM mode
[ 223.369801] [drm] UVD ENC is enabled in VM mode
[ 223.374326] [drm] VCE enabled in VM mode
[ 223.378238] amdgpu 0002:01:00.0: amdgpu: Trusted Memory Zone (TMZ) feature not supported
[ 223.386321] amdgpu 0002:01:00.0: amdgpu: PCIE atomic ops is not supported
[ 223.393107] [drm] GPU posting now...
[ 223.528769] [drm] vm size is 64 GB, 2 levels, block size is 10-bit, fragment size is 9-bit
[ 223.557315] amdgpu 0002:01:00.0: amdgpu: VRAM: 8192M 0x000000F400000000 - 0x000000F5FFFFFFFF (8192M used)
[ 223.566885] amdgpu 0002:01:00.0: amdgpu: GART: 256M 0x000000FF00000000 - 0x000000FF0FFFFFFF
[ 223.575235] [drm] Detected VRAM RAM=8192M, BAR=0M
[ 223.579928] [drm] RAM width 256bits GDDR5
[ 223.584060] [drm] amdgpu: 8192M of VRAM memory ready
[ 223.589015] [drm] amdgpu: 1964M of GTT memory ready.
[ 223.594020] [drm] GART: num cpu pages 65536, num gpu pages 65536
[ 223.602023] [drm] PCIE GART of 256M enabled (table at 0x000000F5FFF80000).
[ 223.609009] mtk-pcie-gen3 11290000.pcie: msi#0x1 address_hi 0x0 address_lo 0x11290c00 data 1
[ 223.636698] [drm] Chained IB support enabled!
[ 223.668590] amdgpu: hwmgr_sw_init smu backed is polaris10_smu
[ 223.736536] [drm] Found UVD firmware Version: 1.130 Family ID: 16
[ 223.761840] [drm] Found VCE firmware Version: 53.26 Binary ID: 3
[ 223.769200] amdgpu 0002:01:00.0: enabling bus mastering
[ 223.838080] [drm:amdgpu_mm_wdoorbell [amdgpu]] *ERROR* writing beyond doorbell aperture: 0x000001e8!
[ 228.114505] amdgpu: SMU load firmware failed
[ 228.118773] amdgpu: fw load failed
[ 228.122172] amdgpu: smu firmware loading failed
[ 228.126695] amdgpu 0002:01:00.0: amdgpu: amdgpu_device_ip_init failed
[ 228.133130] amdgpu 0002:01:00.0: amdgpu: Fatal error during GPU init
[ 228.139471] amdgpu 0002:01:00.0: amdgpu: amdgpu: finishing device.
After moving to 0x16
root@OpenWrt:/lib/modules/6.6.92# modprobe amdgpu
[ 95.048985] [drm] amdgpu kernel modesetting enabled.
[ 95.054057] amdgpu 0002:01:00.0: assign IRQ: got 115
[ 95.059205] amdgpu 0002:01:00.0: enabling device (0000 -> 0003)
[ 95.065130] [drm] initializing kernel modesetting (POLARIS10 0x1002:0x67DF 0x1458:0x22FC 0xE7).
[ 95.073910] [drm] register mmio base: 0x16000000
[ 95.078523] [drm] register mmio size: 262144
[ 95.082874] [drm] add ip block number 0 <vi_common>
[ 95.087755] [drm] add ip block number 1 <gmc_v8_0>
[ 95.092534] [drm] add ip block number 2 <tonga_ih>
[ 95.097433] [drm] add ip block number 3 <gfx_v8_0>
[ 95.102211] [drm] add ip block number 4 <sdma_v3_0>
[ 95.107094] [drm] add ip block number 5 <powerplay>
[ 95.111960] [drm] add ip block number 6 <dm>
[ 95.116227] [drm] add ip block number 7 <uvd_v6_0>
[ 95.121015] [drm] add ip block number 8 <vce_v3_0>
[ 95.125875] amdgpu 0002:01:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000
[ 95.134593] amdgpu 0002:01:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000
[ 95.143309] amdgpu 0002:01:00.0: amdgpu: Unable to locate a BIOS ROM
[ 95.149657] amdgpu 0002:01:00.0: amdgpu: Fatal error during GPU init
[ 95.156002] amdgpu 0002:01:00.0: amdgpu: amdgpu: finishing device.
[ 95.162189] amdgpu: probe of 0002:01:00.0 failed with error -22
I have tried disabling all other pcie devices in the mt7988a-banana-bpi-r4.dtsi file, and then setting pcie3(where my m.2 to pci-e adapter is plugged in) to the memory range of pcie2 since it starts at 0x20, but I seem to get the same Invalid PCI ROM header error.
I saw in the documentation that the BAR size can be 256MB, so maybe it is just impossible for me to stretch them to 512MB?
0x2000_0000 0x27FF_FFFF 256MB PCIE2 data block (64-bit address)
0x2800_0000 0x2FFF_FFFF 256MB PCIE3 data block (64-bit address)
0x3000_0000 0x37FF_FFFF 256MB PCIE0 data block (64-bit address)
0x3800_0000 0x3FFF_FFFF 256MB PCIE1 data block (64-bit address)
But when I lspci -vv I do see this line which makes me think it is possible.
Capabilities: [200 v1] Physical Resizable BAR
BAR 0: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB
Here is a copy of /proc/iomem I was using to try to find enough room to fit a 512MB BAR space.
root@OpenWrt:~# cat /proc/iomem
0c000000-0c03ffff : GICD
0c080000-0c27ffff : GICR
1000b000-1000bfff : 1001f000.pinctrl eint
1001b000-1001bfff : 1001b000.topckgen topckgen@1001b000
1001c000-1001cfff : 1001c000.watchdog watchdog@1001c000
1001f000-1001ffff : 1001f000.pinctrl gpio_base
10048000-10048fff : 10048000.pwm pwm@10048000
10080000-10080fff : 10080000.pcs pcs@10080000
10081000-10081fff : 10081000.pcs pcs@10081000
100e0000-100e0fff : 100e0000.mcusys mcusys@100e0000
10217080-102170ff : 11003000.i2c i2c@11003000
10217180-102171ff : 11005000.i2c i2c@11005000
11000000-1100001f : serial
11000100-1100011f : serial
11000200-1100021f : serial
11003000-11003fff : 11003000.i2c i2c@11003000
11005000-11005fff : 11005000.i2c i2c@11005000
11007000-110070ff : 11007000.spi spi@11007000
1100a000-1100afff : 1100a000.lvts lvts@1100a000
11200000-11202dff : 11200000.usb mac
11203e00-11203eff : 11200000.usb ippc
11230000-11230fff : 11230000.mmc mmc@11230000
11c10000-11c10fff : 1001f000.pinctrl iocfg_tr_base
11c50000-11c506ff : phy-soc:[email protected] usb-phy@11c50000
11c50700-11c50fff : phy-soc:[email protected] usb-phy@11c50700
11d00000-11d00fff : 1001f000.pinctrl iocfg_br_base
11d20000-11d20fff : 1001f000.pinctrl iocfg_rb_base
11d60000-11d60fff : 11230000.mmc mmc@11230000
11e00000-11e00fff : 1001f000.pinctrl iocfg_lb_base
11e10000-11e103ff : phy-soc:[email protected] usb-phy@11e10000
11e13400-11e138ff : phy-soc:[email protected] usb-phy@11e13000
11f00000-11f00fff : 1001f000.pinctrl iocfg_tl_base
11f20000-11f2ffff : 11f20000.phy phy@11f20000
11f30000-11f3ffff : 11f30000.phy phy@11f30000
11f50000-11f50fff : 11f50000.efuse efuse@11f50000
15020000-15027fff : 15020000.switch switch@15020000
15100000-1517ffff : 15100000.ethernet ethernet@15100000
15400000-1577ffff : 15100000.ethernet ethernet@15100000
40000000-42ffffff : System RAM
42ff0000-42ffffff : reserved
43000000-4304ffff : reserved
43050000-47cbffff : System RAM
46010000-46baffff : Kernel code
46bb0000-46c1ffff : reserved
46c20000-46d6ffff : Kernel data
47cc0000-47dbffff : reserved
47dc0000-4f5fffff : System RAM
4f600000-4f6bffff : reserved
4f6c0000-4f6fffff : System RAM
4f700000-4fefffff : reserved
4ff00000-13fffffff : System RAM
fa7d1000-fe7dbfff : reserved
13b000000-13f5fffff : reserved
13f6ab000-13f76afff : reserved
13f76b000-13f76bfff : reserved
13f76c000-13f7b3fff : reserved
13f7b6000-13f7b6fff : reserved
13f7b7000-13f7b7fff : reserved
13f7b8000-13f7c8fff : reserved
13f7c9000-13fffffff : reserved
So really my main question is does anyone know why I suddenly cannot access ROM BAR space when I move to either 0x16 or 0x20? Also if anyone knows of anywhere to read up more on this kind of stuff I’d appreciate it. Hopefully this dumb project makes someone laugh.