MD Drivers for RAID

[william]

HI Frank,

I’ve wondering if there is intention on building one of your builds with the MD drivers? I currently have your 4.14.13 installed and noticed there is no RAID support yet.

I attempted to compile a new build but I’m far from knowledgeable enough to know the full process in doing so. I’m having difficulty in locating the source to gcc-arm-linux-gnueabihf.

Also has the HDMI (video) been enabled and working in 4.14 or better? I’m a little confused with the forums/documents in stating whats working and not. The forum states it is not but the readme says it is.

Also from in reading I want to thank you for your outstanding work you have been doing with the BPI

[frank-w]

Hi,

4.14.13 is a bt old. Current version is 4.14.53.

frank-w/BPI-R2-4.14/blob/12e0082dcb395200bc1b958d65351807887e4e15/arch/arm/configs/mt7623n_evb_fwu_defconfig#L294

Basic support is enabled, but i have not tested yet so maybe anything is missing.

hdmi is working with basic resolutions since 4.14.4x

[frank-w]

Just to continue from here: (precompiled) Kernels for BPI-R2

@william have you any errors/warnings while creating the array? Maybe anything is missing, the pcie-card is full recognized??

[william]

No issues with the PCIe card. I’ve copied 16GB images each disk separately multiple times.

> Disk /dev/sdb: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> Disklabel type: gpt
> Disk identifier: 13857800-19D0-455B-A98B-8F8670E8F7E5
> 
> Device     Start        End    Sectors  Size Type
> /dev/sdb1   2048 7814035455 7814033408  3.7T Linux filesystem
> 
> 
> Disk /dev/sdc: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> Disklabel type: gpt
> Disk identifier: 13857800-19D0-455B-A98B-8F8670E8F7E5
> 
> Device     Start        End    Sectors  Size Type
> /dev/sdc1   2048 7814035455 7814033408  3.7T Linux filesystem
> 
> 
> Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> Disklabel type: gpt
> Disk identifier: 7E225729-72A8-4BDF-82B3-286181C32471
> 
> Device     Start        End    Sectors  Size Type
> /dev/sda1   2048 7814035455 7814033408  3.7T Linux filesystem
> 
> root@bpi-r2:~# mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sd[abc]1

> mdadm: layout defaults to left-symmetric

> mdadm: layout defaults to left-symmetric

> mdadm: chunk size defaults to 512K

> mdadm: /dev/sda1 appears to contain an ext2fs file system
>        size=3907016704K  mtime=Wed Dec 31 16:00:00 1969

> mdadm: /dev/sdb1 appears to contain an ext2fs file system
>        size=3907016704K  mtime=Wed Dec 31 16:00:00 1969

> mdadm: /dev/sdc1 appears to contain an ext2fs file system
>        size=3907016704K  mtime=Wed Dec 31 16:00:00 1969

> mdadm: size set to 3906885632K

> mdadm: automatically enabling write-intent bitmap on large array

> Continue creating array? y

> **mdadm: unexpected failure opening /dev/md0**

As I said before everything I’ve read this error expecting MD to be a loadable module. So maybe there is stuff missing. I haven’t gotten that deep into it yet. I’ve mknode(s) and set proper permissions but no luck. I have been trying to find something similar where MD is builtin but not found much yet. I did however see one blip that it actually preferred.

[frank-w]

You can try to build Kernel with md as module…imho it is better build-in because rootfs can also be on raid-system this way…i don’t think the problem is md-builtin

What says

cat /proc/mdstat

After creating the raid? at least this is missing

Symbol: DM_RAID [=n]                                                    │  
  │ Type  : tristate                                                        │  
  │ Prompt: RAID 1/4/5/6/10 target   

i add raid-options and upload image again => done…same name 4.14.53-main

[william]

That did it Fank, You are the man !

It is currently building the raid…

Every 2.0s: cat /proc/mdstat bpi-r2: Sun Jul 8 13:00:33 2018

Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md0 : active raid5 sdc1[3] sdb1[1] sda1[0] 7813771264 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_] [>…] recovery = 0.3% (13682200/3906885632) finish=1318.0min speed=49230K/sec bitmap: 0/30 pages [0KB], 65536KB chunk

unused devices:

This is Raid5 with three disks. I can’t use the 4th until the new enclosure comes. Crossing fingers I have no problem with that. But at least I can run some tests.

Thank you

[william]

I think I may have spoken too soon about things working well. Not sure whats going on yet but it seems the SATA bus disconnects. Waiting and it recovers.

> 42.180155] device eth0 entered promiscuous mode
> [   43.203289] mt7530 switch@0 lan0: Link is Down
> [   43.761852] mt7530 switch@0 lan3: Link is Down
> [   46.321931] mt7530 switch@0 lan0: Link is Up - 1Gbps/Full - flow control off
> [  469.678548] ata1.00: failed to read SCR 1 (Emask=0x40)
> [  469.685171] ata1.01: failed to read SCR 1 (Emask=0x40)
> [  469.690543] ata1.02: failed to read SCR 1 (Emask=0x40)
> [  469.695830] ata1.03: failed to read SCR 1 (Emask=0x40)
> [  469.701144] ata1.04: failed to read SCR 1 (Emask=0x40)
> [  469.706425] ata1.05: failed to read SCR 1 (Emask=0x40)
> [  469.711742] ata1.06: failed to read SCR 1 (Emask=0x40)
> [  469.717029] ata1.07: failed to read SCR 1 (Emask=0x40)
> [  469.722336] ata1.08: failed to read SCR 1 (Emask=0x40)
> [  469.727617] ata1.09: failed to read SCR 1 (Emask=0x40)
> [  469.732915] ata1.10: failed to read SCR 1 (Emask=0x40)
> [  469.738223] ata1.11: failed to read SCR 1 (Emask=0x40)
> [  469.743500] ata1.12: failed to read SCR 1 (Emask=0x40)
> [  469.748796] ata1.13: failed to read SCR 1 (Emask=0x40)
> [  469.754075] ata1.14: failed to read SCR 1 (Emask=0x40)
> [  469.759389] ata1.15: exception Emask 0x14 SAct 0x0 SErr 0x680100 action 0x6 frozen
> [  469.767165] ata1.15: SError: { UnrecovData 10B8B BadCRC Handshk }
> [  469.773450] ata1.00: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> [  469.780902] ata1.01: exception Emask 0x100 SAct 0x7fffff01 SErr 0x0 action 0x6 frozen
> [  469.788962] ata1.01: failed command: READ FPDMA QUEUED
> [  469.794254] ata1.01: cmd 60/40:00:b0:ac:7e/05:00:05:00:00/40 tag 0 ncq dma 688128 in
>                         res 40/00:01:06:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  469.809776] ata1.01: status: { DRDY }
> [  469.813542] ata1.01: failed command: READ FPDMA QUEUED
> [  469.818855] ata1.01: cmd 60/40:40:f0:33:7e/05:00:05:00:00/40 tag 8 ncq dma 688128 in
>                         res 40/00:01:09:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  469.834377] ata1.01: status: { DRDY }
> [  469.838166] ata1.01: failed command: READ FPDMA QUEUED
> [  469.843454] ata1.01: cmd 60/40:48:30:39:7e/05:00:05:00:00/40 tag 9 ncq dma 688128 in
>                         res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  469.858978] ata1.01: status: { DRDY }
> [  469.862742] ata1.01: failed command: READ FPDMA QUEUED
> [  469.868052] ata1.01: cmd 60/40:50:70:3e:7e/05:00:05:00:00/40 tag 10 ncq dma 688128 in
>                         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  469.883672] ata1.01: status: { DRDY }
> [  469.887435] ata1.01: failed command: READ FPDMA QUEUED
> [  469.892744] ata1.01: cmd 60/40:58:b0:43:7e/05:00:05:00:00/40 tag 11 ncq dma 688128 in
>                         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  469.908361] ata1.01: status: { DRDY }
> [  469.912125] ata1.01: failed command: READ FPDMA QUEUED
> [  469.917410] ata1.01: cmd 60/40:60:f0:48:7e/05:00:05:00:00/40 tag 12 ncq dma 688128 in
>                         res 40/00:01:09:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  469.933018] ata1.01: status: { DRDY }
> [  469.936782] ata1.01: failed command: READ FPDMA QUEUED
> [  469.942093] ata1.01: cmd 60/40:68:30:4e:7e/05:00:05:00:00/40 tag 13 ncq dma 688128 in
>                         res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  469.957715] ata1.01: status: { DRDY }
> [  469.961506] ata1.01: failed command: READ FPDMA QUEUED
> [  469.966795] ata1.01: cmd 60/40:70:70:53:7e/05:00:05:00:00/40 tag 14 ncq dma 688128 in
>                         res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  469.982409] ata1.01: status: { DRDY }
> [  469.986173] ata1.01: failed command: READ FPDMA QUEUED
> [  469.991481] ata1.01: cmd 60/40:78:b0:58:7e/05:00:05:00:00/40 tag 15 ncq dma 688128 in
>                         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.007102] ata1.01: status: { DRDY }
> [  470.010888] ata1.01: failed command: READ FPDMA QUEUED
> [  470.016177] ata1.01: cmd 60/40:80:f0:5d:7e/05:00:05:00:00/40 tag 16 ncq dma 688128 in
>                         res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.031788] ata1.01: status: { DRDY }
> [  470.035552] ata1.01: failed command: READ FPDMA QUEUED
> [  470.040866] ata1.01: cmd 60/40:88:30:63:7e/05:00:05:00:00/40 tag 17 ncq dma 688128 in
>                         res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.065006] ata1.01: status: { DRDY }
> [  470.077435] ata1.01: failed command: READ FPDMA QUEUED
> [  470.091315] ata1.01: cmd 60/40:90:70:68:7e/05:00:05:00:00/40 tag 18 ncq dma 688128 in
>                         res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.124172] ata1.01: status: { DRDY }
> [  470.136836] ata1.01: failed command: READ FPDMA QUEUED
> [  470.151085] ata1.01: cmd 60/40:98:b0:6d:7e/05:00:05:00:00/40 tag 19 ncq dma 688128 in
>                         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.185054] ata1.01: status: { DRDY }
> [  470.198253] ata1.01: failed command: READ FPDMA QUEUED
> [  470.213065] ata1.01: cmd 60/40:a0:f0:72:7e/05:00:05:00:00/40 tag 20 ncq dma 688128 in
>                         res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.248086] ata1.01: status: { DRDY }
> [  470.261824] ata1.01: failed command: READ FPDMA QUEUED
> [  470.277856] ata1.01: cmd 60/40:a8:30:78:7e/05:00:05:00:00/40 tag 21 ncq dma 688128 in
>                         res 40/00:01:06:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.314072] ata1.01: status: { DRDY }
> [  470.328214] ata1.01: failed command: READ FPDMA QUEUED
> [  470.343813] ata1.01: cmd 60/40:b0:70:7d:7e/05:00:05:00:00/40 tag 22 ncq dma 688128 in
>                         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.380043] ata1.01: status: { DRDY }
> [  470.394194] ata1.01: failed command: READ FPDMA QUEUED
> [  470.409756] ata1.01: cmd 60/40:b8:b0:82:7e/05:00:05:00:00/40 tag 23 ncq dma 688128 in
>                         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.445894] ata1.01: status: { DRDY }
> [  470.460008] ata1.01: failed command: READ FPDMA QUEUED
> [  470.475583] ata1.01: cmd 60/40:c0:f0:87:7e/05:00:05:00:00/40 tag 24 ncq dma 688128 in
>                         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.511720] ata1.01: status: { DRDY }
> [  470.525848] ata1.01: failed command: READ FPDMA QUEUED
> [  470.541423] ata1.01: cmd 60/40:c8:30:8d:7e/05:00:05:00:00/40 tag 25 ncq dma 688128 in
>                         res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.577588] ata1.01: status: { DRDY }
> [  470.591722] ata1.01: failed command: READ FPDMA QUEUED
> [  470.607318] ata1.01: cmd 60/40:d0:70:92:7e/05:00:05:00:00/40 tag 26 ncq dma 688128 in
>                         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.643499] ata1.01: status: { DRDY }
> [  470.657647] ata1.01: failed command: READ FPDMA QUEUED
> [  470.673239] ata1.01: cmd 60/40:d8:b0:97:7e/05:00:05:00:00/40 tag 27 ncq dma 688128 in
>                         res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.709438] ata1.01: status: { DRDY }
> [  470.723581] ata1.01: failed command: READ FPDMA QUEUED
> [  470.739188] ata1.01: cmd 60/40:e0:f0:9c:7e/05:00:05:00:00/40 tag 28 ncq dma 688128 in
>                         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.775382] ata1.01: status: { DRDY }
> [  470.789533] ata1.01: failed command: READ FPDMA QUEUED
> [  470.805128] ata1.01: cmd 60/40:e8:30:a2:7e/05:00:05:00:00/40 tag 29 ncq dma 688128 in
>                         res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.841347] ata1.01: status: { DRDY }
> [  470.855503] ata1.01: failed command: READ FPDMA QUEUED
> [  470.871113] ata1.01: cmd 60/40:f0:70:a7:7e/05:00:05:00:00/40 tag 30 ncq dma 688128 in
>                         res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
> [  470.907331] ata1.01: status: { DRDY }
> [  470.921510] ata1.02: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> [  470.939314] ata1.03: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> [  470.956999] ata1.04: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> [  470.974495] ata1.05: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> [  470.991916] ata1.06: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> [  471.009185] ata1.07: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> [  471.026335] ata1.08: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> [  471.043302] ata1.09: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> [  471.060197] ata1.10: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> [  471.076926] ata1.11: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> [  471.093537] ata1.12: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> [  471.109969] ata1.13: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> [  471.126348] ata1.14: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> [  471.142567] ata1.15: hard resetting link
> [  472.768311] ata1.15: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
> [  472.787580] ata1.00: hard resetting link
> [  473.299890] ata1.00: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
> [  473.338597] ata1.01: hard resetting link
> [  473.849801] ata1.01: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
> [  473.867275] ata1.02: hard resetting link
> [  474.380111] ata1.02: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
> [  474.449081] ata1.03: hard resetting link
> [  474.793086] ata1.03: SATA link down (SStatus 0 SControl 330)
> [  474.867010] ata1.04: hard resetting link
> [  475.212311] ata1.04: SATA link down (SStatus 0 SControl 330)
> [  475.224485] ata1.05: hard resetting link
> [  475.561694] ata1.05: SATA link down (SStatus 0 SControl 330)
> [  475.584152] ata1.06: hard resetting link
> [  475.933008] ata1.06: SATA link down (SStatus 0 SControl 330)
> [  475.962488] ata1.07: hard resetting link
> [  476.302203] ata1.07: SATA link down (SStatus 0 SControl 330)
> [  476.322871] ata1.08: hard resetting link
> [  476.663467] ata1.08: SATA link down (SStatus 0 SControl 330)
> [  476.698746] ata1.09: hard resetting link
> [  477.042274] ata1.09: SATA link down (SStatus 0 SControl 330)
> [  477.062441] ata1.10: hard resetting link
> [  477.402273] ata1.10: SATA link down (SStatus 0 SControl 330)
> [  477.422393] ata1.11: hard resetting link
> [  477.762872] ata1.11: SATA link down (SStatus 0 SControl 330)
> [  477.785653] ata1.12: hard resetting link
> [  478.132415] ata1.12: SATA link down (SStatus 0 SControl 330)
> [  478.159996] ata1.13: hard resetting link
> [  478.501144] ata1.13: SATA link down (SStatus 0 SControl 330)
> [  478.523434] ata1.14: hard resetting link
> [  478.860552] ata1.14: SATA link down (SStatus 0 SControl 330)
> [  478.886639] ata1.00: configured for UDMA/133
> [  478.900221] ata1.01: configured for UDMA/133
> [  478.910723] ata1.02: configured for UDMA/133
> [  478.922206] ata1.01: device reported invalid CHS sector 0
> [  478.933670] ata1: EH complete

[frank-w]

Seems like cable/drive error…

READ FPDMA QUEUED
device reported invalid CHS sector 0

[william]

I’m thinking It is this enclosure. The new one comes tomarrow. Have to wait and see.

[william]

Update to All:

Working Hardware:
4 Western Digital Blue - WDC WD40EZRZ-00GXCB0 - 4TB
Syba SY-ENC50104 4 Bay 3.5” SATA III HDD NON-RAID Enclosure – Supports USB 3.0 & eSATA Interface
IOCREST - mini PCIe SATA card - AS1061R

This error would occur and the system would redirect the attached drives. Things would work for a little while and the error would occur again. When mdadm was building the RAID it would happen about every 30 seconds. It took 2 days to finish building the RAID with no issues. Just this Error happening causing the redirects and the delays. When it was done I built the ext4 filesystem. The same occurrences would repeat util it finished. This took about 1 hour. When it was done I mounted the RAID and started testing file transfers. File transfers were showing the same results.

At this point I decided to turn off the WRITE CACHE to the drives. When I did this amazingly the problem stopped. I’ve been doing file transfer test for 2 days now with out an occurrence yet.

ERROR message
[  750.620645] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[  750.627869] ata1.00: failed command: FLUSH CACHE EXT
[  750.632908] ata1.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 8
[  750.632908]          res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[  750.646454] ata1.00: status: { DRDY }
[  750.650299] ata1: hard resetting link

Google Search most people say it is a SATA cable. I’ve tried 4 different cables all with exact same results. It is not a cable issue. Others say it is hard drive issue. Smart info below indicating no issue. The same for all 4 drives. The drives are all brand-new. Tried a different PCIe SATA card - AS1061R

Smart info: === START OF INFORMATION SECTION === Model Family: Western Digital Blue Device Model: WDC WD40EZRZ-00GXCB0 Serial Number: WD-XXXXXXXXXXXX LU WWN Device Id: 5 0014ee 20ff6bc36 Firmware Version: 80.00A80 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-3 T13/2161-D revision 5 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Jul 8 18:34:16 2018 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED 

Performance was not as good as I would have liked. I received the replacement enclosure were that turned out to be a total mess. None of the drives were even being recognized. I return the enclosure for a replacement and the replacement worked. I tried with the WRITE CACHE back on and it still is working flawlessly.

So it seems the problems were compounded by both the enclosure and the miniPCIe card.

I have been testing ever sense. Passing close to 30TB in read and writes to the disk without error. Performance is what is expected with a software raid. > /dev/md0:

     Timing cached reads:   1036 MB in  2.00 seconds = 517.83 MB/sec
     Timing buffered disk reads: 174 MB in  3.01 seconds =  57.81 MB/sec

Thank you very much Frank

BPI should be paying you for all your contribution !

[frank-w]

Also some users reported that sata-ports not soldered correctly…maybe you can check this too