BPI-M4 USB host dying (uvcvideo and usb rtl8821cu) (solution)

greymfm76 · March 29, 2022, 2:46pm

Hi, We purchased 100 pieces BPI-M4 RTD1395 and are trying to get the USB host (USB 2.0) running using an USB uvc camera (elp-usbfhd01m, 2mb pixel usb camera module) and the BPI-M4 built-in USB WiFi (RTL8821CU, 6221E-UUC module). All different uvc-cameras work, however the USB host dies after 1-2 minutes (no further USB communication is possible) while WiFi is activated.

BPI-M4 v1.0 (RTD1395) with the built-in USB WiFi chip (RTL8821CU)
Linux: 2020-05-18-raspbian-stretch-bpi-m4-sd-emmc.img.zip (https://download.banana-pi.dev/d/ca025d76afd448aabc63/?p=%2FImages%2FBPI-M4%2Flinux&mode=list) with BPI-M4 RTD1395 Kernel 4.9.119 drivers
‘uname -a’ gives:

Linux bpi-iot-ros-ai 4.9.119-BPI-M4-Kernel #1 SMP PREEMPT Mon May 18 10:41:36 HKT 2020 aarch64 GNU/Linux

uvc cam: elp-usbfhd01m as well as other tested models (“dmesg” and “lsusb” logs further below)

How to reproduce:

Boot into above Linux image.
Activate WiFi network (do not use wired Ethernet) and log into the Banana PI via WiFi and SSH. Optionally, turn off WiFi power saving:

iw wlan0 set power_save off

“lsmod” shows:

8821cu
rtkbt_usb

Attach USB cam - “dmesg” shows: plug_in.txt (2.2 KB)

“lsusb -t” shows: Screenshot%20from%202022-03-29%2016-27-11

verbose “lsusb -v” shows: lsusb.txt (41.4 KB)

“v4l2-ctl --list-devices” shows:

USB 2.0 Camera (usb-xhci-hcd.4.auto-1.4): /dev/video0

Run test command (which streams from camera to nothing (/dev/null), “usbtop” indicates speed is about 16 MBit/sec):

streamer -q -c /dev/video0 -f rgb24 -s 320x240 -r 10 -t 00:30:00 -o /dev/null

After 1-2 minutes the USB host dies (no further USB communication is possible) - dmesg shows:

> [ 847.874393] xhci-hcd xhci-hcd.4.auto: xHCI host not responding to stop endpoint command. > [ 847.874416] xhci-hcd xhci-hcd.4.auto: Assuming host is dying, halting host.

full dmesg log: usb_load.txt (14.7 KB)

This is the location in the Kernel code that stops the USB host:

github.com

BPI-SINOVOIP/BPI-M4-bsp/blob/d3515ce953acc61d435e4677b4f108c04fcacf3d/linux-rtk/drivers/usb/host/xhci-ring.c#L891


		xhci_kill_ring_urbs(xhci, ring);
	}
	while (!list_empty(&ep->cancelled_td_list)) {
		cur_td = list_first_entry(&ep->cancelled_td_list,
				struct xhci_td, cancelled_td_list);
		list_del_init(&cur_td->cancelled_td_list);
		xhci_giveback_urb_in_irq(xhci, cur_td, -ESHUTDOWN);
	}
}


/* Watchdog timer function for when a stop endpoint command fails to complete.
 * In this case, we assume the host controller is broken or dying or dead.  The
 * host may still be completing some other events, so we have to be careful to
 * let the event ring handler and the URB dequeueing/enqueueing functions know
 * through xhci->state.
 *
 * The timer may also fire if the host takes a very long time to respond to the
 * command, and the stop endpoint command completion handler cannot delete the
 * timer before the timer function is called.  Another endpoint cancellation may
 * sneak in before the timer function can grab the lock, and that may queue
 * another stop endpoint command and add the timer back.  So we cannot use a

Any ideas?

We tested with different USB camera modules (all tested camera modules work perfectly on a laptop), different BPI-M4 boards, additional cooling, even added additional USB hubs with external power - same results.

It would be fine if we can find a solution as we would like to use the 100 pieces Banana BPI-M4 in our robots with an USB camera.

Regards, Alexander

greymfm76 · April 2, 2022, 7:45am

UPDATE: It seem to happen with all our BPI-M4, it just takes longer (1 day) for some, but they all have this USB host dying with BPI-M4 running with USB camera and the built-in WiFi (and without wired Ethernet).

greymfm76 · April 2, 2022, 12:43pm

This is what “/var/log/messages” shows after 1 day running with USB cam and WiFi and without touching the BPI-M4:

Apr  2 03:15:18 bananapi kernel: [38145.860347] xhci-hcd xhci-hcd.4.auto: xHCI host not responding to stop endpoint command.
Apr  2 03:15:18 bananapi kernel: [38145.860354] xhci-hcd xhci-hcd.4.auto: Assuming host is dying, halting host.
Apr  2 03:15:18 bananapi kernel: [38145.876233] RTW: reg 0x210, usb read 4 fail, status:-110 value=0x0, vendorreq_times:1
Apr  2 03:15:18 bananapi kernel: [38145.876262] RTW: reg 0x210, usb read 4 fail, status:-22 value=0x0, vendorreq_times:2
Apr  2 03:15:18 bananapi kernel: [38145.876273] RTW: reg 0x210, usb read 4 fail, status:-22 value=0x0, vendorreq_times:3
Apr  2 03:15:18 bananapi kernel: [38145.876280] RTW: reg 0x210, usb read 4 fail, status:-22 value=0x0, vendorreq_times:4
Apr  2 03:15:18 bananapi kernel: [38145.876283] RTW: ###=> usb_read_port_complete => urb.status(-108)
Apr  2 03:15:18 bananapi kernel: [38145.876288] RTW: [dvobj:ffffffc03a69c000][ERROR] continual_io_error:5 > 4
Apr  2 03:15:18 bananapi kernel: [38145.876293] RTW: usb_read_port_complete() RX Warning! bDriverStopped(True) OR bSurpriseRemoved(False)
Apr  2 03:15:18 bananapi kernel: [38145.876302] RTW: usb_read_port_complete() RX Warning! bDriverStopped(True) OR bSurpriseRemoved(True)
Apr  2 03:15:18 bananapi kernel: [38145.876310] RTW: usb_read_port_complete() RX Warning! bDriverStopped(True) OR bSurpriseRemoved(True)
Apr  2 03:15:18 bananapi kernel: [38145.876319] RTW: usb_read_port_complete() RX Warning! bDriverStopped(True) OR bSurpriseRemoved(True)
Apr  2 03:15:18 bananapi kernel: [38145.876326] RTW: usb_read_port_complete() RX Warning! bDriverStopped(True) OR bSurpriseRemoved(True)
Apr  2 03:15:18 bananapi kernel: [38145.876335] RTW: usb_read_port_complete() RX Warning! bDriverStopped(True) OR bSurpriseRemoved(True)
Apr  2 03:15:18 bananapi kernel: [38145.876345] RTW: usb_read_port_complete() RX Warning! bDriverStopped(True) OR bSurpriseRemoved(True)
Apr  2 03:15:18 bananapi kernel: [38145.876362] RTW: reg 0x4e, usb read 1 fail, status:-22 value=0x4097700, vendorreq_times:1
Apr  2 03:15:18 bananapi kernel: [38145.876369] RTW: [dvobj:ffffffc03a69c000][ERROR] continual_io_error:6 > 4
Apr  2 03:15:18 bananapi kernel: [38145.876377] RTW: reg 0x4e0, usb write 1 fail, status:-22
Apr  2 03:15:18 bananapi kernel: [38145.876392] RTW: ERROR _halmac_reg_write_8: I/O FAIL!
Apr  2 03:15:18 bananapi kernel: [38145.876396] RTW: ERROR _halmac_reg_write_32: I/O FAIL!
Apr  2 03:15:18 bananapi kernel: [38145.876400] RTW: ERROR _halmac_reg_write_8: I/O FAIL!
Apr  2 03:15:18 bananapi kernel: [38145.876404] RTW: ERROR _halmac_reg_write_8: I/O FAIL!
Apr  2 03:15:18 bananapi kernel: [38145.876409] RTW: ERROR _halmac_reg_write_16: I/O FAIL!
Apr  2 03:15:18 bananapi kernel: [38145.876473] usb 1-1: USB disconnect, device number 2
Apr  2 03:15:18 bananapi kernel: [38145.876489] usb 1-1.2: USB disconnect, device number 3

Complete /var/log/messages: var_log_messages.txt (361.4 KB)

Average CPU load is 5-10%, the only process with USB load is the USB streaming test command shown in my first post above. We produce no WiFi traffic during this testing. CPU temperature is ok (not more than 40 degreeC)…

We will try to switch off USB autosuspend for all devices now (like described here) and find out what happens after 1 day…

greymfm76 · April 5, 2022, 1:45am

Adding an external USB hub with external power supply seem to have solved the issue …

greymfm76 · April 15, 2022, 6:46pm

I’m sorry, however the problem appears again after 1 week. The internal USB hub on the BPI-M4 doesn’t seem to work stable for a longer time with an uvc camera. The external USB hub (with splitted, external power supply) can only extend the time until the issue described above happens. If we use another camera (elp-usb3mp01h-l29), the issue even happens with external USB hub after only 1 hour.

greymfm76 · April 20, 2022, 7:48am

It seems you can reproduce a related issue (kernel crashes) like this:

Cut the uvcvideo-camera’s USB DATA+ line and add a jumper, so you can simulate temporary data noise on the USB line to the camera.
Attach your uvcvideo-camera with connected jumper.
Start the camera:

streamer -q -c /dev/video0 -f rgb24 -s 640x480 -r 5 -t 00:30:00 -o /dev/null

Remove the USB DATA+ line jumper for 1sec, then insert it again.
‘dmesg’ will show a kernel-crash (kernel_crash.txt (81.0 KB) ).

[  +1.484271] usb 1-1-port2: port 2, status 0101, change 0001, 12 Mb/s
[  +0.000044] usb 1-1.2: USB disconnect, device number 3
[  +0.000704] uvcvideo: Failed to resubmit video URB (-19).
[  +0.259416] usb 1-1.2: new high-speed USB device number 6 using xhci-hcd
[  +0.126955] uvcvideo: Found UVC 1.00 device HD USB Camera (05a3:4689)
[  +0.008233] uvcvideo 1-1.2:1.0: Entity type for entity Extension 4 was not initialized!
[  +0.000015] uvcvideo 1-1.2:1.0: Entity type for entity Extension 3 was not initialized!
[  +0.000008] uvcvideo 1-1.2:1.0: Entity type for entity Processing 2 was not initialized!
[  +0.000008] uvcvideo 1-1.2:1.0: Entity type for entity Camera 1 was not initialized!
[  +0.000873] input: HD USB Camera as /devices/platform/98013c00.rtk_dwc3_u2host/98029000.dwc3_u2host/xhci-hcd.4.auto/usb1/1-1/1-1.2/1-1.2:1.0/input/input3
[  +0.036585] RTW: wlan0 wakeup m0=0x00000003, ori reg_0x4d4=0x00000003
[  +0.001253] RTW: survey done event(c) band:0 for wlan0
[  +0.000056] RTW: rtw_indicate_scan_done(wlan0)
[  +0.063261] RTW: rtw_set_ps_mode(wlan0) Enter 802.11 power save - WIFI-TRAFFIC_IDLE
[  +0.001069] RTW: rtl8821c_set_FwPwrMode_cmd(wlan0): HW port id=0
[  +0.000009] RTW: rtl8821c_set_FwPwrMode_cmd(wlan0): fw ps mode = LPS, drv ps mode = 2, rlbm = 1 , smart_ps = 2, allQueueUAPSD = 0
[  +0.000005] RTW: rtl8821c_set_FwPwrMode_cmd=> psmode:02, smart_ps:02, PowerState:00
[  +0.833498] sysfs group 'power' not found for kobject 'event2'
[  +0.000030] ------------[ cut here ]------------
[  +0.000021] WARNING: CPU: 3 PID: 2965 at fs/sysfs/group.c:237 sysfs_remove_group+0x98/0xa0
[  +0.000004] Modules linked in: 8821cu rtkbt_usb 
[  +0.000021] CPU: 3 PID: 2965 Comm: streamer Not tainted 4.9.119-BPI-M4-Kernel #8
[  +0.000004] Hardware name: Sinovoip_Bananapi_M4 (DT)

…

Maybe this kernel crash is related to the USB host crash, maybe not…

greymfm76 · April 24, 2022, 8:54pm

Somehow related: the Raspberry PI suffers the same issue (sometimes and with certain Kernel versions) https://archlinuxarm.org/forum/viewtopic.php?f=65&t=14172 Someone suggested to use the ‘USB park mode’… I still do not understand what exactly the problem is here in the Linux kernel (why it does not try a recovery)…

greymfm76 · July 8, 2022, 10:06am

Solution: The solution seem to be to replace the UVC USB video driver in the BPI 4.9.119 Linux kernel by the latest Linux kernel. Replace the uvcvideo.h, uvcvideo.c and configure UVC USB video driver as a kernel module (CONFIG_USB_VIDEO_CLASS=m). Good luck!

rtd139x_bpi_defconfig (146.2 KB) uvcvideo.h (21.9 KB) uvc_video.c (56.4 KB)