BPI-R3 RTC and PPS

bib · April 18, 2023, 8:35am

As you are missing:

kernel: pps pps0: bound kernel consumer: edge=0x1

it means ntpsec is not using pps.

So, running ntpd --help, the last 3 lines should show something similar to (you need PPS & GPSD):

This version was compiled with the following clock drivers:
LOCAL     GENERIC        NMEA         PPS         SHM
GPSD

if not, then you need to rebuild ntpsec.

If so, it could be the user running ntpd does not have permissions to open /dev/ppsX - check logs.

Read the man page on what stats are produced, create the stats directory with suitable permissions. You should be able to see in clockstats what PPS signals it sees. This is my config without keys and restrict lines and the server line is a reference to gpsd:

logfile   /var/log/ntpstats/ntp.log
logconfig =all
statsdir /var/log/ntpstats/
filegen peerstats  file peerstats  type day enable
filegen loopstats  file loopstats  type day enable
filegen clockstats file clockstats type day enable
filegen protostats file protostats type day enable
filegen rawstats   file rawstats   type day enable
filegen sysstats   file sysstats   type day enable
filegen usestats   file usestats   type day enable
refclock pps unit 0 minpoll 3 maxpoll 3 flag3 1 prefer
server 127.127.28.0 minpoll 4 maxpoll 4

In the ntpsec logs, when started, you should see similar lines to:

2023-04-10T10:46:48 ntpd[1160]: PROTO: 0.0.0.0 c01d 0d kern kernel time sync enabled
2023-04-10T10:46:48 ntpd[1160]: PROTO: 0.0.0.0 c012 02 freq_set kernel -15.644409 PPM
2023-04-10T10:46:48 ntpd[1160]: PROTO: 0.0.0.0 c016 06 restart
2023-04-10T10:47:06 ntpd[1160]: PROTO: 0.0.0.0 c41c 0c clock_step -0.171691 s
2023-04-10T10:47:05 ntpd[1160]: PROTO: 0.0.0.0 c415 05 clock_sync
2023-04-10T10:47:10 ntpd[1160]: PROTO: 0.0.0.0 c418 08 no_sys_peer
2023-04-10T10:47:26 ntpd[1160]: PROTO: 0.0.0.0 011d 0d kern PPS enabled status: 2001 -> 0007

ShawnO · April 18, 2023, 1:20pm

I’ve tried a few different kernels, and I keep seeing the same behavior. Even added a RTC (ds1307 based) and it still wants to keep going off.

I’m honestly getting more curious about this, but just don’t have time at the moment to dig into it deeper.

bib · April 18, 2023, 6:26pm

And you do not seem to have configured PPS kernel consumer support, since there is no CONFIG_NTP_PPS=y line.

bib · April 18, 2023, 6:27pm

You have to build your own kernel since kernel PPS support is not enabled due to NO_HZ_COMMON being selected.

bib · April 18, 2023, 9:17pm

Or perhaps CONFIG_NO_HZ being set to “Y”. What can I say, this side of things is a nightmare…

gustavobsch · April 19, 2023, 3:21pm

I can build my own kernel, and I’ve been doing it all along. I guess my problem is that I’m not sure what to enable. I think the problem is between pps and ntpsec then, beause I definetely can see the pps0 from the OS, so I dont think I missing support.

What version of ntpsec are you using? maybe the different outputs are expected. I’m using prepackaged version 1.2.0+dfsg1-4 on both the RPI and BPI and the RPI has an offset of 0.0002 with the PPS. Same ntpsec config file.

CONFIG_PPS=y
# CONFIG_PPS_DEBUG is not set
# PPS clients support
# CONFIG_PPS_CLIENT_KTIMER is not set
# CONFIG_PPS_CLIENT_LDISC is not set
CONFIG_PPS_CLIENT_GPIO=y
# PPS generators support
root@bpi-r3:~# zcat /proc/config.gz | grep HZ
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
# CONFIG_NO_HZ_IDLE is not set
CONFIG_NO_HZ_FULL=y
CONFIG_NO_HZ=y
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100

gustavobsch · April 19, 2023, 3:28pm

I also tried enabling CONFIG_NTP_PPS but it doesnt seem to work, I found this description, is the BPI-R3 a tickless system?

PPS kernel consumer support

configname: CONFIG_NTP_PPS

Linux Kernel Configuration

└─> Device Drivers

└─> PPS support

└─> PPS kernel consumer support

This option adds support for direct in-kernel time synchronization using an external PPS signal.

It doesn’t work on tickless systems at the moment.

bib · April 19, 2023, 3:51pm

I really despise double negatives that the Americans seem to like so much. It just confuses the hell out of life.

By default, the kernel does NOT support NTP, therefore NO_HZ_COMMON must be unset before NTP_PPS can be selected.

To do so, navigate to General setup->Timers subsystem and select Periodic timer ticks (constant rate, no dynticks).

Choosing either of the other 2 (Idle dynticks system (tickless idle) - NO_HZ_IDLE or Full dynticks system (tickless) - NO_HZ_FULL), sets NO_HZ_COMMON to Y which means it will NOT work.

Indeed, you will not even see the following if it is set:

From the Kconfig:
PPS kernel consumer support (NTP_PPS)
CONFIG_NTP_PPS:
This option adds support for direct in-kernel time
synchronization using an external PPS signal.
It doesn't work on tickless systems at the moment.
Symbol: NTP_PPS [=y]
Type : bool
Defined at drivers/pps/Kconfig:32
Prompt: PPS kernel consumer support
Depends on: PPS [=y] && !NO_HZ_COMMON [=n]
Location:
-> Device Drivers
-> PPS support (PPS [=y])

gustavobsch · April 19, 2023, 4:44pm

Looks better now. Although it seems like ntpsec is not seeing the pps yet

root@bpi-r3:~#  zcat /proc/config.gz | grep PPS
CONFIG_PPS=y
# CONFIG_PPS_DEBUG is not set
CONFIG_NTP_PPS=y
# PPS clients support
# CONFIG_PPS_CLIENT_KTIMER is not set
# CONFIG_PPS_CLIENT_LDISC is not set
CONFIG_PPS_CLIENT_GPIO=y
# PPS generators support
root@bpi-r3:~#  zcat /proc/config.gz | grep HZ
CONFIG_HZ_PERIODIC=y
# CONFIG_NO_HZ_IDLE is not set
# CONFIG_NO_HZ_FULL is not set
# CONFIG_NO_HZ is not set
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100



root@bpi-r3:~# dmesg | grep pps
[    0.040431] pps_core: LinuxPPS API ver. 1 registered
[    0.040438] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    1.837968] pps pps0: new PPS source pps.-1
[    1.842234] pps pps0: Registered IRQ 80 as PPS source

refclock pps unit 0 minpoll 3 maxpoll 3 flag3 1 prefer server 127.127.28.0 minpoll 4 maxpoll 4

root@bpi-r3:~# tail -f /var/log/ntp.log | grep PROTO
2022-08-07T09:31:42 ntpd[3951]: PROTO: 0.0.0.0 c012 02 freq_set kernel 500.000000 PPM
2022-08-07T09:31:42 ntpd[3951]: PROTO: 0.0.0.0 c016 06 restart
2022-08-07T09:31:45 ntpd[3951]: PROTO: SHM(0) 801b 8b clock_event clk_no_reply

gustavobsch · April 19, 2023, 4:46pm

Actually, now I see this entry in the ntp.log after fixing the /dev/pps0 permissions, so it might start working.

2023-04-19T12:29:06 ntpd[4024]: PROTO: PPS(0) 8014 84 reachable

Also seeing this in dmesg

[ 604.634856] pps pps0: bound kernel consumer: edge=0x1

bib · April 19, 2023, 5:19pm

Yup, that’s what you want to see.

If you run ntpq -4 -u -c peer <SERVER>, you should see something similar to:

 remote           refid      st t when poll reach   delay   offset   jitter
===============================================================================
oPPS(0)          .PPS.            0 l    3    8  377      0ns   -343ns    676ns
+SHM(0)          .SHM.            0 l    -   16  377      0ns    -44ns    350ns

You want to see that o against PPS.

You may have to set $PYTHONPATH appropriately.

gustavobsch · April 19, 2023, 6:09pm

Immediately after the ’ bound kernel consumer’ event I see this message on the console, this is repeated non-stop. I tried rebooting just in case without any luck

bpi-r3 login: [  176.115399] pps pps0: bound kernel consumer: edge=0x1
[  177.535097] hardpps: PPSJITTER: bad pulse
[  178.534036] hardpps: PPSJITTER: bad pulse
[  179.532968] hardpps: PPSJITTER: bad pulse

The ntp log never shows ‘kern PPS enabled status’ and the offset/jitter increases over time

 remote                                  refid      st t when poll reach   delay   offset   jitter
======================================================================================================
 16.14.168.192.wifi                     .DNS.           16 u    -   64    0      0ns      0ns    477ns
xPPS(0)                                 .PPS.            0 l    4    8  377      0ns 394.19ms 20.674ms
xSHM(0)                                 .SHM.            0 l    3   16  377      0ns 149.68ms 65.476ms

‘PPSJITTER: bad pulse’ seems to indicate just that, a bad pulse. Even though I can see it with ppstest, so it’s there, and shows a high offset as well.

root@bpi-r3:~# ppstest /dev/pps0 
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
source 0 - assert 1681926577.526630572, sequence: 779 - clear  0.000000000, sequence: 0
source 0 - assert 1681926578.526065270, sequence: 780 - clear  0.000000000, sequence: 0
source 0 - assert 1681926579.525500430, sequence: 781 - clear  0.000000000, sequence: 0
source 0 - assert 1681926580.524934436, sequence: 782 - clear  0.000000000, sequence: 0
source 0 - assert 1681926581.524368134, sequence: 783 - clear  0.000000000, sequence: 0
source 0 - assert 1681926582.523802910, sequence: 784 - clear  0.000000000, sequence: 0

bib · April 19, 2023, 6:29pm

Ugh, does not look good. Assuming systemd, does the unit file have:

ConditionCapability=CAP_SYS_TIME

listed and it must run as root (I think)?

You may also want to look at the output of journalctl -u <unit> and see what it produces. Mine is:

Apr 10 10:46:47 feeder systemd[1]: Starting Network Time Service...
Apr 10 10:46:47 feeder ntpd[1152]: INIT: ntpd ntpsec-1.2.1+110-ga49d53b7f-dirty: Starting
Apr 10 10:46:47 feeder ntpd[1152]: INIT: Command line: /opt/ntpsec-20220626/sbin/ntpd -g -N -u ntp:ntp -c /opt/ntpsec-conf/etc/ntp.conf
Apr 10 10:46:47 feeder ntpd[1152]: 2023-04-10T10:46:47 ntpd[1152]: INIT: ntpd ntpsec-1.2.1+110-ga49d53b7f-dirty: Starting
Apr 10 10:46:47 feeder ntpd[1152]: 2023-04-10T10:46:47 ntpd[1152]: INIT: Command line: /opt/ntpsec-20220626/sbin/ntpd -g -N -u ntp:ntp -c /opt/ntpsec-conf/etc/ntp.conf
Apr 10 10:46:47 feeder ntpd[1160]: INIT: precision = 0.370 usec (-21)
Apr 10 10:46:47 feeder ntpd[1160]: INIT: successfully locked into RAM
Apr 10 10:46:47 feeder ntpd[1160]: CONFIG: readconfig: parsing file: /opt/ntpsec-conf/etc/ntp.conf
Apr 10 10:46:47 feeder ntpd[1160]: CONFIG: requestkey is a no-op because ntpdc has been removed.
Apr 10 10:46:47 feeder ntpd[1160]: LOG: switching logging to file /var/log/ntpstats/ntp.log
Apr 10 10:46:47 feeder systemd[1]: Started Network Time Service.

gustavobsch · April 19, 2023, 7:02pm

Yes, it’s systemd and has ConditionCapability=CAP_SYS_TIME

It was running as ntpsec:ntpsec I switched to root:root but that didnt make a different, same “bad pulse” message and offset increasing.

Do you have a BPI-R3? what OS ?

bib · April 19, 2023, 7:06pm

I’m running opensuse on a Pi4. I’m waiting on kernel 6.3 and opensuse updating the tumbleweed release with it before trying it on my BPi-R3, so it will be a few weeks as I learn how to build it.

bib · April 19, 2023, 9:12pm

I wonder if the offset is too great for the PPS signal. At 0.5s, the system may not know whether to go up or down to the nearest seconds, so…

First step, comment out the server 127.127.28.0 minpoll 4 maxpoll 4 line in ntp.conf and replace it with a server somewhere on the internet and see it can sync with that.

If that works, find the fastest stable serial sync speed from the gps device. And from that speed, calculate the line delay for ~80 characters and use that as an offset.

Next, configure the gps to only send out the minimum required sentences for ntp/gpsd to function.

There are some dire gps units out there. The one I had from Adafruit, the “ultimate”, was poor, so please tell us which device it actually is.

gustavobsch · April 20, 2023, 12:14am

What you said about 0.5s and the system not knowing to go up or down go me thinking…

Comparing the output of ntpq -pcrv on the RPI, frequency=-2.153992 whereas BPI shows a fixed value of frequency=500.0. This is described as “frequency offset (PPM) relative to hardware clock”, I wonder if a fixed value is expected

#  rpi: ntpq -pcrv
     remote                                   refid      st t when poll reach   delay   offset   jitter
=======================================================================================================
oPPS(0)                                  .PPS.            0 l   13   16  377   0.0000  -0.0055   0.0006
*SHM(0)                                  .GPS.            0 l   12   16  377   0.0000   3.8226   3.5603
status=0115 leap_none, sync_pps, 1 event, clock_sync,
leap=00, stratum=1, precision=-19, rootdelay=0.0, rootdisp=1.195, refid=PPS, reftime=e7eafce9.84e618c4 2023-04-19T23:51:05.519Z, tc=4, peer=17768, offset=-0.005473, **frequency=-2.153992,** sys_jitter=0.000619,
clk_jitter=0.000742, clock=e7eafcf7.3766f7f7 2023-04-19T23:51:19.216Z, processor="aarch64", system="Linux/5.15.84-v8+", version="ntpd ntpsec-1.2.0 2021-06-17T05:15:04Z", clk_wander=0.001296, tai=37,
leapsec="2017-01-01", expire="2023-06-28T", mintc=0

# bpi: ntpq -pcrv
     remote                                   refid      st t when poll reach   delay   offset   jitter
=======================================================================================================
oPPS(0)                                  .PPS.            0 l    -    8  377   0.0000  20.1187   0.0053
xSHM(0)                                  .SHM.            0 l    9   16  377   0.0000 -251.056  10.1729
status=0138 leap_none, sync_pps, 3 events, no_sys_peer,
leap=00, stratum=1, precision=-21, rootdelay=0.0, rootdisp=20.252, refid=PPS, reftime=e7eafde4.b4d050fa 2023-04-19T23:55:16.706Z, tc=3, peer=17768,
offset=20.118738, frequency=500.0, sys_jitter=0.005303, clk_jitter=0.00157, clock=e7eafde5.b4bdf9fa 2023-04-19T23:55:17.706Z, processor="aarch64",
system="Linux/6.3.0-rc6-bpi-r3-pwm", version="ntpd ntpsec-1.2.0 2021-06-17T05:15:04Z", clk_wander=0.0, tai=37, leapsec="2017-01-01", expire="2023-12-28T",
mintc=0

bib · April 20, 2023, 11:00am

Well, that looks better. I assume SHM is not being used due to the offset being ~250ms. If you give the -u switch to ntpq (ntpsec version only), as in ntpq -u -pcrv, it give you the units which you really want it to be in nanoseconds. You’ll also see that frequency is in units ofppm`.

So, as per my previous reply, configure an external ntp time source in conjunction with PPS, just to ensure that it does work given a sane time signal.

Once that is done, start by increasing the serial baud rate, 9600 is too slow, and modifying the offset times.

You’ll also need to reduce the sentence type the gps emits to the bare minimum.

Still like to know what device you are using.

gustavobsch · April 20, 2023, 11:20am

I tried with a ublox neo6m and now I’m using ATGM336H attached to uart2 and GPIO 65

Here’s the output with -u. It seems like the offset decreased but it’s really floating.

# ntpq -u -pcrv
     remote                                  refid      st t when poll reach   delay   offset   jitter
======================================================================================================
+16.14.168.192.wifi                     .GPS.            1 u   56   64  377 2.3518ms 62.608ms 32.305ms
oPPS(0)                                 .PPS.            0 l    6   16  377      0ns 72.997ms 8.6598ms
*SHM(0)                                 .GPS.            0 l    8   16  377      0ns -30.32ms 14.851ms
status=0118 leap_none, sync_pps, 1 event, no_sys_peer,
leap=00, stratum=1, precision=-21, rootdelay=0us, rootdisp=94.499ms, refid=PPS,
reftime=e7eb98f9.bcaa7f90 2023-04-20T10:56:57.736Z, tc=4, peer=17768, offset=83.056912ms,
frequency=500.000000ppm, sys_jitter=0ns, clk_jitter=19.942439ms,
clock=e7eb9910.a042667c 2023-04-20T10:57:20.626Z, processor="aarch64",
system="Linux/6.3.0-rc6-bpi-r3-pwm", version="ntpd ntpsec-1.2.0 2021-06-17T05:15:04Z",
clk_wander=2.291196ppm, tai=37s, leapsec="2017-01-01", expire="2023-12-28T", mintc=0

bib · April 20, 2023, 12:11pm

I would say user side is now setup correctly, but kernel side is not. The o & + should mean they are working together.

From ntpq, you do not want to see no_sys_peer, but you do want kern, and I assume you are missing either of:

Apr 10 10:46:13 feeder kernel: pps pps0: new PPS source [email protected]
Apr 10 10:46:13 feeder kernel: pps pps0: Registered IRQ 67 as PPS source

So, it would seem you are not loading the pps-gpio dts.

You need someway to load:

dtoverlay=pps-gpio,gpiopin=18

Where, as you may expect, 18 is the gpio pin number the pps is connected to. And you should also have the pps-gpio kernel module loaded.