Time Synchronization

[MTCAT]

Hi everyone,

With and incredible amount of help (and patience) from Rich, the driver we built for the 24DSI is working 100 percent.

Next step is taking care of time synchronization with a U-Blox EVK-M8T pumping out NMEA sentences for ntp, but also with PPS signal detected and made use of by gpsd (I think), to further "discipline" ntp. The way I understand it is, the NMEA sentences gets ntp close (ms), and then the PPS really ratchets down the time-sync to us level.

Looking at some old posts, in order to see my GPS device (U-Blox EVK-M8T), I found I needed to install usb-serial-2.6.33.3-tinycore.tcz, after install, I saw a device "ttyACM0"  (was expecting "ttyUSB0").

But, that worked, typing cat/dev/ttyACM0 pumps out all the NMEA sentences from the U-Blox, so neat.

Then I installed gpsd.tcz (ver 2.96) and also ntp (but apparently chrony will work too, and is easier to set-up than ntp ?).

Finally my questions;

1) The version of gpsd I downloaded from TC repository is 2.96, this website recommends using versions newer than 3.22, see "Introduction", point 2

https://gpsd.gitlab.io/gpsd/gpsd-time-service-howto.html

Would it be possible to get a newer version of gpsd for my TC 3.8_4 ?, although I don't know yet if my current 2.96 gpsd really is a problem, I did type "gpsd -D 5 -N -n /dev/ttyACM0" and it was working, output a bunch of stuff to screen....

2) The same link above indicates that "kernel" pps (KPPS) is about 5 times more accurate (1 us vs 5 us)  than "regular" PPS (see "PPS Quality Issues"), does Tiny-core have any PPS "kernel" support, or, a "pps-tools" package ? I didn't see one in the repository.

3) I have no experience really with ntp, or chrony, I would be interested to hear what you all think would be easiest to setup, and if you think which one (chrony or ntp) is "better", accurate, stable.

Thanks,

David

[MTCAT]

Hi everyone,

Some small progress, from what I've been able to read up on, I have two options;

1) Kernel PPS, more accurate, but I think I would need to "patch" my kernel, right now "CONFIG_PPS" is not set in 2.6.33.3, I think I would need to turn that on with "make menuconfig" (?) and then patch the kernel ?, and then attach the PPS signal to ttyACM0 with "ldattach" command (I think we have this in TC3_8_4), but on  top of that, I would also need to install Linuxpps (pps-tools), make another extension I guess.

https://github.com/redlab-i/pps-tools

2) Using gpsd (less accurate) to feed the PPS to ntpd, I installed both gpsd and ntpd already but I haven't got it working 100 percent yet, I'm not sure if gpsd is even picking up the PPS signal on the serial port at all (ttyS0), gpsd also produces some errors on startup related to IPv4 and IPv6 sockets or something. This assumes that the ver 2.96 gpsd is okay, if having to "upgrade" to versions newer than 3.22, I'll have to build a new extension anyway ? Or, maybe a newer gpsd package exists for newer version of TC that might also work in TC3_8.4 ?

I would prefer (1) but I'm not sure if anyone has tried this already and may have words of wisdom regarding it ? Or, if reverting to (2), how I can get gpsd to pick up the PPS signal on the serial port, and/or just simply check for proper operation of gpsd.

Thanks,

David

[Rich]

Hi MTCAT
I haven't dealt with  gpsd  or  ntpd  so I have nothing to offer there.

I do have some questions though.
Why do you think you need that level of accuracy?
What happens if you're off by 1uS? 1mS? 1Sec?

The kernel is configured to task switch at about 300 Hz or every 3.33 milliseconds. Since it's non-preemptive, that's about
the fastest response time you could expect if nothing else is running.

If you want 1 uS accuracy, that's better suited for dedicated hardware than a kernel. And it sounds like your hardware is
capable sampling at precise intervals and feeding that data to a buffer you can access as required.

[MTCAT]

Hi Rich,

I'm hoping to have multiple independent recorders out at the same time, so, each logger needs ntp (or chrony) to adjust the system time on each logger at least well enough that a transient on one logger will be seen to occur at a similar time as on another logger (within several ms say ?). But then, we're also using the 24DSI12 to collect the PPS signal on each logger to allow an even finer time offset analysis.

But, I don't need absolute timing accuracy (I don't think), just good relative accuracy/synchronization between all the loggers themselves.

The problem is with the high frequency data, the measurement bandwidth is up to 40 kHz, so a time sync error of even 500 usec equals a phase error of > 20 degrees at 40 kHz (if I recall correctly), which is large for "good" data. I think even with 500 usec, we might have a respectable quality phase offset of < 5 degrees at 10 kHz.

So I think it's all about trying to get the high frequency phase between two events, recorded on two separate loggers, as accurate as possible, and to as high a frequency of possible.

Thanks,

David

[MTCAT]

Hi Rich,

By the way, I was able to get chrony to do something, kind a sort of, it synchronized to the "pool" servers, but gpsd is not detecting my PPS signal, in fact, gpsd seems to be looking for the PPS on ttyACM0 (the USB port).

So I'm not sure how to fix that, I've also been playing around with chrony config files, using both "SOCK" and "SHM" definitions for the GPS NMEA string (ttyACM0) and the PPS (ttyS0)...

I hope I don't have to re-compile the entire kernel to enable "CONFIG_PPS" and "CONFIG_PPS_CLIENT_LDISC", right now, in 2.6.33.3 these are not enabled !

Thanks,

David

[MTCAT]

Hi Rich,

I've been having problems posting since Friday, been dying to talk about all this, maybe I'm babbling on too long on each one....

I was glad to see that gpsd was at least trying to look for the PPS, but this is the error I get,

gpsd: PROG PPS Create Thread gpsd_ppsmonitor
gpsd: PROG PPS can not connect chrony socket: /tmp/chrony.ttyACM0.sock

To me, this looks like gpsd is trying to get the PPS off the USB port (ttyACM0), when the PPS signal is actually on the serial port (ttyS0) ?

David

[MTCAT]

Hi Rich,

I'm using a U-Blox EVK-M8T, it has both USB and serial port outputs. Right now, the old power-hungry setup used the USB port for the NMEA strings, and the serial port for the PPS, for lack of knowing any better, this is how I've been trying to do it on the VortexDX3 as well.

There's "U-Control" software that comes with the U-Blox units as well, with that I think I could configure the PPS **and** NMEA strings to come out on the serial port, maybe that would work with everything I have installed "as-is", as long as gpsd I got from TCZ repository is happy with reading both the PPS and NMEA strings from ttyS0. According to gpsd document, this would be like "software" PPS, or good to something like 5 usec. I think that's what this fellow is doing;

https://www.lammertbies.nl/comm/info/gps-time

To get the "holy-grail" of kernel level PPS, around 1 usec, I think I would need to re-build the kernel with CONFIG_PPS enabled (could also enable GPIO support, can input PPS that way too), but I would also need to build and install "pps-tools" and add "ldattach" capability (to attach pps to serial port), so would be a lot of extra work...

But the serial port PPS and NMEA would be a lot better than trying to do it all over USB, apparently USB adds a fair amount of extra jitter along with a 100 usec plus offset.

https://blog.dan.drown.org/pps-over-usb/

Maybe I will try the PPS and NMEA strings on the serial port first, if that works with my TC3.8.4 "as-is", that would probably be the way to go... I think chrony was close to doing it actually....

By the way, I did get the "kernel" level PPS going in Lubuntu, kind of, I installed pps-tools and was able to see both NMEA strings on ttyACM0, and the PPS on ttyS0 (or actually /dev/pps1), but ntp was being difficult for me, ignoring my ntp.conf in /etc, I guess ntp looks for dhcp clients first and ignores ntp.conf in that case, that was about 4 hours of Googling to find that out, but, I almost got the kernel level PPS going in Lubuntu....

Thanks for letting me talk to you about all this,

David

[Rich]

Hi MTCAT
Look at page 6 of this manual:
http://www.generalstandards.com/specs/pc104p24dsi12_spec_011111.pdf

Does your board have this connector?
Pins 29,30 and 33,34 of row B are listed as clock and sync outputs respectively.
Pins 29,30 and 33,34 of row A are listed as clock and sync inputs respectively.
Isn't their purpose for syncing multiple data acquisition cards by daisy chaining the outputs from one card to the
inputs of the next?

[MTCAT]

Hi Rich,

That would be good to make use of that except that even the "roving" recorders are typically quite far apart, maybe 100 m at best, and more typically 200 m to 300 m, maybe even 1000 m apart sometimes. The fixed (for a given survey) base station can be even further. maybe something like 5 to 10 km away. Having "hard-wired" connection is pretty tough/awkward in the field, even for the closer roving stations, a lot of cabling would be required (heavy).

I think the 24DSI12 does have a GPS input though, which I don't really understand what it's used for.....would be nice if we could just sync all the A-D's to PPS directly and just use the NMEA strings to give ntp or chrony milli-second type of accuracy....

Thanks,

David

[Rich]

Hi MTCAT
I didn't realize they were that far apart. I'm pretty sure 100 meters too far to run cabling for the LVDS outputs.

... I think the 24DSI12 does have a GPS input though, which I don't really understand what it's used for.....would be nice if we could just sync all the A-D's to PPS directly and just use the NMEA strings to give ntp or chrony milli-second type of accuracy....

Have you read section 3.12 (page 35) of this:
http://www.generalstandards.com/user-manuals/pc104p_24dsi12_man_091113.pdf

[MTCAT]

Hi Rich,

I guess the main challenge with using the PPS input on the 24DSI12 is that it would require a whole new re-write of the data acquisition program (I think anyway)...

As you well know, it's hard enough (for me anyway) to even just replicate my two existing "high power" receivers......... As well, I'm not sure I can completely visualize how the PPS input on the 24DSI12 could be used other than forcing all loggers collect on a pre-set schedule ?, i.e., all loggers start collecting on the rising edge of the first PPS pulse after the top of every minute or something ? Not all loggers have truly identical sample rates either though, I know my existing program corrects for differences between the desired and actual sample rate, but I'm not completely sure what the fellow did in that regard. The GSC GPS synchronization stuff also discusses correcting the sample rate based on the PPS.

This timing stuff is very annoying, and wouldn't be needed at all if I didn't want to be able to compare events (in terms of phase) across widely separated loggers.

Thanks,

David

[Rich]

Hi MTCAT

... As well, I'm not sure I can completely visualize how the PPS input on the 24DSI12 could be used other than forcing all loggers collect on a pre-set schedule ...

The 1 PPS signal provides a very precise 1 second window that the 24DSI12 uses as a reference for setting
the sample rate. The board compares how many samples occur during the 1 second window to the Target Sample Rate.
It then adjusts the sampling rate up or down until it matches the Target Sample Rate. It essentially phase locks the
sampling rate to the 1 PPS signal.

[MTCAT]

Hi Rich,

I got the U-Blox setup to output NMEA and PPS over the serial port, made up a new RS232 "straight-thru" cable, cut up the USB cable so the USB port only powers the U-Blox, used the SHM definition in chrony.conf, pointed gpsd at /dev/ttyS0, and voila, chrony is happy ! !, using both the NMEA and PPS info on the serial port !

Last night, chrony was able to sync the time on the Vortex DX3 to generally better than 1 us !, usually between 500 and 800 ns !! I have screenshots but I won't send them

So now I just have to figure out how to get this all going on boot-up.

There's also some other options I need to investigate, there's an "offset" parameter in chrony.conf that needs to be adjusted, as well, some neat options like "rtsync" that will let chrony sync the real time clock as well, and drift files, so that chrony will remember the last time the board was turned off, given the drift it "knows", will try to compensate on the next powerup. As well, there's a "makestep" option that will allow chrony to take a couple of big steps if it needs to make a large correction, so that the sync process doesn't take too long, apparently ntpd can be quite slow to sync sometimes.

But, first thing will be to getting chrony and gpsd fired up on boot-up, and then can do the tweaking after, I'm just very relieved it doesn't look like I will need to rebuild the kernel or anything, thank goodness.

Thanks,

David

[Rich]

Hi MTCAT
There's something I'm still not clear on, so let's do a little sanity check.

You can set the system time to within about 1 uS. Fine.

If you read the system time, it should be accurate to within about 1 uS. Also fine.

How are you going to trigger your data acquisition board to that level of accuracy?
Your kernel is not preemptive and each process gets to run for about 3.3 mS. So when it comes time to trigger your
board, your process may be waiting for one or more other processes ahead of it to finish before it's your turn to run.

[bmarkus]

You need a Real Time kernel.