[Solved] TC3.8_4 Real Time Upgrade

[MTCAT]

Hi Rich,

No, there was no "." in front of config, the path to the config file that mt_logger was looking for was/is  "/home/pete/config/mt_logger.conf", weird thing is, in my "user manual" for running my existing systems, I have written down that mt_logger.conf should be in home/config, so a different path.... There's some filter settings and other things that can be changed in mt_logger.conf

I also found written down that there is a script called "start_logger.sh" in "~/bin", this may be what gets run when I right click on the desktop icon (in Slackware) and select "Execute" ? This script is less intended to poke around in I think, but there is a switch inside it to turn file compression on and off, but it's the mt_logger.conf file that's more commonly played around in.

I need to poke around my existing fully assembled receiver and see what I can find there...I need to find out how to point mt_logger to a different place, to load mt_logger.conf, among other things, like using the "start_logger.sh" script too.

I'm sure I will continue to need schooling in the world of Linux for quite some time  , probably forever, lol.

Thanks,

David

[MTCAT]

Hi Rich,

Sorry, forgot to add, no, unfortunately there is no help facility in the mt_logger or mt_adc "home-brew" programs.

Thanks,

David

[Rich]

Hi MTCAT

... weird thing is, in my "user manual" for running my existing systems, I have written down that mt_logger.conf should be in home/config, ...

As far as I know, the home directory should only contain user subdirectories:
https://www.linuxnix.com/linux-directory-structure-home-root-folders/

... I also found written down that there is a script called "start_logger.sh" in "~/bin", ...

Tinycore does not implement  ~/bin.  If you want something user specific that is included in the path, place it in  ~/.local/bin.

Code: [Select]

... /home/pete/config/mt_logger.conf ... Was pete the author of the software?

[MTCAT]

Hi Rich,

Thanks for the help and advice, yes, Peter was the fellow who wrote the acquisition software and set-up the digital portion (hardware and software) of my two original receivers.

Hopefully I can poke around on one of my receivers in the next few days, I still can't find any reference to "/home/pete/config/mt_logger.conf" in any of the C programs yet !
Thanks,

David

[Rich]

Hi MTCAT

... I still can't find any reference to "/home/pete/config/mt_logger.conf" ...

It could be in a header file, a makefile, a configure script, etc. Try this:

Code: [Select]

sudo grep -ri pete ~/*This will search every file in your home directory for the text  pete  regardless of case. This also presumes the source
code for the acquisition software is somewhere in your home directory.

[MTCAT]

Hi Rich,

Thanks for the help, after looking through all the C (C++ ?) programs in the two directories I was able to find a few "#define xxxx" lines, like, "#define CFGFILE /home/pete/config/mt_adc.conf", so that's great, I just need to change those to what I want, and put the config files in those spots, and that should be okay.

I also need to change ttyUSBO to ttyS0 inside "gps.c", so that's an easy one as well.

I do need to take care that my data directory (where data is stored) right now is "/home/data", that would currently be written to the pen drive ! (slow), I want it to rather go to the CF-card, I'm wondering if "/mnt/sda5/data" or something like that would work, or, perhaps in the future, maybe I should just install TC completely onto the CF card....

Just a couple of other things though, Peter defines a shared memory segment of 256 MBytes for the logger program, I'm not sure yet if he has this in a script file or something but in the comments of a C (C++) program called "shared_mem.c" he says that we can do this by typing "echo 268435456 > /proc/sys/kernel/shmmax".

I guess I could run that (or similar) with a script file called out of bootlocal.sh ?, or perhaps I could just edit sysctl.conf and make that change in there permanently ?, assuming that file exists in Tinycore ?

Then the only other thing relates to the good old PPS, I'm planning on using a TTL level PPS (0 to 3.3 V) whereas the current setup uses an RS232 level PPS which has been divided down from  it's original ~ +6 to - 5 V range to a roughly symmetrical +/- 1.3 V (kind of messily done with two voltage dividers).

Inside a program called pps_stats.c, Peter has a bunch of "cumulative distribution functions" (in lookup tables) that he's using to predict where the zero crossing (defined as the midpoint of the low-high transition) of the PPS signal is (I think anyway), but I'm not really sure, C (C++ ?) code always looks odd to me, so, I'm  not sure, if I redefine the PPS waveform do I need to collect a bunch of these "new" PPS waveforms, make a probability amplitude distribution and then integrate it to get a new cdf ?? I don't know if that's what Peter really did, I don't recall him ever talking about that, only that we need to know the +/- levels of the PPS signal.

I'm puzzled on this one, and have always been a bit puzzled as to what Peter was/is doing with the PPS signal, I even recall him saying something about adding extra samples if need be (from where ??-talk about distorting your measured data - -making up new samples ??) to account for a slightly off sample rate. Peter uses the PPS to measure the actual sample rate of the 24DSI, maybe it's 199999.79 Hz in reality when asking for 200 kHz for example....

A colleague of mine at work thinks Peter may be compensating for the effect of the FIR filter delay (in the delta-sigma ADC) on the PPS, but I don't know about that, why do it for the PPS channel and not for the other channels, and also, if all channels are delayed the same (same FIR filter applied to all channels), then there's nothing lost in terms of relative timing on the data, so I'm not sure about that one.

My confusion around the PPS continues. I was wondering if I could simply use a 2 nano-second sampling oscilliscope (8 bit voltage res though) to "hyper-fine" define the PPS signal that way, and then simply interpolate PPS zero-cross location based on the "sparse" 200 kHz sampled PPS waveform, I think that's what Peter is doing with the cdf, but I don't really understand how he's doing it, so I'm not sure.. I guess I don't want to re-write Peter's program either though, as I'm pretty much clueless with C (C++).

I could send you the pps_stats.c code for you t look at, if you have the time, and see what you think, it's only about 20 lines of actual code, the rest is tables. Peter says he's "compensating for lag between the A/D and the PPS", so that does kind of sound like what my co-worker suggested.

But I do recall Peter saying some odd things about ways of finding the zero-cross of the PPS with sparsely sampled (200 kHz) data too, so, going in circles on this one.

Thanks,

David

[Rich]

Hi MTCAT

... I was able to find a few "#define xxxx" lines, like, "#define CFGFILE /home/pete/config/mt_adc.conf", so that's great, I just need to change those to what I want, and put the config files in those spots, and that should be okay. ...

A better solution would be to not hard code a particular user, but have the program default to the home directory of
the user currently running the program:

Code: [Select]

char cfgfile[256];
snprintf(cfgfile, 256, "%s/.config/mt_adc.conf", getenv("HOME"));If user  pete  runs the program, charcter array  cfgfile  gets set to  /home/pete/.config/mt_adc.conf.
If user  tc  runs the program, charcter array  cfgfile  gets set to  /home/tc/.config/mt_adc.conf.

That way it gets set at run time, not compile time.

... I want it to rather go to the CF-card, I'm wondering if "/mnt/sda5/data" or something like that would work ...

You could move  /home/data  and replace it with a link to  /mnt/sda5/data:

Code: [Select]

mv /home/data /home/dataold

Code: [Select]

mkdir /mnt/sda5/data

Code: [Select]

ln -s /mnt/sda5/data /home/dataMake sure  sda5  is mounted at the time. Be aware, devices such as  sda  can move depending on what devices are
connected when you boot. Identifying devices by UUID is safer.

... but in the comments of a C (C++) program called "shared_mem.c" he says that we can do this by typing "echo 268435456 > /proc/sys/kernel/shmmax". ...

Without seeing what else is around the comments (and actual comments) I have no context and can't respond to that.

... I could send you the pps_stats.c code for you t look at, if you have the time, and see what you think, it's only about 20 lines of actual code, the rest is tables. ...

Sure, I could take a peek at it.

[MTCAT]

Hi Rich,

Thanks a lot, I like the idea of having the config file location being set "on the fly", that's an awesome idea I would say, thanks for that code, thanks as well for the idea to make a symbolic link, is the "mv /home/data /home/dataold" done just to make a backup copy though ?

I attached three programs here; "shared_mem.c" so you can see the shared memory segment stuff, and then "process_adc.c" and "pps_stats.c", inside "process_adc.c" around line 303 is where some PPS related stuff begins to happen, and then inside "pps_stats.c" is where all the "cumulative distribution functions" (CDF) are tabled and a bit of code is located as well.

Right now, we're using the U-Blox 200 kHz sample rate with 40 kHz filter table, Peter has a lot of extra tables in there that could be cleaned out, for example, I had an "ORCA GS101" GPS device in the past that we were using initially, it's quite nice but also quite expensive (~ 1500 bucks I think), so we switched over to the U-Blox's (~ 300 bucks) which work very well. The ORCA was really used more for my old Windows XP laptops that were used with my version 1 receiver.

But, if I redefine the PPS signal to be a 0 to 3.3 V TTL level (which would be similar to the ORCA actually --->maybe I could even just use the ORCA CDF tables...) then I'm wondering if I need to redo the CDF table for a U-Blox 200 kHz, 40 kHz filter, TTL level PPS.

Also, in "process_adc.c", there's two functions "pps_vmin()" (line 394) and "pps_vmax()" (line 402) that would need to be tweaked for the new signal amplitude of the U-Blox TTL output, which is about 0 V low and 3.3 V high, although I need to measure that to find out what the levels are more accurately.

Thanks,

David

[MTCAT]

Hi Rich,

Re: the symbolic link question, thinking about it, I suppose we can't make a symbolic link with /home/data pointing to /mnt/sda5/data if /home/data also exists on the pen drive, makes sense that the OS would be confused to say the least.....hopefully I answered my own question, at least approximately correctly !

Thanks,

David 

[Rich]

Hi MTCAT

...  is the "mv /home/data /home/dataold" done just to make a backup copy though ? ...

I don't know whether  /home/data  is a file or a directory ( I presume directory). Either way, I didn't want to give
instructions that might destroy it. So  /home/data  gets renamed to  /home/dataold  to serve as a backup.

[Rich]

Hi MTCAT

... I suppose we can't make a symbolic link with /home/data pointing to /mnt/sda5/data if /home/data also exists on the pen drive ...

I don't recommend it. However, you can alter the link later on if you need to, for example:

Code: [Select]

unlink /home/data
link -s /home/dataold /home/data

[Rich]

Hi MTCAT

...

... but in the comments of a C (C++) program called "shared_mem.c" he says that we can do this by typing "echo 268435456 > /proc/sys/kernel/shmmax". ...

Without seeing what else is around the comments (and actual comments) I have no context and can't respond to that. ...

That is absolutely not what he said. The kernel can limit the amount of shared memory that can be allocated. If
shared_mem.c  tries to allocate more shared memory than the kernel will allow, you may have to raise the limit using
that command. You can check what the limit is with this command:

Code: [Select]

tc@E310:~$ cat /proc/sys/kernel/shmmax
4278190079
tc@E310:~$

... Also, in "process_adc.c", there's two functions "pps_vmin()" (line 394) and "pps_vmax()" (line 402) that would need to be tweaked for the new signal amplitude of the U-Blox TTL output, which is about 0 V low and 3.3 V high, although I need to measure that to find out what the levels are more accurately.

Don't waste your time. Those levels will change with temperature and supply voltage variations. Uncomment the first
double and comment the second double in both functions like this:

Code: [Select]

static double pps_vmin()
{
    /* the pps signal swings between .136 and 3.460 volts
     */
     static double v_min = 0.137 ; // orca
//     static double v_min = -1.310 ; // ublox with resister divider
     return( v_min ) ;
}

... But, if I redefine the PPS signal to be a 0 to 3.3 V TTL level (which would be similar to the ORCA actually --->maybe I could even just use the ORCA CDF tables...) then I'm wondering if I need to redo the CDF table for a U-Blox 200 kHz, 40 kHz filter, TTL level PPS. ...

Do you know how those tables were created?

[MTCAT]

Hi Rich,

Thanks for the help, I'll check to see what my shmmax is set to and go from there.

It could be a "fools hope" as you say, but I think it's fairly important that we know the mid-point of the high-low PPS signal, that's what Peter defines as the start of a second (I think anyway). But maybe using the ORCA TTL levels would be good enough as you suggest, it should be quite close to the U-Blox TTL level output (0 to 3.3 V).

No, unfortunately I don't know how Peter generated or obtained these CDF tables, from what little I know of statistics I think we could make one by collecting a nice long chunk of PPS data (ten minutes maybe ??), and then make a finely binned histogram plot, divide each bin in the histogram plot by the total number of samples (to obtain a probability amplitude distribution), and then lastly integrate that to get the CDF. But I have no recollection of Peter ever discussing this with me, so I'm not sure if this is what he actually did.....

I suppose I could try this process on the existing fully assembled receiver (U-Blox RS232 PPS that has been knocked down with a voltage divider) and see if I can ~ reproduce the CDF table for that case...Or just use the ORCA settings....

Can you hazard a guess at what the "lag" value is that Peter is returning out of pps_stats.c ? I'm guessing it's the interpolated location of the PPS "zero-crossing" (midpoint of the low to high transition), but I'm not totally sure, the CDF's are the probability that a function takes on a value less than or equal to some value, so I'm not sure how that works better to obtain zero crossings, or if Peter is just being overcomplicated ?

Thanks,

David

[Rich]

Hi MTCAT

... It could be a "fools hope" as you say, but I think it's fairly important that we know the mid-point of the high-low PPS signal, that's what Peter defines as the start of a second (I think anyway). ...

It sounds like he's taking measurements to detect when the signal makes a transition. It's not important that you sync
to the exact center of the transition, just that you always sync to the same level during the transition.

... I suppose I could try this process on the existing fully assembled receiver (U-Blox RS232 PPS that has been knocked down with a voltage divider) and see if I can ~ reproduce the CDF table for that case...Or just use the ORCA settings....

If you have TTL levels, go with the ORCA settings.

... Can you hazard a guess at what the "lag" value is that Peter is returning out of pps_stats.c ? ...

He returns a decimal value between 0 and 100.

This shows up at the end of  process_adc.c:

Code: [Select]

double process_precise_sample_rate()
{
    /* measure the rate by looking at last two pps crossings and lags
     *
     * the nominal sample time between the crossings (1 second) is adjusted
     * by the difference in lags of the two scans.
     *
     * lag is defined at a percent (0:100) of the sample interval (dt)
     */

[MTCAT]

Hi Rich,

I think you're right, using the ORCA defined table (for TTL) is the way to go, I may have figured out how Peter generated the tables but I'm not 100 percent sure, so...., probably just easiest/best to move forward with the ORCA table. Hopefully on Friday I can make some headway on getting the config files sorted out, and changing the code to use the ORCA table, re-compiling and try starting the acquisition program again, see what happens...

Down the road, I wonder if we could improve the zero-cross location by doing a super-fast sampling of the PPS waveform (1 GHz sample rate with a storage oscilloscope), even though it's only 8 bit voltage resolution, although I think there's a high resolution mode as well which gives 12 bits resolution, perhaps if I averaged enough cycles, I could get a good definition of the PPS waveform ? With that, we would have anywhere from 5,000 (200 kHz) to 10,000 (100 kHz) extra points between each 24DSI sampled point on the PPS waveform, I think we could find quite accurately where the zero-cross lies between two 24DSI sample points that bracket the zero-cross, this might be better, and a lot simpler. But I would have to collect a bunch of PPS waveforms and then average them, and of course modify the nasty (in my opinion) C++ code.

Besides the 1 GHz oscilloscope, I also have a 16 bit, 1 MHz, A-D (PDAQ 3005), and a 12 bit, 1 MHz A-D (Wavebook 512), but that would only be 5 (or 10) extra points between 24DSI samples.

Can save that for later though maybe..

Thanks,

David