[Solved] TC3.8_4 Real Time Upgrade

[MTCAT]

Hi Rich,

The startup script starts two programs, mt_adc and mt_logger, mt_adc starts first in a terminal window, and then a second command uses tail to show the output of the mt_adc log file in a second (separate) terminal window, and then lastly mt_logger starts in a third terminal window, and creates/opens up the GUI.

But you're right, I have no idea whether the problem is with mt_adc, or mt_logger, I just assumed it was mt_logger since that's where all the GUI stuff is and my first symptom of a problem was a hung GUI.

I'm not sure, but I think mt_adc is basically sampling all the time and then the mt_logger program is responsible for "catching" transient events and displaying them in the GUI, and it's the mt_logger program where you can change sample rate, A-D scale, etc, inside the GUI.

When I run mt_adc in gdb it just runs, I can see it with "ps" as well, I just get a solid green cursor in the terminal window that's running gdb. I'm not totally sure, but I think it's doing what it's supposed too.

Back to my "original" testing method, which is to run the start script with the first two commands only, and then run mt_logger in gdb in another window, I'm seeing the segmentation fault, but now in a different spot Tonight it (mt_logger) seems to get farther into the code before the segmentation fault happens By the way, when I just type "gdb mt_logger" and "run", gdb gets hung up (I think) and I have to hit Crtl-C to get the prompt back, I can backtrace then but not sure that's valid ?

I did my creep through the mt_logger code with "next" and now I see the following,"Program received signal SIGSEGV, Segmentation fault. No function contains program counter for selected frame." And then when I type "bt" I see,
#0 0x00000003 in ?? ()
#1 0x00000005 in ?? ()
#2 0x0xbfb498cc in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack ?)

I don't know why in gdb when I type "layout next" I'm seeing assembly language code now too, sorry, can't figure anything out here...

I did try just running "gdb mt_logger", and then "run" and then "Ctrl-C" when it hangs, and then "bt", this is what I get in that case,"#13 0x080512a7 in main (argc=1, argv=0xbfcc4d64) at main.c:34"

This is different as this is quite near the last line of main.c now ?? ( gtk_main( ) )

Sorry for this novel, thrashing I guess, confused myself even further (if that's possible)...

Thanks,

David

[Rich]

Hi MTCAT

... mt_adc starts first in a terminal window, and then a second command uses tail to show the output of the mt_adc log file in a second (separate) terminal window, ...

OK, when you do that, does the second window show data being produced? Should it show data being produced?

[MTCAT]

Hi Rich,

Thanks for the help, yeah, the second terminal window shows the results of the 24DSI12 startup and intitialization tests and the log file (mt_adc.log) is also "fresh" judging by the time stamp on the file (it saves a new one on every startup), it is odd though, the log file (in /home/tc/logs) shows as an executable file I think, not related to the problem most likely but seems odd ?

I've also noticed another issue (surprise, surprise), if I start mt_adc once, and then try to start again, I have problems with shared memory, I get an error that the shared memory segment (if that's the right word) already exists, I have to do a full re-boot every time it seems.

I'm not sure that's the case with my fully functioning receivers, although I usually do just turn it on, collect data at a station, and then turn it off and get to the next station, so could be the same thing there.

Thanks,

David
 

[Rich]

Hi MTCAT

... shows the results of the 24DSI12 startup and intitialization tests and the log file (mt_adc.log) is also "fresh" judging by the time stamp on the file  ...

But is the 24DSI12 collecting and saving data?

... it is odd though, the log file (in /home/tc/logs) shows as an executable file I think ...

Assuming its not the program doing it, that will happen if a file contacts a file system that does not support permissions,
such as FAT. Running  ls -l /home/tc/logs  will show all the bits set (-rwxrwxrwx).

... I'm not sure that's the case with my fully functioning receivers ...

You might want to check that. It may or may not provide an additional clue as to what's going on.

[MTCAT]

Hi Rich,

No, there's no digitized data being saved to disk right now (I don't think anyway, I should look on the CF card to make sure), I can't tell for sure if the 24DSI12 is collecting anything at all, both of those functions are seen within the GUI (mt_logger), which is frozen of course. I think mt_logger is responsible for grabbing the output of mt_adc, filtering and decimating it to create a "continuous" data stream (@ 20 kHz sample rate), and then triggering and localizing "transient" events at the full 200 kHz sample rate, and saving those transient files too.

I'm not totally sure, but I think mt_adc is working but there's something hanging up in mt_logger, maybe something to do with threads ? Grasping at straws here, Peter is using "-lpthread" link flag in the Makefile for mt_adc, yet the only programs that have an include statement "#include <pthread.h>" are part of the mt_logger build , rc_server.c and another one I can't recall off the top of my head. I found a post yesterday about using "gthread" and changing a link flag to "gthread-2.0" to solve a seg fault in gtk_init, I tried to compile mt_adc with a link option "-lpthread-2.0" but that didn't work, gcc complained about that...

I guess it would be ideal if I knew the overall "flow-chart" of the program, and could step through it to see where the problem is, that's what I was trying to do with gdb but I'm getting kind of confusing results out of gbd, last night I wasn't seeing the seg fault occurring in gtk_init now, so where does the seg fault actually occur ??

Thanks,

David

[Rich]

Hi MTCAT

... I found a post yesterday about using "gthread" and changing a link flag to "gthread-2.0" to solve a seg fault in gtk_init, I tried to compile mt_adc with a link option "-lpthread-2.0" but that didn't work, gcc complained about that...

How did you get from  gthread-2.0  to  -lpthread-2.0?

I suspect there's more to it than just changing a link flag:
I don't know if  #include <pthread.h>  would need to be removed.
You probably need to add  #include <glib.h>  if it's not already there.
You might have to change the syntax of any thread statements.
The link flag would be  -lgthread-2.0

I've never programmed using threads so I don't know any of that for certain.

[MTCAT]

Hi Rich,

Just an act of pure desperation !, I thought there might be a "2.0" option for pthread as well, but it didn't work, and as you say, it would be too easy if it was just a link flag.

I'm a bit overwhelmed with mt_logger especially, the final executable being the result of linking 33 separate object files !!, to me, it would be way easier if it was written sequentially in one big file, at least then I would feel like I better know the structure of the program, the way it is right now, to me anyway, it just seems all over the place, what programs are executed in what order ? I think it would help to debug if I knew the proper path through the code so to speak, I am aware of "profiling" but I think I need a proper functioning executable for that to work ?

Maybe the only choice is to continue with gdb, but I'm not sure why the segfault seemed to occur in a different spot last night.

Thanks,

David

[MTCAT]

Hi Rich,

Home from work and fired up gdb, I'm not seeing segfaults unless I step through the program with "next", when I just do the startup script to open two terminal windows, one running mt_adc and another showing the output of mt_adc.log, and start mt_logger manually in another terminal by typing "gdb mt_logger" and then "run", I get a hung GUI, I hit Ctrl-C in the gdb window, and then "backtrace", the last command mt_logger gets to is line 218 in filter.c, which is just applying a 4'th order Butterworth low-pass filter (used for low-pass filtering before downsampling to make the 20 kHz data stream I think).

Attached filter.c here, in gdb, when I type "print tic" I get 154666080, and for "print i" I get 1, I also can see f_counter (from backtrace) as 0x9380414 which converts to 154666004, but yet when I type "print f_counter" I get returned "<value optimized out>".

The "tic" value is not correct, it should be between 0 and 4, I don't even know how order=4 is getting passed in to filter.c, I could try passing in order=2 (if I could find the right spot in the 33 separate C programs) to see if that solves the problem by avoiding apply_bwf_filter4 and apply the second order filter instead.

I'm not sure if you can see any issues with apply_bwf_filter4 ? It's odd too, in the backtrace, it shows "order=4" as an argument of apply_bwf_filter4 right after "nchan" but if you look at 184 in filter.c, there's no "order" in that list of arguments that I can see ?

Thanks,

David

[Rich]

Hi MTCAT
The compiler optimizations are screwing up the gdb output. Set the optimization flag to  -O0 (that's oh zero).
Recompile and see if the gdb output looks any better.

[Juanito]

You can also substitute "-ggdb" for "-march=i486 -mtune=i686 -Os -pipe" - in addition, you may need to search the Makefile for "-g" and "-O2" and remove them.

[MTCAT]

Hi Rich (and Juanito),

I changed the optimizer setting to "-O0" and was able to get somewhat better information out of gdb. I did also find where the low-pass filter order is set (to make the downsampled continuous data).

I then re-ran the code with both second order and fourth order filters (with the "-O0" optimization).

In both cases, the code still gets stuck in filter.c, in the filter loop, either "apply_bwf_filter4" or "apply_bwf_filter2".

In both cases, I seem to be getting nonsensical values for data input to the filter (and of course for data output from the filter).

In both cases, I'm seeing input data (when converted to decimal) are like 2.8 x 10^9 (integer), max integer value for a 24 bit ADC is ~ 16,777,216 .

It doesn't always seem to bomb on the same iteration, but "i" is usually 0 or 1, in the case of the 4'th order filter routine "tic" was 1, another time it was 2, so at least they are reasonable numbers.

Seems like garbage input right now ?, which would mean of course that I'm not getting proper data from the ADC, which would mean mt_adc is not working properly still.

Not sure how to test that, should be at least seeing bit-noise of the A-D getting through, so, need to test mt_adc somehow I think.

Thanks,

David

[Rich]

Hi MTCAT
I don't know how much difference it will make, but I would also add  -ggdb.  I just checked some of my compile
scripts, and I use a  GDEBUG  variable that when set compiles with  -O0 -ggdb  for running with  gdb.

[MTCAT]

Hi Rich,

I tried the -ggdb compile flag as well and it didn't really add anything (that I could tell anyway). I'm going to try to get a consultant to login to the little computer remotely and see if he can debug the code, I just don't know C at all (although slowly learning with your help) but shudder to think how long it may take to get the code running at the current pace.

I'm attempting to install "Team Viewer" (version 8, the oldest one, and 32 bit version) for the remote login but I'm missing the following libraries; libasound.so.2 and libwine.so.1.

Need to install WINE I guess ?, not sure where libasound.so.2 would be found ?

Thanks,

David

[MTCAT]

Hi Rich,

Found both libraries in the good old repository ! Software seems to work !, wow !, so apparently now this fellow should be able to log in to my little computer and get the C code working !, hopefully anyway.

Thanks,

David

[MTCAT]

Hi Rich,

What would be the best way to make a clone of my existing TinyCore real-time pen drive (sdb1, 16 GBytes). I tried using Clonezilla to do the job but it didn't work for some reason...I'm a little gun-shy to use "dd" but maybe that would be best ? Would the below be okay ? The bigger sdc1 drive (64 GByte) was originally a FAT32 (W95) volume, Clonezilla changed that, but something went wrong as when I plug in the big 64GByte drive in TinyCore now, it shows up in dmesg and fdisk -l okay (as sdc1) but it doesn't show up in the mount tool.

dd if=/dev/sdb1 of=/dev/sdc1

Obviously don't want to mess up the 8 months of progress to date, but thought it might be good to have a spare duplicate pen drive.

Thanks,

David