Re: Tiny Core v17.0 upgrade issues

[Stefann]

So.. still running at 15:00; that means 30hrs runtime without crash

So, to summarize....
TC15:
- including kext: usb-serial-6.6.8-tinycore.tcz
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- no crash over multiple month

TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- 4..6x tested, always crashes, mostly after about 15hrs; always within 24hrs

TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- SAME application, NO recompile
- run application with disabled read from /dev/ttyUSB0 by configuration constant
- 1x tested, no crash after 30hrs

Now...
This does NOT proof that the read-action is the rootcasue because earlier experiments did show that "more general load" makes the crashing earlier.

What I now have done:
- modify application code
- same "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- but close device and re-open after every 15minutes
- started at 16:00 European time

If this keeps running than I think its demonstrated that usb-serial-6.18.2-tinycore.tcz has an issue with "reading long time without intermediate closing"

fingers crossed

[patrikg]

Have you consider the BIOS with USB energy saving.

[Stefann]

I doubt my 2008 board has that.
But..... isn't that only applicable while idle? the computer is never idle but in 24/7 use.

[Stefann]

Sad.... an other crash this night at 1:30 am; after 33.5 hours

So.....
On TC17 crash after 33.5 hours if constantly reading reading from serial-over-usb port AND close/open filedescriptor every 15mins.

But....
2 days ago I concluded that running without reading serial-over-usb was crash-free as I manually stopped after 30hr of crash-free run.
>> that conclusion may be premature

Still..
WITH constant reading of serial-over-usb crash is normally in about 15hrs, and always within 24 hours.
So... although there is variation... it still looks like "altering settings of serial-over-usb" influences the behavior.

I got a tip from my brother that "SELECT" is a quite complicated kernel call.
I'm using SELECT to check that the pipe is ready for read but I can just as well call read from it in nonblocking mode without checking.
So... I now changed that...
- removed the SELECT
- read in non-blocking mode
I also made using read, select, 15min_close/open configurable by a runtime settable variable such that I can test different configurations without recompile (to exclude impact of compilation on results)

Started at 8:22 am:

TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- new compiled executable with runtime configurable serial read modes
- run application with non-blocking read without SELECT
- "every second 812byte read from /dev/ttyUSB0; baudrate 11520" from /dev/ttyUSB0


Recap of results so far:

TC15:
- including kext: usb-serial-6.6.8-tinycore.tcz
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- no crash over multiple month

TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- 4..6x tested, always crashes, mostly after about 15hrs; always within 24hrs

TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- SAME application, NO recompile
- run application with disabled read from /dev/ttyUSB0 by configuration constant
- 1x tested, NO CRASH after 30hrs >> I thought this was good but as next config crashed after 33.5hrs I may not have been running long enough

TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- include every 15 minute close/open file descriptor for /dev/ttyUSB0
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- 1x tested, crash after 33.5hrs

[Zhe]

try upgrade tc 16 ,maybe can shrink this problem

[Rich]

Hi Stefann

... I got a tip from my brother that "SELECT" is a quite complicated kernel call.
I'm using SELECT to check that the pipe is ready for read but I can just as well call read from it in nonblocking mode without checking.
So... I now changed that...
- removed the SELECT
- read in non-blocking mode ...

You mentioned in a previous post you were opening files without
closing them. If you were using those file descriptors with select(),
that could be a problem:

Code: [Select]

C_Programs/AutoCursor/AutoCursor.c:     int fdmax=0;                                    // Highest file descriptor select() should monitor.
C_Programs/AutoCursor/AutoCursor.c:             select((fdmax + 1), &Dcursor, NULL, NULL, NULL);Select is limited to somewhere around 1024 file descriptors. If you keep
opening the same file over and over again, you will eventually reach
that limit and bad things things will happen. You can test for this fairly
easily. When you open a file for select() to monitor, compare the file
descriptor to FD_SETSIZE:

Code: [Select]

if(file_descriptor >= FD_SETSIZE)
{
     print error message
     exit program
}

[Stefann]

Hi,
I appreciate a lot that you are reading my reports, I can sure use any help I'm getting.

Yes,...
I'm opening the file descriptor at the start of my program and than never close it:

Code: [Select]

fd = open("/dev/ttyUSB0", O_RDWR);
My home automation program (originally) simply keeps reading from it.
Before each read it did checks whether there was data with a select command.
The fd remains the same, that never changes, so as far as I know nothing increments and reaches a limit

Code: [Select]

call this as part of forever loop:
{ static char buf[BUFSIZE+1];
   fd_set fd_check;   
   struct timeval wait = {0};
 
  FD_ZERO(&fd_check);
  FD_SET(fd, &fd_check);
  select(fd+1, &fd_check, NULL, NULL, &wait);
  if (FD_ISSET(fd, &fd_check)  )
  { j = read(fd, &buf[i], BUFSIZE-i);
    ... process results...
   }
   other commands...
}


So.. I think in principle THIS should work, and actually it does on TC15.

However...
You are very right that "if I would open other pipes without closing" I could get in trouble.
Yes.. it's not very safely programmed, it does not include the range check.

I never intentionally keep opening files without closing when I no longer need them.
What I meant is that I open "this specific file" and keep reading without ever closing.
In itself "that is not strange", I'm actively using it.
But if "it collects some garbage over time", it never gets a fresh start.

Anyway, last experiment was to "close & reopen" every 15mins. Its seems that that about doubled the time to crash but it still crashes.

Thanks for the input, I can include a logging on number of fd's when I do a next compilation.
"never intentionally" is different from "absolutely sure I did not"
For now I keep it running. Its 13 hrs down the road now... And it's almost bedtime here. Tomorrow morning I will know whether it at least passed the "15hr average crash milestone"

[Stefann]

try upgrade tc 16 ,maybe can shrink this problem

I thought about that but that would not help me.
Just to "not have crashes" I could simply stay on TC15. But.... Than I would miss out on upgrades.
Getting upgrades is the whole point of getting to TC to begin with. I started on DSL in 2008 but DSL quickly stopped doing upgrades.
2 years ago I really had to move to a newer OS. Without upgrades I had no https and missed lots of libraries on gcc, php, python.
So... getting OS updates is important.
A stepping stone on TC16 would not give other insights.
Only when its clear which component has the error it makes sense to figure out whether that component broke from 15 to 16 or from 16 to 17.

[Paul_123]

Through all this, you still have not managed to capture any sort of log or crash dump.  You likely need a direct console connection on a monitor and local keyboard.

[Stefann]

Through all this, you still have not managed to capture any sort of log or crash dump.  You likely need a direct console connection on a monitor and local keyboard.

Before answering,..
System is still running with excluded select() call. It passed the “15hr mark”, running 22 hours now. I will try to get to 48.

Not managed to get crash logging….  Very true.
Is still not managed to get any logging.
And I sure tried.

At this moment:
- I have a monitor hardwired connected by vga (indeed a local console)
- a keyboard hardwired connected by ps2
- I write rsyslog logging at lowest level (kern.*) to a network connected 2nd system

But after a crash:
- monitor is still on “screensaver”, not able to wake up. If I press a button on the monitor it says “no signal”
- the logging on the network connected systems shows zero logging from the crash.
>> in fact, I donot get a lot of logging anyway. I see a big chunk of logging at booting of tinycore but after booting no kern.* logging is created.

Any suggestion to get logging is welcome.
In fact I should probably find a way to keep the signal on the monitor. I have no idea how.
It’s running fvwm.
When I boot in text mode (which would be more straightforward) the “time to crash” is also longer.
>> I’ll see whether I can find a setting on the monitor.

Any advice is welcome. I indeed had the objective to get some logging but “it’s not there”. It looks like the system “simply drops”.

[Stefann]

Still running, 25hrs and counting

@Paul_123, I checked my monitor but did not find any timeout things.
screensave is probably a fvwm setting? or other setting?
It may be a bios setting, I can check that after this run.

@rich,
your comment on "not letting amount of open file descriptors grow" makes perfect sense.
I'm not using any malloc in my application so I thought I was relatively "save" on risk for memory leak.
However... repeatedly opening filedescriptors without closing COULD have been a leak.
I checked all source files, conclusion:
- No, no unlimited growing, I'm very clean in closing file descriptors after I opened them. As said, the usb-serial fd is 1x opened and never closed. But NOT incrementally opened without closing
- I DID see error in checking valid opening. I generally consider it a "open failed" by comparing to null. That should have been by checking on negative.
This can lead to:
- writing or reading from negative fd's > I guess that could potentially crash the application but linux should not really crash without any logging.
- 2nd open of the same device if it had opened with fd=0, but only 1x, only in the unlikely event that stdi is not connected to 0.
 
Not nice,
I will definitely fix
But I do not think its a reason for these crashes

[Rich]

Hi Stefann

... But after a crash:
- monitor is still on “screensaver”, not able to wake up. If I press a button on the monitor it says “no signal” ...

It's not the monitor doing that. It's the operating system disabling the
video because of no keyboard or mouse activity.

Add this boot code:

Code: [Select]

consoleblank=0
I don't remember, is there a GUI running?
If there is, run these commands just before starting your program:

Code: [Select]

xset dpms 0 0 0
xset s noblank
xset s off
xset -dpms
That should keep your monitors screen from blanking.
You'll know after about 10 minutes if that worked.

[Stefann]

Thanks a lot!
I will do that.
Yes there is a gui and vnc server running.
I had switched to text mode during some earlier experiments but that made the "time to crash" much longer.
> the whole crash thing seems to have a "rootcause that triggers" and "load situation that depicts duration".

System is now running 32 hours.
> it passed the 15hr milestone: average crash duration in original state
> it is very close tot the 33.5hr milestone: crash with "every 15minute fd-close/reopen"
So.. fingers crossed.

If it does not crash I think I will keep running it for something like a week.
Because current configuration not only serves as an analysis but also can serve as a fix.
Reading without select by means of nonblocking read() statement can be a perfectly acceptable solution for my application
So... it is quite valuable to test whether this has the potential to be "crash free for multiple month". If it survives a week I get good feelings about that.

So...
- I will likely keep it running for a week if it does not crash
- if it DOES crash I will modify the monitor settings as you indicated. I will than run the original "likely crash in 15hr" configuration.

If it survives a week I will make a dedicated separate testprogram that calls select() in an endless loop with about 100x more calls/minute than my application. That would probably be able to get the system to crash in about an hour or so.

It still would make sense to find the rootcasue of select() shutting down the whole system. But that would likely take quite some time. I will definitely try to find or at least help to find the cause. But until than.... not using it can be a perfect solution for my application.