Re: Tiny Core v17.0 upgrade issues

[Stefann]

small update:
- still running after 48 hour.

This is the first time I have more than 34 hours "full functional on TC17".
- on the one hand this is a stromg indication that select() is the root cause
- on the other hand "not using select() and doing nonblocking read" is a perfect fix for my application

I keep it running for 7 days, until next Friday. If it did not crash by than I call it "demonstrated OS/kext rootcause" and "demonstrated application fix/workaround".

So: No posts on this thread for coming 5 days unless there is a crash.

[Stefann]

3x 24hr = 72hr crashfree continuous run and counting..

[Stefann]

4x 24hr = 96hr crashfree continuous run and counting..

[Stefann]

5x 24hr = 120hr crashfree continuous run and counting..

[Stefann]

6x 24hr = 144hr crashfree continuous run and counting..

Tomorrow around this time I will stop it manual if it still did not crash.
A full week of crash free run I would consider "demonstrated".
That will give me at least a "fix/workaround to run my application under TC17".

I will however do a bit more work to zoom-in on rootcause...

I will get the screensaver deactivated by the instructions from @Rich
Than I will restart with select() back in but read() still on non-blocking (the select() will than not have any function but just be there to test whether it hurts).

The difference between original program and currently running version is not only that select() is no longer called, the read() is now called in non-blocking mode in stead of "select() gated blocking mode". So.... rootcause could be select() but it could also be that "blocking read() call" blocks forever even if there is no reason to do so. This would actually connect to the fact that zero logging gets produced at a crash. "The system just locks".
I'm running on a single core cpu. So if this single core gets stuck there is nothing else to take over (not sure it works like that, but it is at least a fact).

Current version of the application is made configurable on this by a user setting so I can run this configuration change without need for recompile. That is a benefit. This brings the amount of changes to an absolute minimum.

So....
- Any advice from a different timezone on "what to do with next run" is welcome.
- Again... although I'm logging rsyslog towards a network connected second computer, I get zero logging from rsyslog with a crash, even though I'm logging at kern.* level. Any advice to get more logging would also be welcome.

[Stefann]

7x 24hr (-1hr, started early today) = 167hr crashfree continuous run
Good!

I just stopped the application manually.

I restarted the application with select(). The read() is still in nonblocking mode.
That means that the select() does not have a function. Its only there to see whether it crashes.
As said yesterday I originally planned to disable the screensaver to allow my monitor to show logging. But... I decide not to do that. I have earlier seen that addition changes bring different behavior.
I now did zero changes. I only started the application with a different config-file.

So... if the select() is the crash-cause it should crash within 24hr, likely around 15hr
If it does not crash the "non blocking nature" of the read() is probably the crash-cause.
fingers crossed again...

Recap of results so far:

TC15:
- including kext: usb-serial-6.6.8-tinycore.tcz
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- no crash over multiple month

TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- 4..6x tested, always crashes, mostly after about 15hrs; always within 24hrs

TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- SAME application, NO recompile
- run application with disabled read from /dev/ttyUSB0 by configuration constant
- 1x tested, NO CRASH after 30hrs >> I thought this was good but as next config crashed after 33.5hrs I may not have been running long enough

TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- not use select()
- use read() in non-blocking mode
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- 1x tested, no crash, manually stopped after 7 days, 167hr.
SO:
- using the read() in non-blocking mode has all functionality I need, so to my application this is a very acceptable fix.
- Rootcasue for crash is likely either the select() OR the blocking nature of the read(), I keep zooming in to find out.

[gadget42]

thanks for taking the time to relate your testing and subsequent results!

[Stefann]

@gadget42, thanks for the warm comment.
It feels good. Assuming this ends in finding an OS-bug and not a “stupid stupid application mistake after all”, it’s my contribution to the community which is more than fair given all that I’m taken from it.

This morning, after 23 hours, still running.
So I’m happy that I did not change the screensaver setup as than I would have been doubting that to be the reason.
So far “faulty configurations” crash on average after 15 hours, max I have seen after 35 hours. So I keep it running until tomorrow morning to see whether it’s still alive after 48 hours. I have obligations today so no opportunity to do anything earlier anyways.

If this keeps running it would indicate that select() is NOT the cause.
It would than strongly point tot the blocking setting of read() to be the cause.
If it still runs tomorrow morning I will restart without select() and with read() in blocking mode.
I get data bursts every second so the blocking read() will pass regularly. This can NOT be a solution for my application because it basically freezes with 1 second gaps. But that’s ok. It’s testing.

[Stefann]

Still running after 47hrs.
So far the longest it has been running in a faulty mode was 35hr so it looks like "non faulty" but I may need 7days to be fully sure.
- that means that the select() does not look like to cause trouble
- the only other change compared to crashing April 20 and april21 run was the use of non-blocking read()

So... I now started a run that uses blocking read() but NO select.
The serial data comes in as a continuous per second stream of 0.5sec datablocks. As the read() is no longer protected by select() this means the application is pausing for 0.5sec/second so it basically runs at half speed.
I had to adapt and recompile the program to make "blocking" a configuration item.

To summarize:

Just halted:
TC17: 47 hrs crash free run; start may-1 8:00 > may-3 7:00; MANUALLY STOPPED
- including kext: usb-serial-6.18.2-tinycore.tcz
- same executable as before
- configuration: non-blocking read() AND select()
- "every second 812byte read from /dev/ttyUSB0; baudrate 11520" from /dev/ttyUSB0
- 1x tested, 47hrs

Just started:
TC17: xxxx; start may-3 7:20 > xxxx
- including kext: usb-serial-6.18.2-tinycore.tcz
- recompiled executable with configuration option to call read() in blocking mode
- configuration: blocking read(), NO select()
- "every second 812byte read from /dev/ttyUSB0; baudrate 11520" from /dev/ttyUSB0
- 1x tested, xxxx

Note:
Few weeks ago I transferred the network connected pheripherals to my 2nd computer. It's not optimal but my home automation is "with less nice gui" operational being split over 2 computers during this test phase.

[Stefann]

Oops, still running after 24hrs but I miscounted a day and manually stopped while I could better have let it run for an other day.

So:
- running with blocking read() without select() for 24hrs without crash
- accidentally manually stopped it
- restarted. I will let it run for 48hrs unless it crashes before that

For those interested, this is how the code looks likes

I use enable_meter as a global variable that is read from a configuration file to control what functions are active.

Selection bit masks:

Code: [Select]

#define meterREAD    1
#define meterFD      2
#define meterSELECT  4
#define meterREFRESH 8
#define meterBLOCKING 16
Initialize (call once):

Code: [Select]

int start_meter(tmeter *X)
{ int flags;
  meterMODE = enable_meter;

     /* some things we want to set arbitrarily */
   (X->pts).c_lflag &= ~ICANON;
   (X->pts).c_lflag &= ~(ECHO | ECHOCTL | ECHONL);
   (X->pts).c_cflag |= HUPCL;
   (X->pts).c_cc[VMIN] = 1;
   (X->pts).c_cc[VTIME] = 0;
   
   /* Standard CR/LF handling: this is a dumb terminal.
    * Do no translation:
    *  no NL -> CR/NL mapping on output, and
    *  no CR -> NL mapping on input.
    */
   (X->pts).c_oflag |= ONLCR;

   (X->pts).c_iflag &= ~ICRNL;

  /* set hardware flow control by default */
  (X->pts).c_cflag |= CRTSCTS;
  (X->pts).c_iflag &= ~(IXON | IXOFF | IXANY);
  /* set 115200 bps speed by default */
  cfsetospeed(&(X->pts), B115200);
  cfsetispeed(&(X->pts), B115200);

 
  X->fd = open(X->dev, O_RDWR);
  if (X->fd>=0)
  {  flags = fcntl(X->fd, F_GETFL);
      if ( !(meterMODE & meterBLOCKING) )
        fcntl(X->fd, F_SETFL, flags | O_NONBLOCK);
     tcsetattr(X->fd, TCSANOW, &(X->pts));
     X->SecLastRefresh = SECNOW();
  }
  return 1;
}
service (call every 250ms):

Code: [Select]

int do_readmeter(tmeter *X)
{ static int i=0;
  static int j=0;
  static int n=0;
  static int k=0;
  static char buf[BUFSIZE+1];
  fd_set fd_check;   
  struct timeval wait = {0};
 
meterMODE = enable_meter;


  if (meterMODE & meterFD)
  { FD_ZERO(&fd_check);
    FD_SET(X->fd, &fd_check);
  }
  if (meterMODE & meterSELECT)
    select(X->fd +1, &fd_check, NULL, NULL, &wait);
 
  if (meterMODE & meterFD)
    FD_ISSET(X->fd, &fd_check);
 
  //if (FD_ISSET(X->fd, &fd_check)  ) //this was originally there but removed for this investigation
  if (meterMODE & meterREAD)
  { j = read(X->fd, &buf[i], BUFSIZE-i);
   
    while (j>0)
    { buf[i] &=0x007F;
      if  (  buf[i]=='\n' )
      { do_parsemeter(buf, X);
        i++; j--;
        n=i;
        i=0;
        for (k=0; k<=j; k++)
          buf[i+k]=buf[i+n+k];
      }
      else
      { i++; j--;
      }
      if (i >= BUFSIZE)
      {  i=0; j=0;
      }
    }   
  }
  return 1;
}