Find cause for crashes on my VIA EDEN 500MHz 1core 32bit cpu

[Stefann]

Continuation of rootcause finding towards the crashing of my VIA EDEN 500MHz 1core 32bit cpu on TC17.1 as started in this thread: https://forum.tinycorelinux.net/index.php?topic=28078.0

My system is still crashing from time to time. But its is very likely a hardware issue not related to TC17 for which reason it is now closed as TC17 issue.

Short summary of where I am:
- 1.5 weeks ago my system crashed 2x in a row.
- Than I extended an existing nanosleep() delay in my application from 3x 0 to 3x100ms per second by configuration item (no recompile) after which the system did not crash for 5 days.
- Under assumption it is "load" dependent I included a 2x 4700uF capacitor on 5V power line and brought delay back to 3x0ms per second after which it did run without crash for 5 days.
- As ultimate proof this is power related I removed the capacitors "during run" (they were mounted on a screw terminal) under assumption it should crash within 24 hours. but... no crash

- I now started an additional "continuous loop stress program (code at bottom). This brought cpu-load from 10% to 90% and added 0.6% (=10%) power consumption.

Let's see whether there is crash again in 24hrs.

@Rich, I tried your python program. It however fails:

Code: [Select]

tc@huis:~/python$ ./SimplePi.sh
./CalcPi.sh: line 4: bc: not found
./CalcPi.sh: line 6: bc: not found
So I just keep running my own stress program for now. Can change that later. Better to run "something" than nothing

Stress program I'm running (C code):

Code: [Select]

int Stress()
{ int i;
  float d1[200], d2[200];
  while (1)
  { for (i=0; i<100; i++)
    { d1[i] = 1.111*i; }
    for (i=0; i<100; i++)
    { d2[i] = d1[i]/ 1.111; }
    for (i=0; i<100; i++)
    { d1[i] = d2[i] * 1.111; }
  }
  return 1;
}It has some calculations (*1.111 and /1.111) and some memory-swaps

[Rich]

Hi Stefann

... @Rich, I tried your python program. It however fails:

Code: [Select]

tc@huis:~/python$ ./SimplePi.sh
./CalcPi.sh: line 4: bc: not found
./CalcPi.sh: line 6: bc: not found

It's not python. It's just an ash script.
The reason you got those errors is because you didn't read the scripts
help message which I even post here:
https://forum.tinycorelinux.net/index.php?topic=24096.msg152111#msg152111
It clearly states you need to install  bc.tcz.

The script creates a script to calculate  pi  to the number of decimal
places you specified when you launched  SimplePi.sh.
It then launches a copy of that script on every processor in that system.

The default number of decimal places it calculates  pi  to is 1000.
If you start it like this:

Code: [Select]

./SimplePi.sh 4000It will change the default value of  SimplePi.sh  from 1000 to 4000 and
calculate  pi  to 4000 decimal places.

[Stefann]

@Rich,
Ah.. I feel bit ashamed... THAT is something I could have figured out myself.
To be honest, I felt a bit grumphy yesterday. I really thought I was catching the issue and once again "not".

With that said... I may from time to time be bit grumpy but especially for that reason I highly appreciate all the support.

Anyway..
- I tried the SimplePi.sh script. It added +0.6W power consumption which is similar to what I had.
- I improved the stress program to also include network load by writing to a file on Ramdisk of my 2nd computer. Powerconsumption is now +1W (code at bottom of post), around 7.5W total now.
- now started it without capacitor including +1W stress program to see whether it crashes again.
- all this also shows that the 2A power supply (=10W) should in principle be enough. Even when trying I did not see anything above 8W.

Note...
- It all becomes quite "not handy". I actually really want to extend my home-control application instead of debugging the platform.
- I think I will simply start working without capacitor and if I see frequent fail install capacitor and start multi-day run. Last time I did several powermeasurements, adapter swaps before the 5 day run. Although I don't feel I have I still "may" have slightly altered the config.

Story continues.....

Code: [Select]

int Stress()
{ int i;
  float d1[200], d2[200];
  FILE *fd;
 
  while (1)
  { fd = fopen("/remote/home/tc/stress.txt", "w");
    if (fd)
    { for (i=0; i<100; i++)
       d1[i] = 1.111*i;
      for (i=0; i<100; i++)
        d2[i] = d1[i]/ 1.111;
      for (i=0; i<100; i++)
        d1[i] = d2[i] * 1.111;
      for (i=0; i<100; i++)
        fprintf(fd, "dum %d\n",d1[i]);
      fclose(fd);
     }
  }
  return 1;
}

[Rich]

Hi Stefann
When compiling your stress program, do you have optimizations
turned off  (-O0) ?

If not, the compiler may be optimizing away some of your code.
For example, your code could be reduced to:

Code: [Select]

int Stress()
{ int i;
  FILE *fd;
 
  while (1)
  { fd = fopen("/remote/home/tc/stress.txt", "w");
    if (fd)
    {  for (i=0; i<100; i++)
        fprintf(fd, "dum %d\n",(((i*1.111)/1.111)*1.111);
      fclose(fd);
     }
  }
  return 1;
}
Or if the compiler feels there would be no loss of floating point precision:

Code: [Select]

int Stress()
{ int i;
  FILE *fd;
 
  while (1)
  { fd = fopen("/remote/home/tc/stress.txt", "w");
    if (fd)
    {  for (i=0; i<100; i++)
        fprintf(fd, "dum %d\n",(i*1.111);
      fclose(fd);
     }
  }
  return 1;
}

[Stefann]

Hi Rich,
Wow... you figured out how to compress my program. I'm now seriously impressed.
>> I will look into that tomorrow, It's now evening and I now have family social obligations.
>> highly appreciated!

But....
I was out this afternoon but on return the thing had crashed again.
- NO capacitor on supply line
- application running
- improved stress program running (with added network activity)
That is within 5 hrs!
I actually do not have an exact timestamp of the crash as I only write to Ramdisk
I figured out I make a crontab that touches a file on the Ramdisk of my 2nd computer. Will do that tomorrow.

Anyway...
- either this is coincidence
- OR the added network activity to the stress program has boosted the crash-probability.

First I thought of testing with capacitor installed but than I thought I probably better to get some statistics of "crash probability in this configuration"
- I will be out tomorrow afternoon & night
- That is 20hrs from now
- I can try to let it crash 1 or more times until than.
- zero modifications, just restart with exact same programs.
- and than try to run it longer when I will be out

For info:
output of "top"

my application: /krubo/work/krubo /krubo/work/1wire.def
stress program: ./SEtst

Code: [Select]

Mem: 431448K used, 530020K free, 40140K shrd, 4448K buff, 187492K cached
CPU: 24.9% usr 49.2% sys  0.0% nic 12.2% idle  1.7% io  0.0% irq 11.8% sirq
Load average: 1.23 0.41 0.15 3/209 4179
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 4143  4090 tc       R     3128  0.3   0 44.0 ./SEtst
 4124     1 tc       R    11584  1.2   0 21.2 /krubo/work/krubo /krubo/work/1wire.def
 3359     2 root     SW       0  0.0   0 10.3 [cifsd]
   52     2 root     RW       0  0.0   0  3.0 [kworker/u4:3-ev]
   12     2 root     IW       0  0.0   0  2.7 [kworker/u4:0-ev]
   15     2 root     IW       0  0.0   0  1.7 [rcu_sched]
 4178  4150 tc       R     3600  0.3   0  0.4 top

[Stefann]

Hi Rich,
Compiler flags I use:

Code: [Select]

CFLAGS = -DDEBUG -DLINUX -Wall -m32 -I src -DDEBUG -c
LFLAGS = -DDEBUG -DLINUX -L/usr/local/lib -lusb-1.0 -lusb -lm -m32 -no-pie -I src  -g -o $@

At this moment I do not want to change anything. Just see whether I can again get a crash in few hours and "if so" than add the capacitors.

I realize that I could have seen the crash-timestamp from the stress.txt file I have on my 2nd computer. I did already overwrite that when I thought about that.
Anyways... I created a crontab that extends a file on my 2nd computer with a "date output" every 30 mins.
Starting automatically at reboot and not overwriting anything so I cannot forget about that.

.... still not sure about what to do on finger crossing...

[Rich]

Hi Stefann

Wow... you figured out how to compress my program. I'm now seriously impressed. ...

It's been a while, but I have done some C Programming.

[Stefann]

Well... this morning computer had crashed again, in 9 hours
I restarted, application auto-starts at boot.
But..... when I manually started the stress program I got an instant crash

So...
- With application running
- With stress program running (continuous RAM-use, Floating point & network)
- Without capacitor on power line

I now have following statistics:
- 5 hour to crash
- 9 hour to crash
- 1 minute to crash

I restarted, again with stress program. I have about 6.5 hours before I leave and be out until tomorrow.

[Stefann]

Well.....
No crash until I needed to go out this afternoon.
So...
- With application running
- With stress program running (continuous RAM-use, Floating point & network)
- Without capacitor on power line

Statistics:
- 5 hour to crash
- 9 hour to crash
- 1 minute to crash
- 7hr NO crash

Just before I went out I installed capacitor and did start again:
- With application running
- With stress program running (continuous RAM-use, Floating point & network)
- WITH 2x 4700uF capacitor on power line

To my surprise... On arrival I checked connection and connection was dead.
I was able to remote ssh-connect to 2nd computer to check the output of the "every half hour date-stamp written to 2nd system" only to conclude it had died within 30 minuted form start.

So.... statistics WITH capacitor
- 30min to crash...

arrrrgghhhhh

Luckily someone was still in my house who I was able to instruct how to press the reset button.
System now runs without the stress-program

Well.....
I need a night of sleep to think of next steps....

Clearly:
- stress program brings quick crashes, especially after including 85% nic-load
- but it looks like crash probability is same with & without power line filtering

Back in march I concluded all was fine on TC15. Maybe I have to test that again. And than include the stress program...
I need to sleep on this.

sad.....

[Stefann]

Well, I'm back home.
System did run without problem since yesterday but after the crash it got started without stress program.

I now:
- went back to TC15: vmlinuz, core.gz and all tcz in /optional
- recompiled my application under TC15
- recompiled the stress program under TC15
- started application
- started stress program
- just like under TC17 I'm NOT using sub-serial, usb-serial kext is NOT loaded
- power from mini ups WITH 2x 4700uF capacitor

Until march this year I NEVER had a crash on TC15.
- Until that time running WITH usb-serial
- from 3.5A adapter, not from mini-ups.
- without stress program

We'll see whether TC15 runs stable under these slightly different use conditions.

[Stefann]

Well... its still running after 28 hrs

So,.. statistics:
- With application running
- With stress program running (continuous RAM-use, Floating point & network)
- vanilla TC15 just downloaded from TC download location
- no usb-serial kext, no use of serial port

TC17 without capacitor:
- 5 hour to crash
- 9 hour to crash
- 1 minute to crash
- 7hr NO crash, manual stop

TC17 with capacitor:
- 30min to crash

TC15 with capacitor:
- 28 hour with capacitor still running > hot-remove capacitor

I switched to a less aggressive mode of finding rootcasue.
Instead of strict "not touching the system" I today did programming work on the system
- stress program keeps running
- making extensions of the application and restarting it few times.
- if it accidentally crashes "that is not real evidence", but if it keeps running "it is evidence of being healthy"

I hot removed the capacitor to see whether it still runs.
target: see whether TC15 is really crash-free

[Stefann]

Nja….
On TC15 55hr without crash
Half WITH capacitor on Powerline, half WITHOUT.
- Application running, recompiled under TC15
- Stress program running, recompiled under TC15

Looks like it is really software…..

[gadget42]

i honestly don't remember if we discussed ram failure in the previous thread(s)?

decided to start a fresh/new/stand-alone thread for MEMTEST in case we've already hammered it out in your specific application/equipment/hardware/etc.

https://forum.tinycorelinux.net/index.php?topic=28279.0

[Stefann]

i honestly don't remember if we discussed ram failure in the previous thread(s)?

decided to start a fresh/new/stand-alone thread for MEMTEST in case we've already hammered it out in your specific application/equipment/hardware/etc.

https://forum.tinycorelinux.net/index.php?topic=28279.0

Thanks for the input.

To be honest....
I am really back on the track that it is software, not hardware.

After reboot and recompile on TC15 I started the application AND stress program 4 days ago and its still running:
- 4 days running in total form mini-ups
- 1st day WITH capacitor on power line
- last 3 days WITHOUT capacitor

I'm a little less strict on "no configuration changes". I made extensions to my application, recompiled, stopped and started several times.
But the stress program was running all the time. I have not doen a single reboot over past 4 days.

While under TC17 I had 4 crashes in 36hrs.

In itself it seems "not justified" to have this thread in the "lounge" area but I feel my problem is very specific and I also no longer wnat to spent 100% of my runtime on trouble shooting. So I'm fine to be a bit in a corner of the forum.

Anyway... I doubt it's ram failure. TC17 crashed 4x in 36hrs. TC15 is already running spotless for 4 days.

I think the next thing I will try is to run under TC17 without update of ./optional folder. So all applications INCLUDING gcc compiler still at TC15 level.

I remember from one of the earlier tests that upgrading to TC17 without recompiling the application was not crashing.
So... "without using the TC17 included gcc compiler".
Since than I learned that "crash frequency" is heavily dependent on load so that conclusion may have been false. But I think it's good to go back to that, include the stress program, and work like that for a week or so.

I need to find a convenient time to test this.