Find cause for crashes on my VIA EDEN 500MHz 1core 32bit cpu

[Stefann]

Continuation of rootcause finding towards the crashing of my VIA EDEN 500MHz 1core 32bit cpu on TC17.1 as started in this thread: https://forum.tinycorelinux.net/index.php?topic=28078.0

My system is still crashing from time to time. But its is very likely a hardware issue not related to TC17 for which reason it is now closed as TC17 issue.

Short summary of where I am:
- 1.5 weeks ago my system crashed 2x in a row.
- Than I extended an existing nanosleep() delay in my application from 3x 0 to 3x100ms per second by configuration item (no recompile) after which the system did not crash for 5 days.
- Under assumption it is "load" dependent I included a 2x 4700uF capacitor on 5V power line and brought delay back to 3x0ms per second after which it did run without crash for 5 days.
- As ultimate proof this is power related I removed the capacitors "during run" (they were mounted on a screw terminal) under assumption it should crash within 24 hours. but... no crash

- I now started an additional "continuous loop stress program (code at bottom). This brought cpu-load from 10% to 90% and added 0.6% (=10%) power consumption.

Let's see whether there is crash again in 24hrs.

@Rich, I tried your python program. It however fails:

Code: [Select]

tc@huis:~/python$ ./SimplePi.sh
./CalcPi.sh: line 4: bc: not found
./CalcPi.sh: line 6: bc: not found
So I just keep running my own stress program for now. Can change that later. Better to run "something" than nothing

Stress program I'm running (C code):

Code: [Select]

int Stress()
{ int i;
  float d1[200], d2[200];
  while (1)
  { for (i=0; i<100; i++)
    { d1[i] = 1.111*i; }
    for (i=0; i<100; i++)
    { d2[i] = d1[i]/ 1.111; }
    for (i=0; i<100; i++)
    { d1[i] = d2[i] * 1.111; }
  }
  return 1;
}It has some calculations (*1.111 and /1.111) and some memory-swaps

[Rich]

Hi Stefann

... @Rich, I tried your python program. It however fails:

Code: [Select]

tc@huis:~/python$ ./SimplePi.sh
./CalcPi.sh: line 4: bc: not found
./CalcPi.sh: line 6: bc: not found

It's not python. It's just an ash script.
The reason you got those errors is because you didn't read the scripts
help message which I even post here:
https://forum.tinycorelinux.net/index.php?topic=24096.msg152111#msg152111
It clearly states you need to install  bc.tcz.

The script creates a script to calculate  pi  to the number of decimal
places you specified when you launched  SimplePi.sh.
It then launches a copy of that script on every processor in that system.

The default number of decimal places it calculates  pi  to is 1000.
If you start it like this:

Code: [Select]

./SimplePi.sh 4000It will change the default value of  SimplePi.sh  from 1000 to 4000 and
calculate  pi  to 4000 decimal places.

[Stefann]

@Rich,
Ah.. I feel bit ashamed... THAT is something I could have figured out myself.
To be honest, I felt a bit grumphy yesterday. I really thought I was catching the issue and once again "not".

With that said... I may from time to time be bit grumpy but especially for that reason I highly appreciate all the support.

Anyway..
- I tried the SimplePi.sh script. It added +0.6W power consumption which is similar to what I had.
- I improved the stress program to also include network load by writing to a file on Ramdisk of my 2nd computer. Powerconsumption is now +1W (code at bottom of post), around 7.5W total now.
- now started it without capacitor including +1W stress program to see whether it crashes again.
- all this also shows that the 2A power supply (=10W) should in principle be enough. Even when trying I did not see anything above 8W.

Note...
- It all becomes quite "not handy". I actually really want to extend my home-control application instead of debugging the platform.
- I think I will simply start working without capacitor and if I see frequent fail install capacitor and start multi-day run. Last time I did several powermeasurements, adapter swaps before the 5 day run. Although I don't feel I have I still "may" have slightly altered the config.

Story continues.....

Code: [Select]

int Stress()
{ int i;
  float d1[200], d2[200];
  FILE *fd;
 
  while (1)
  { fd = fopen("/remote/home/tc/stress.txt", "w");
    if (fd)
    { for (i=0; i<100; i++)
       d1[i] = 1.111*i;
      for (i=0; i<100; i++)
        d2[i] = d1[i]/ 1.111;
      for (i=0; i<100; i++)
        d1[i] = d2[i] * 1.111;
      for (i=0; i<100; i++)
        fprintf(fd, "dum %d\n",d1[i]);
      fclose(fd);
     }
  }
  return 1;
}

[Rich]

Hi Stefann
When compiling your stress program, do you have optimizations
turned off  (-O0) ?

If not, the compiler may be optimizing away some of your code.
For example, your code could be reduced to:

Code: [Select]

int Stress()
{ int i;
  FILE *fd;
 
  while (1)
  { fd = fopen("/remote/home/tc/stress.txt", "w");
    if (fd)
    {  for (i=0; i<100; i++)
        fprintf(fd, "dum %d\n",(((i*1.111)/1.111)*1.111);
      fclose(fd);
     }
  }
  return 1;
}
Or if the compiler feels there would be no loss of floating point precision:

Code: [Select]

int Stress()
{ int i;
  FILE *fd;
 
  while (1)
  { fd = fopen("/remote/home/tc/stress.txt", "w");
    if (fd)
    {  for (i=0; i<100; i++)
        fprintf(fd, "dum %d\n",(i*1.111);
      fclose(fd);
     }
  }
  return 1;
}

[Stefann]

Hi Rich,
Wow... you figured out how to compress my program. I'm now seriously impressed.
>> I will look into that tomorrow, It's now evening and I now have family social obligations.
>> highly appreciated!

But....
I was out this afternoon but on return the thing had crashed again.
- NO capacitor on supply line
- application running
- improved stress program running (with added network activity)
That is within 5 hrs!
I actually do not have an exact timestamp of the crash as I only write to Ramdisk
I figured out I make a crontab that touches a file on the Ramdisk of my 2nd computer. Will do that tomorrow.

Anyway...
- either this is coincidence
- OR the added network activity to the stress program has boosted the crash-probability.

First I thought of testing with capacitor installed but than I thought I probably better to get some statistics of "crash probability in this configuration"
- I will be out tomorrow afternoon & night
- That is 20hrs from now
- I can try to let it crash 1 or more times until than.
- zero modifications, just restart with exact same programs.
- and than try to run it longer when I will be out

For info:
output of "top"

my application: /krubo/work/krubo /krubo/work/1wire.def
stress program: ./SEtst

Code: [Select]

Mem: 431448K used, 530020K free, 40140K shrd, 4448K buff, 187492K cached
CPU: 24.9% usr 49.2% sys  0.0% nic 12.2% idle  1.7% io  0.0% irq 11.8% sirq
Load average: 1.23 0.41 0.15 3/209 4179
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 4143  4090 tc       R     3128  0.3   0 44.0 ./SEtst
 4124     1 tc       R    11584  1.2   0 21.2 /krubo/work/krubo /krubo/work/1wire.def
 3359     2 root     SW       0  0.0   0 10.3 [cifsd]
   52     2 root     RW       0  0.0   0  3.0 [kworker/u4:3-ev]
   12     2 root     IW       0  0.0   0  2.7 [kworker/u4:0-ev]
   15     2 root     IW       0  0.0   0  1.7 [rcu_sched]
 4178  4150 tc       R     3600  0.3   0  0.4 top

[Stefann]

Hi Rich,
Compiler flags I use:

Code: [Select]

CFLAGS = -DDEBUG -DLINUX -Wall -m32 -I src -DDEBUG -c
LFLAGS = -DDEBUG -DLINUX -L/usr/local/lib -lusb-1.0 -lusb -lm -m32 -no-pie -I src  -g -o $@

At this moment I do not want to change anything. Just see whether I can again get a crash in few hours and "if so" than add the capacitors.

I realize that I could have seen the crash-timestamp from the stress.txt file I have on my 2nd computer. I did already overwrite that when I thought about that.
Anyways... I created a crontab that extends a file on my 2nd computer with a "date output" every 30 mins.
Starting automatically at reboot and not overwriting anything so I cannot forget about that.

.... still not sure about what to do on finger crossing...

[Rich]

Hi Stefann

Wow... you figured out how to compress my program. I'm now seriously impressed. ...

It's been a while, but I have done some C Programming.

[Stefann]

Well... this morning computer had crashed again, in 9 hours
I restarted, application auto-starts at boot.
But..... when I manually started the stress program I got an instant crash

So...
- With application running
- With stress program running (continuous RAM-use, Floating point & network)
- Without capacitor on power line

I now have following statistics:
- 5 hour to crash
- 9 hour to crash
- 1 minute to crash

I restarted, again with stress program. I have about 6.5 hours before I leave and be out until tomorrow.