Find cause for crashes on my VIA EDEN 500MHz 1core 32bit cpu

[Stefann]

Ok.. BAM... crash in 5hrs.
At least the stress program really helps to trigger crashes within 24hrs when the system is in a vulnerable state.
So... its really linux.

I'm out now but had someone at home to reboot. Reboot goes to TC15, I had made a special menu-label in the boot loader to manually bring it to the hybrid boot for test.

So...
- TC15: run 7 days without any problem
- TC17: 5x crashed within 9hrs

Next time I probably want to see what happens if I run the stress program without my main application under TC17.
I'm now out, will do tomorrow afternoon.

[Stefann]

Nha..
Rebooted
- TC17.1 vmlinuz & core.gz
- TC15 applications
- NOT running my home controls application
- running stress program
- also running some standard apps: vnc-server, ssh, samba, apache

The objective being "see whether stress-conditions crash TC17 if I doNOT run my own application"
Rationale: My application is quite a hobby project and I would not be surprised if it hits into some illegal situation (although I also think it got hardened quite a lot over past 18 years), but it should still nowhere be able to crash the OS.
However....; let's put that to a test by "completely not running it".

It was a bit of an effort. I moved the "http-controlled peripherals" to get controlled from my 2nd system. 1wire and x10 devices are now "non functional". I can live with that temporarily.

output of top:

Code: [Select]

Mem: 288244K used, 673224K free, 39704K shrd, 4284K buff, 176652K cached
CPU: 23.8% usr 37.2% sys  0.0% nic 25.9% idle  6.3% io  0.0% irq  6.6% sirq
Load average: 1.09 0.77 0.35 5/186 4940
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 4937  4905 tc       D     3128  0.3   0 47.0 ./stress15
 4260     2 root     RW       0  0.0   0 11.0 [cifsd]
   38     2 root     IW       0  0.0   0  3.0 [kworker/u4:1-ev]
   52     2 root     IW       0  0.0   0  1.9 [kworker/u4:3-ev]
   12     2 root     IW       0  0.0   0  1.5 [kworker/u4:0-ev]
   15     2 root     IW       0  0.0   0  1.5 [rcu_sched]
 4875     1 root     S     8172  0.8   0  0.4 x0vncserver -PasswordFile=/home/tc/.vnc/passwd
   14     2 root     SW       0  0.0   0  0.4 [ksoftirqd/0]
 4904  4901 tc       S     7432  0.7   0  0.2 sshd: tc@pts/0
 4938  4905 tc       R     3600  0.3   0  0.2 top

[Stefann]

well.... NO crash after 16hrs

configuration:
- TC17.1 vmlinuz & core.gz
- TC15 applications
- NOT running my home controls application
- RUNNING stress program
- also running some standard apps: vnc-server, ssh, samba, apache

previously 5x crash within 9hrs
- TC17
- RUNNING my home controls application
- RUNNING stress program
- also running some standard apps: vnc-server, ssh, samba, apache

So:
- either the lower load from not running application still makes it crash but after "more than 16hrs"
- or the application has something that triggers the crash

To be 100% sure on the 1st option I should run like this for few days. That however is not so practical as my application controls some essential things in my home

I now started:
configuration:
- TC17.1 vmlinuz & core.gz
- TC15 applications
- RUNNING my home controls application
  - WITHOUT http calls
  - WITHOUT x10-pheripheral being connected to usb and using libusb 1.0 library calls
  - WITH 1wire-pheripheral being connected to usb and using libusb 2.0 library calls
- RUNNING stress program
- also running some standard apps: vnc-server, ssh, samba, apache

The reason I try my luck here is:
- crash frequency went up significantly (from few days to less than 9 hrs) when adding network activity to the stress program. This made me suspect it's something with network.
- 2 month ago I had a crash after 5weeks without crash. After that I modified my program to no longer use the 1read/second usb-serial connection (as that was the suspected rootcasue) and started using 1call/second http-calls to a different interface to get the data I needed. Instead of reducing crashes that increased the amount of crashes.
- the http-function in my application is something I wrote in 2009 from scratch using socket calls. That part is definitely not completely robust. In case of failed calls the application could keep using invalide filedescriptors.

So...
By running my application without http calls I certainly test whether libusb calls cause the crash. But apart from the libusb calls my application does nothing more spectacular than opening/reading/writing/closing files and basic program-flow.
>> if this does not crash that hints towards the http-calls (or the libusb1.0 calls, that would be a next test).

I keep it running like this for a day and see where that brings me.

Note:
IF it appears that my http handling is causing the crashes, than this is certainly caused by "not adequate robustness of my application", I already checked the code and this is factually the case. I see some cases in which a failed socket-call can later result in socket call with invalide filedescriptior.
However.....
That should still not cause linux to crash, it could crash my application but the OS should be robust to this.
It's "kind of probable" that this is the rootcasue. I wrote this component in 2009 when I was using DSL (Damn Small Linux). At that time I had little other options for doing http calls and my application was very minimal, just calling 1 local http-connected device from time to time.
These days raw socket calls are quite uncommon in hobby programs, it would be more normal to use curl calls. Applications that use raw socket calls are likely professional and have good error handling.
So... it is at least somewhat imaginable that some OS-vulnerability on dirty socket use could have gone unnoticed during linux release testing.

[Stefann]

Sooo..... 23hr without crash!
- TC17.1 vmlinuz & core.gz
- TC15 applications
- RUNNING my home controls application
  - WITHOUT http calls
  - WITHOUT x10-pheripheral being connected to usb and using libusb 1.0 library calls
  - WITH 1wire-pheripheral being connected to usb and using libusb 2.0 library calls
- RUNNING stress program
- also running some standard apps: vnc-server, ssh, samba, apache

previously 5x "crash within 9hrs"
- TC17.1
- RUNNING my home controls application including http & libusb calls
- RUNNING stress program
- also running some standard apps: vnc-server, ssh, samba, apache


This brings big suspicion that the http calls in my application are the problem.
As said, the http calls from my application are written in a "not so clean part of code" using raw socket calls. It's a very old part of code that I wrote in 2009. If I would do that today I would likely use curl.
A further hint that the socket calls are the problem is that the crash-frequency went from multiple days to less than 9hrs when I added network-writes to my stress program (code at bottom). I'm writing to my network connected 2nd computer that I mounted using gifs:

Code: [Select]

sudo mount -t cifs -o vers=1.0,username=tc //192.168.2.203/root /remoteSo, NOT using nfs. That means that the network actions also use tcp sockets. (note, there is no wisdom behind this choice, it was just the first option I found on google when I did this).

Going forward....
Yesterday I hardened the http module of my application using my 2nd system. I made sure that all socket related calls now have proper error handling. I also included some syslog messages for every "added error-handling". This way I will see what errornous behavior previously went uncatched.
I did already testrun this on my 2nd system and I did sure see socket-errors that previously went uncatched.

I just rebooted basically full blown configuration:
- TC17.1 vanilla from TC download area including applications
- full-blown RUNNING my home controls application, including hardened http calls and libusb calls freshly compiled under TC17.1
- RUNNING stress program, freshly compiled under TC17.1
- also running some standard apps: vnc-server, ssh, samba, apache
Let's see whether this can run without crash!

I skipped a few steps in gradually working from yesterday configuration to current situation. Reason:
- running TC17.1 kernel with TC15 application did not seem to bring more robustness, not much reason to run like that
- disabling the X10 interface I kind of judge "less likely" to have added much to the robustness. It uses the libusb1.0 interface (by libusb compatibility library) but as the 1wire interface using the libusb2.0 library is functioning fine I donot see much danger.
If it starts crashing I will work backwards.

snippet of top to show load:

Code: [Select]

Mem: 476748K used, 484720K free, 42048K shrd, 4116K buff, 291860K cached
CPU: 24.9% usr 46.9% sys  0.0% nic 12.1% idle  1.6% io  0.0% irq 14.2% sirq
Load average: 1.91 1.93 1.98 4/207 4353
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 4084  3882 tc       R     3128  0.3   0 42.2 ./stress17
 4176     1 tc       R    11584  1.2   0 21.9 /krubo/work/krubo17 /krubo/work/1wire.def
 3360     2 root     SW       0  0.0   0  9.8 [cifsd]
 4331     2 root     IW       0  0.0   0  3.1 [kworker/u4:0-ev]
 4234     2 root     IW       0  0.0   0  2.7 [kworker/u4:3-ev]
   15     2 root     IW       0  0.0   0  1.6 [rcu_sched]
 4352  3882 tc       R     3600  0.3   0  0.6 top
 3382     1 root     S    54584  5.6   0  0.4 /usr/local/sbin/rsyslogd
   14     2 root     SW       0  0.0   0  0.4 [ksoftirqd/0]
 4009     1 root     S     8192  0.8   0  0.2 x0vncserver -PasswordFile=/home/tc/.vnc/passwd
 3877  3785 tc       S     7972  0.8   0  0.2 sshd-session: tc@pts/0
 3919  3874 tc       S     405m 43.0   0  0.0 /usr/local/sbin/httpd -k start
 3921  3874 tc       S     404m 42.9   0  0.0 /usr/local/sbin/httpd -k start
 4087  3874 tc       S     391m 41.6   0  0.0 /usr/local/sbin/httpd -k start
 3937  3874 tc       S     384m 40.9   0  0.0 /usr/local/sbin/httpd -k start
 3874     1 root     S     162m 17.2   0  0.0 /usr/local/sbin/httpd -k start
 3878     1 root     S    44976  4.6   0  0.0 Xvesa -br -screen 1024x768x32 -shadow -2button -mouse /dev/input/mice
 4142  3687 root     S    19208  1.9   0  0.0 /usr/local/sbin/smbd -D
 3701  3687 root     S    18868  1.9   0  0.0 /usr/local/sbin/smbd -D
 3687     1 root     S    18812  1.9   0  0.0 /usr/local/sbin/smbd -D
 4031     1 tc       S     9852  1.0   0  0.0 wbar
 3914     1 tc       S     9628  1.0   0  0.0 flwm
code of stress program:

Code: [Select]

int Stress()
{ int i;
  float d1[200], d2[200];
  FILE *fd;
 
  while (1)
  { fd = fopen("/remote/home/tc/stress.txt", "w");
    if (fd)
    { for (i=0; i<100; i++)
       d1[i] = 1.111*i;
      for (i=0; i<100; i++)
        d2[i] = d1[i]/ 1.111;
      for (i=0; i<100; i++)
        d1[i] = d2[i] * 1.111;
      for (i=0; i<100; i++)
        fprintf(fd, "dum %d\n",d1[i]);
      fclose(fd);
     }
  }
  return 1;
}

[Stefann]

Unfortunately I got an other crash yesterday.
Crash around 23:00
That still "kind of fits the theory" of socket related error:
- "somewhere between 22:00 and 23:00" my solar energy server has a 1 minute "automatic service" during which it is unresponsive.
- When I tested the "improved http module" on my 2nd system a day earlier I could see that the http-module had zero errors during day but a big stream of errors during that 1 minute non-responsive time.

So... IF the crash is related to this downtime it may still very well be related to socket issue.

My improved application had proper error handling included on all sockets calls.... except for 1.... the connect() call.
Reason being that I call "connect()" in nonblocking mode and than it always throws an error. Yesterday I did not know how to handle that.
So.... its is still a valid theory that connect() throws errors when server is non-responsive and my system crashes for reason of not handling those errors.

Today I learned how to handle connect() errors:
- ignore "in progress" error
- but abort connection on all other errors
- check connection by "getpeername()" before using
I added that.

Just started again, full blown program under TC17.1:
- TC17.1 vanilla from TC download area including applications
- full-blown RUNNING my home controls application, including hardened http calls and libusb calls freshly compiled under TC17.1
- RUNNING stress program, freshly compiled under TC17.1
- also running some standard apps: vnc-server, ssh, samba, apache

My hopes are not super high, at 1st try I got a crash the moment I started the stress program.
After that I rerouted the syslog for socket related errors to the network connected 2nd computer. By doing that I have latest messages in case I get a next crash
On every catched error from socket operations I write a syslog message

And I routed all kernel logging to console (physical)

So...
- on 2nd start it survived the starting of the stress program
- lets see how it runs
- lets see whether it survives the 23:00-ish network challenge
- lets see whether I have useful logging IF it crashes

Note,
One can argue that I better move to curl instead of attempting a home-brew http module.
-> That is an entirely correct thought.
However... "moving to something like curl" is quite some work.
- I need a nonblocking implementation OR I need to include threading. I'm pretty sure I will be able to figure out how to do that BUT I'm also pretty sure that such is quite complicated and takes time.
- At this moment it's still non-proven that this is the issue. Potentially I would do a lot of work "without getting a better live".
- I'm not 100% sure such path would even help. "good weather behavior" is 100% OK for my application. I now have proper error handling around every system call. curl will use the same system calls. "why would it than behave better"?
- anyway... the thought is valid. Maybe at some point.

[gadget42]

again, thanks for keeping us updated!!!

[Stefann]

again, thanks for keeping us updated!!!

Thanks!
An occasional "sign of life" from readers is appreciated. I was already wondering whether my writings are appreciated.

I got multiple crashes since last post.
Also the connection to my network connected drive became unstable (multiple minutes to respond)
I ended up removing the getpeername() call. Since than things are at least running.

All in all I still see a significant likelihood that "socket instability" is the rootcause. Behavior is changing wildly when I'm poking with that.
At least the "connect()" now has proper error-handling. I'm pretty sure that call got uncatched errors until yesterday.
In principle linux should not completely crash, only the application should, but on the other hand "running an application that is supposed to be super robust while not catching communication errors" is bad practice.

We'll see how this continues

[Stefann]

Small update, yesterday evening an other crash.
Today I will not have much time to work on this.

I however think I’m getting into a line of thought:
- I see clear crashing increase when massively using the network adapter with my stress-program that brought network traffic from <1% to >80%.
- I now see consistent crashes around 22:30 which coincides with my solar energy inverter going into a 1 minute service-routine that makes it unreachable.
- Problems started when I updated from TC15 to TC17. This suggests a Linux error but alternatively the network card (network card firmware) could be buggy. Probably never triggered under TC15 but getting triggered under TC17.

With this line of thought there is plenty to test:
- instead of a daily wait for the 22:30 service loop, will things also crash if I simply pull the cable?
- does TC15 crash on this 1 minute unreadable server IF I run the stress program as well (force >80% network traffic)?
- isolate the exact network call that crashes by adding syslog lines around them.

And also….
I now see consistent crashing around 22:30 under 80% network traffic. I however included proper error handling in my application. It may well function OK if I not run the stress program.

So….
Plenty to test.
I will keep you posted but it may take a few days.