Find cause for crashes on my VIA EDEN 500MHz 1core 32bit cpu

[Stefann]

Ok.. BAM... crash in 5hrs.
At least the stress program really helps to trigger crashes within 24hrs when the system is in a vulnerable state.
So... its really linux.

I'm out now but had someone at home to reboot. Reboot goes to TC15, I had made a special menu-label in the boot loader to manually bring it to the hybrid boot for test.

So...
- TC15: run 7 days without any problem
- TC17: 5x crashed within 9hrs

Next time I probably want to see what happens if I run the stress program without my main application under TC17.
I'm now out, will do tomorrow afternoon.

[Stefann]

Nha..
Rebooted
- TC17.1 vmlinuz & core.gz
- TC15 applications
- NOT running my home controls application
- running stress program
- also running some standard apps: vnc-server, ssh, samba, apache

The objective being "see whether stress-conditions crash TC17 if I doNOT run my own application"
Rationale: My application is quite a hobby project and I would not be surprised if it hits into some illegal situation (although I also think it got hardened quite a lot over past 18 years), but it should still nowhere be able to crash the OS.
However....; let's put that to a test by "completely not running it".

It was a bit of an effort. I moved the "http-controlled peripherals" to get controlled from my 2nd system. 1wire and x10 devices are now "non functional". I can live with that temporarily.

output of top:

Code: [Select]

Mem: 288244K used, 673224K free, 39704K shrd, 4284K buff, 176652K cached
CPU: 23.8% usr 37.2% sys  0.0% nic 25.9% idle  6.3% io  0.0% irq  6.6% sirq
Load average: 1.09 0.77 0.35 5/186 4940
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 4937  4905 tc       D     3128  0.3   0 47.0 ./stress15
 4260     2 root     RW       0  0.0   0 11.0 [cifsd]
   38     2 root     IW       0  0.0   0  3.0 [kworker/u4:1-ev]
   52     2 root     IW       0  0.0   0  1.9 [kworker/u4:3-ev]
   12     2 root     IW       0  0.0   0  1.5 [kworker/u4:0-ev]
   15     2 root     IW       0  0.0   0  1.5 [rcu_sched]
 4875     1 root     S     8172  0.8   0  0.4 x0vncserver -PasswordFile=/home/tc/.vnc/passwd
   14     2 root     SW       0  0.0   0  0.4 [ksoftirqd/0]
 4904  4901 tc       S     7432  0.7   0  0.2 sshd: tc@pts/0
 4938  4905 tc       R     3600  0.3   0  0.2 top

[Stefann]

well.... NO crash after 16hrs

configuration:
- TC17.1 vmlinuz & core.gz
- TC15 applications
- NOT running my home controls application
- RUNNING stress program
- also running some standard apps: vnc-server, ssh, samba, apache

previously 5x crash within 9hrs
- TC17
- RUNNING my home controls application
- RUNNING stress program
- also running some standard apps: vnc-server, ssh, samba, apache

So:
- either the lower load from not running application still makes it crash but after "more than 16hrs"
- or the application has something that triggers the crash

To be 100% sure on the 1st option I should run like this for few days. That however is not so practical as my application controls some essential things in my home

I now started:
configuration:
- TC17.1 vmlinuz & core.gz
- TC15 applications
- RUNNING my home controls application
  - WITHOUT http calls
  - WITHOUT x10-pheripheral being connected to usb and using libusb 1.0 library calls
  - WITH 1wire-pheripheral being connected to usb and using libusb 2.0 library calls
- RUNNING stress program
- also running some standard apps: vnc-server, ssh, samba, apache

The reason I try my luck here is:
- crash frequency went up significantly (from few days to less than 9 hrs) when adding network activity to the stress program. This made me suspect it's something with network.
- 2 month ago I had a crash after 5weeks without crash. After that I modified my program to no longer use the 1read/second usb-serial connection (as that was the suspected rootcasue) and started using 1call/second http-calls to a different interface to get the data I needed. Instead of reducing crashes that increased the amount of crashes.
- the http-function in my application is something I wrote in 2009 from scratch using socket calls. That part is definitely not completely robust. In case of failed calls the application could keep using invalide filedescriptors.

So...
By running my application without http calls I certainly test whether libusb calls cause the crash. But apart from the libusb calls my application does nothing more spectacular than opening/reading/writing/closing files and basic program-flow.
>> if this does not crash that hints towards the http-calls (or the libusb1.0 calls, that would be a next test).

I keep it running like this for a day and see where that brings me.

Note:
IF it appears that my http handling is causing the crashes, than this is certainly caused by "not adequate robustness of my application", I already checked the code and this is factually the case. I see some cases in which a failed socket-call can later result in socket call with invalide filedescriptior.
However.....
That should still not cause linux to crash, it could crash my application but the OS should be robust to this.
It's "kind of probable" that this is the rootcasue. I wrote this component in 2009 when I was using DSL (Damn Small Linux). At that time I had little other options for doing http calls and my application was very minimal, just calling 1 local http-connected device from time to time.
These days raw socket calls are quite uncommon in hobby programs, it would be more normal to use curl calls. Applications that use raw socket calls are likely professional and have good error handling.
So... it is at least somewhat imaginable that some OS-vulnerability on dirty socket use could have gone unnoticed during linux release testing.