Find cause for crashes on my VIA EDEN 500MHz 1core 32bit cpu

[Stefann]

Ok.. BAM... crash in 5hrs.
At least the stress program really helps to trigger crashes within 24hrs when the system is in a vulnerable state.
So... its really linux.

I'm out now but had someone at home to reboot. Reboot goes to TC15, I had made a special menu-label in the boot loader to manually bring it to the hybrid boot for test.

So...
- TC15: run 7 days without any problem
- TC17: 5x crashed within 9hrs

Next time I probably want to see what happens if I run the stress program without my main application under TC17.
I'm now out, will do tomorrow afternoon.

[Stefann]

Nha..
Rebooted
- TC17.1 vmlinuz & core.gz
- TC15 applications
- NOT running my home controls application
- running stress program
- also running some standard apps: vnc-server, ssh, samba, apache

The objective being "see whether stress-conditions crash TC17 if I doNOT run my own application"
Rationale: My application is quite a hobby project and I would not be surprised if it hits into some illegal situation (although I also think it got hardened quite a lot over past 18 years), but it should still nowhere be able to crash the OS.
However....; let's put that to a test by "completely not running it".

It was a bit of an effort. I moved the "http-controlled peripherals" to get controlled from my 2nd system. 1wire and x10 devices are now "non functional". I can live with that temporarily.

output of top:

Code: [Select]

Mem: 288244K used, 673224K free, 39704K shrd, 4284K buff, 176652K cached
CPU: 23.8% usr 37.2% sys  0.0% nic 25.9% idle  6.3% io  0.0% irq  6.6% sirq
Load average: 1.09 0.77 0.35 5/186 4940
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 4937  4905 tc       D     3128  0.3   0 47.0 ./stress15
 4260     2 root     RW       0  0.0   0 11.0 [cifsd]
   38     2 root     IW       0  0.0   0  3.0 [kworker/u4:1-ev]
   52     2 root     IW       0  0.0   0  1.9 [kworker/u4:3-ev]
   12     2 root     IW       0  0.0   0  1.5 [kworker/u4:0-ev]
   15     2 root     IW       0  0.0   0  1.5 [rcu_sched]
 4875     1 root     S     8172  0.8   0  0.4 x0vncserver -PasswordFile=/home/tc/.vnc/passwd
   14     2 root     SW       0  0.0   0  0.4 [ksoftirqd/0]
 4904  4901 tc       S     7432  0.7   0  0.2 sshd: tc@pts/0
 4938  4905 tc       R     3600  0.3   0  0.2 top

[Stefann]

well.... NO crash after 16hrs

configuration:
- TC17.1 vmlinuz & core.gz
- TC15 applications
- NOT running my home controls application
- RUNNING stress program
- also running some standard apps: vnc-server, ssh, samba, apache

previously 5x crash within 9hrs
- TC17
- RUNNING my home controls application
- RUNNING stress program
- also running some standard apps: vnc-server, ssh, samba, apache

So:
- either the lower load from not running application still makes it crash but after "more than 16hrs"
- or the application has something that triggers the crash

To be 100% sure on the 1st option I should run like this for few days. That however is not so practical as my application controls some essential things in my home

I now started:
configuration:
- TC17.1 vmlinuz & core.gz
- TC15 applications
- RUNNING my home controls application
  - WITHOUT http calls
  - WITHOUT x10-pheripheral being connected to usb and using libusb 1.0 library calls
  - WITH 1wire-pheripheral being connected to usb and using libusb 2.0 library calls
- RUNNING stress program
- also running some standard apps: vnc-server, ssh, samba, apache

The reason I try my luck here is:
- crash frequency went up significantly (from few days to less than 9 hrs) when adding network activity to the stress program. This made me suspect it's something with network.
- 2 month ago I had a crash after 5weeks without crash. After that I modified my program to no longer use the 1read/second usb-serial connection (as that was the suspected rootcasue) and started using 1call/second http-calls to a different interface to get the data I needed. Instead of reducing crashes that increased the amount of crashes.
- the http-function in my application is something I wrote in 2009 from scratch using socket calls. That part is definitely not completely robust. In case of failed calls the application could keep using invalide filedescriptors.

So...
By running my application without http calls I certainly test whether libusb calls cause the crash. But apart from the libusb calls my application does nothing more spectacular than opening/reading/writing/closing files and basic program-flow.
>> if this does not crash that hints towards the http-calls (or the libusb1.0 calls, that would be a next test).

I keep it running like this for a day and see where that brings me.

Note:
IF it appears that my http handling is causing the crashes, than this is certainly caused by "not adequate robustness of my application", I already checked the code and this is factually the case. I see some cases in which a failed socket-call can later result in socket call with invalide filedescriptior.
However.....
That should still not cause linux to crash, it could crash my application but the OS should be robust to this.
It's "kind of probable" that this is the rootcasue. I wrote this component in 2009 when I was using DSL (Damn Small Linux). At that time I had little other options for doing http calls and my application was very minimal, just calling 1 local http-connected device from time to time.
These days raw socket calls are quite uncommon in hobby programs, it would be more normal to use curl calls. Applications that use raw socket calls are likely professional and have good error handling.
So... it is at least somewhat imaginable that some OS-vulnerability on dirty socket use could have gone unnoticed during linux release testing.

[Stefann]

Sooo..... 23hr without crash!
- TC17.1 vmlinuz & core.gz
- TC15 applications
- RUNNING my home controls application
  - WITHOUT http calls
  - WITHOUT x10-pheripheral being connected to usb and using libusb 1.0 library calls
  - WITH 1wire-pheripheral being connected to usb and using libusb 2.0 library calls
- RUNNING stress program
- also running some standard apps: vnc-server, ssh, samba, apache

previously 5x "crash within 9hrs"
- TC17.1
- RUNNING my home controls application including http & libusb calls
- RUNNING stress program
- also running some standard apps: vnc-server, ssh, samba, apache


This brings big suspicion that the http calls in my application are the problem.
As said, the http calls from my application are written in a "not so clean part of code" using raw socket calls. It's a very old part of code that I wrote in 2009. If I would do that today I would likely use curl.
A further hint that the socket calls are the problem is that the crash-frequency went from multiple days to less than 9hrs when I added network-writes to my stress program (code at bottom). I'm writing to my network connected 2nd computer that I mounted using gifs:

Code: [Select]

sudo mount -t cifs -o vers=1.0,username=tc //192.168.2.203/root /remoteSo, NOT using nfs. That means that the network actions also use tcp sockets. (note, there is no wisdom behind this choice, it was just the first option I found on google when I did this).

Going forward....
Yesterday I hardened the http module of my application using my 2nd system. I made sure that all socket related calls now have proper error handling. I also included some syslog messages for every "added error-handling". This way I will see what errornous behavior previously went uncatched.
I did already testrun this on my 2nd system and I did sure see socket-errors that previously went uncatched.

I just rebooted basically full blown configuration:
- TC17.1 vanilla from TC download area including applications
- full-blown RUNNING my home controls application, including hardened http calls and libusb calls freshly compiled under TC17.1
- RUNNING stress program, freshly compiled under TC17.1
- also running some standard apps: vnc-server, ssh, samba, apache
Let's see whether this can run without crash!

I skipped a few steps in gradually working from yesterday configuration to current situation. Reason:
- running TC17.1 kernel with TC15 application did not seem to bring more robustness, not much reason to run like that
- disabling the X10 interface I kind of judge "less likely" to have added much to the robustness. It uses the libusb1.0 interface (by libusb compatibility library) but as the 1wire interface using the libusb2.0 library is functioning fine I donot see much danger.
If it starts crashing I will work backwards.

snippet of top to show load:

Code: [Select]

Mem: 476748K used, 484720K free, 42048K shrd, 4116K buff, 291860K cached
CPU: 24.9% usr 46.9% sys  0.0% nic 12.1% idle  1.6% io  0.0% irq 14.2% sirq
Load average: 1.91 1.93 1.98 4/207 4353
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 4084  3882 tc       R     3128  0.3   0 42.2 ./stress17
 4176     1 tc       R    11584  1.2   0 21.9 /krubo/work/krubo17 /krubo/work/1wire.def
 3360     2 root     SW       0  0.0   0  9.8 [cifsd]
 4331     2 root     IW       0  0.0   0  3.1 [kworker/u4:0-ev]
 4234     2 root     IW       0  0.0   0  2.7 [kworker/u4:3-ev]
   15     2 root     IW       0  0.0   0  1.6 [rcu_sched]
 4352  3882 tc       R     3600  0.3   0  0.6 top
 3382     1 root     S    54584  5.6   0  0.4 /usr/local/sbin/rsyslogd
   14     2 root     SW       0  0.0   0  0.4 [ksoftirqd/0]
 4009     1 root     S     8192  0.8   0  0.2 x0vncserver -PasswordFile=/home/tc/.vnc/passwd
 3877  3785 tc       S     7972  0.8   0  0.2 sshd-session: tc@pts/0
 3919  3874 tc       S     405m 43.0   0  0.0 /usr/local/sbin/httpd -k start
 3921  3874 tc       S     404m 42.9   0  0.0 /usr/local/sbin/httpd -k start
 4087  3874 tc       S     391m 41.6   0  0.0 /usr/local/sbin/httpd -k start
 3937  3874 tc       S     384m 40.9   0  0.0 /usr/local/sbin/httpd -k start
 3874     1 root     S     162m 17.2   0  0.0 /usr/local/sbin/httpd -k start
 3878     1 root     S    44976  4.6   0  0.0 Xvesa -br -screen 1024x768x32 -shadow -2button -mouse /dev/input/mice
 4142  3687 root     S    19208  1.9   0  0.0 /usr/local/sbin/smbd -D
 3701  3687 root     S    18868  1.9   0  0.0 /usr/local/sbin/smbd -D
 3687     1 root     S    18812  1.9   0  0.0 /usr/local/sbin/smbd -D
 4031     1 tc       S     9852  1.0   0  0.0 wbar
 3914     1 tc       S     9628  1.0   0  0.0 flwm
code of stress program:

Code: [Select]

int Stress()
{ int i;
  float d1[200], d2[200];
  FILE *fd;
 
  while (1)
  { fd = fopen("/remote/home/tc/stress.txt", "w");
    if (fd)
    { for (i=0; i<100; i++)
       d1[i] = 1.111*i;
      for (i=0; i<100; i++)
        d2[i] = d1[i]/ 1.111;
      for (i=0; i<100; i++)
        d1[i] = d2[i] * 1.111;
      for (i=0; i<100; i++)
        fprintf(fd, "dum %d\n",d1[i]);
      fclose(fd);
     }
  }
  return 1;
}

[Stefann]

Unfortunately I got an other crash yesterday.
Crash around 23:00
That still "kind of fits the theory" of socket related error:
- "somewhere between 22:00 and 23:00" my solar energy server has a 1 minute "automatic service" during which it is unresponsive.
- When I tested the "improved http module" on my 2nd system a day earlier I could see that the http-module had zero errors during day but a big stream of errors during that 1 minute non-responsive time.

So... IF the crash is related to this downtime it may still very well be related to socket issue.

My improved application had proper error handling included on all sockets calls.... except for 1.... the connect() call.
Reason being that I call "connect()" in nonblocking mode and than it always throws an error. Yesterday I did not know how to handle that.
So.... its is still a valid theory that connect() throws errors when server is non-responsive and my system crashes for reason of not handling those errors.

Today I learned how to handle connect() errors:
- ignore "in progress" error
- but abort connection on all other errors
- check connection by "getpeername()" before using
I added that.

Just started again, full blown program under TC17.1:
- TC17.1 vanilla from TC download area including applications
- full-blown RUNNING my home controls application, including hardened http calls and libusb calls freshly compiled under TC17.1
- RUNNING stress program, freshly compiled under TC17.1
- also running some standard apps: vnc-server, ssh, samba, apache

My hopes are not super high, at 1st try I got a crash the moment I started the stress program.
After that I rerouted the syslog for socket related errors to the network connected 2nd computer. By doing that I have latest messages in case I get a next crash
On every catched error from socket operations I write a syslog message

And I routed all kernel logging to console (physical)

So...
- on 2nd start it survived the starting of the stress program
- lets see how it runs
- lets see whether it survives the 23:00-ish network challenge
- lets see whether I have useful logging IF it crashes

Note,
One can argue that I better move to curl instead of attempting a home-brew http module.
-> That is an entirely correct thought.
However... "moving to something like curl" is quite some work.
- I need a nonblocking implementation OR I need to include threading. I'm pretty sure I will be able to figure out how to do that BUT I'm also pretty sure that such is quite complicated and takes time.
- At this moment it's still non-proven that this is the issue. Potentially I would do a lot of work "without getting a better live".
- I'm not 100% sure such path would even help. "good weather behavior" is 100% OK for my application. I now have proper error handling around every system call. curl will use the same system calls. "why would it than behave better"?
- anyway... the thought is valid. Maybe at some point.

[gadget42]

again, thanks for keeping us updated!!!

[Stefann]

again, thanks for keeping us updated!!!

Thanks!
An occasional "sign of life" from readers is appreciated. I was already wondering whether my writings are appreciated.

I got multiple crashes since last post.
Also the connection to my network connected drive became unstable (multiple minutes to respond)
I ended up removing the getpeername() call. Since than things are at least running.

All in all I still see a significant likelihood that "socket instability" is the rootcause. Behavior is changing wildly when I'm poking with that.
At least the "connect()" now has proper error-handling. I'm pretty sure that call got uncatched errors until yesterday.
In principle linux should not completely crash, only the application should, but on the other hand "running an application that is supposed to be super robust while not catching communication errors" is bad practice.

We'll see how this continues

[Stefann]

Small update, yesterday evening an other crash.
Today I will not have much time to work on this.

I however think I’m getting into a line of thought:
- I see clear crashing increase when massively using the network adapter with my stress-program that brought network traffic from <1% to >80%.
- I now see consistent crashes around 22:30 which coincides with my solar energy inverter going into a 1 minute service-routine that makes it unreachable.
- Problems started when I updated from TC15 to TC17. This suggests a Linux error but alternatively the network card (network card firmware) could be buggy. Probably never triggered under TC15 but getting triggered under TC17.

With this line of thought there is plenty to test:
- instead of a daily wait for the 22:30 service loop, will things also crash if I simply pull the cable?
- does TC15 crash on this 1 minute unreadable server IF I run the stress program as well (force >80% network traffic)?
- isolate the exact network call that crashes by adding syslog lines around them.

And also….
I now see consistent crashing around 22:30 under 80% network traffic. I however included proper error handling in my application. It may well function OK if I not run the stress program.

So….
Plenty to test.
I will keep you posted but it may take a few days.

[Stefann]

Well.... I'm running without crash for 24hrs now:
- TC17.1 vanilla from download area
- running full blown application
- running stress program that brings cpu-load to 80% and networkload  (Nic) to 90%

Too often I have cried victory only to find out that things got into a crash after multipel days, or,... after 5 weeks last august.
But.....
Before updating my application I got 5 crashes in a row with <9hr runtime.
So... It sure looks promising!

To get here I changed my application:
1/
more strict control of data structures. extra checks to avoid writing more data into strings than the string size allows
2/
proper handling of all network calls that return an error
3/
no longer use "select()" to check whether socket is ready for writing but "just write" and check error code from write().
Note: I basically have a state machine that calls the network module every 200ms. After connect() I return to the main program. After 200ms the module gets called again and select() succeeds 99.99999% of the time towards "ready to write".
By removing the select() I also remove a network call with potential erroneous behavior.
4/
replaced write() bij send(..., ..., ..., MSG_NOSIGNAL)
If I understand well, in case of lost connection write() can trigger a SIGPIPE that could terminate the program while send(... MSG_NOSIGNAL) only generates an errocode on that.
5/
For all of above exceptions: add a syslog command to write to logging "which exception got triggered"
The purpose of this: "in case the program does now NOT crash I can see which exceptions naturally happen during execution of my program and conclude that improper handling of them could have caused the crashing before"

=========
So..... It has run flawless under stress conditions for 24 hr...
Remarkable enough: the added syslog hardly gives logging.
- zero data-size violations (I would have expected application crash on that anyways so its not really surprising)
- also no network errors as long as host is not in service interval (I had expected much more)

The only errors I see (and only during that service interval):
send(): 111 connection refused
read(): 104 SSL_ERROR_SYSCALL, socket reset during call

So..... I tend to conclude:
- either the (now removed) select() call was the trouble maker
- or the replacement of write() by send() fixed things
(or I will see crashes in the coming days)

Furthermore,... I found this article: https://medium.com/@sagarmadala/how-write-system-call-works-d133921dce32
This is about "write() to disk" not internet socket.
But... the commonality is:
- dated march 2026, that is about the same linux version
- kernel crash on write()
- in this case diagnosed as "write() returns no error when writing to disconnected disk and kernel crashes when buffer overflows". The author concludes that he should have used fsynk() more often but personally I would have been understanding towards application crashes but not really to kernel crashes.

Note... my program flow is roughly (many details not included):

Before:
- create and configure socket in non blocking mode
- connect() without looking at error (in nonblocking mode it always returns with not connected error)
- return and get recalled after 200ms
- select() to check socket is ready to write. If not do few return/200ms recalls until maximally 15 seconds. in practice this always succeeds.
- write(), abort in case return <0; no check on return==0 which is not correct because 0 is unspecified behavior
- return and get recalled after 200ms
- read(), abort in case return <0
- close connection

After:
- create and configure socket in non blocking mode
- connect(), abort on any error other than "in progress" which is the expected outcome in nonblocking mode
- return and get recalled after 200ms
- send(), return/recall after 200ms in case EAGAIN or EWOULDBLOCK until max 15seconds; abort in case return <=0; (so ==0 check included now)
- return and get recalled after 200ms
- read(), abort in case return <0
- close connection

Conclusion, Next steps:
- Given all observations I feel the removal of select() is less likely to have attributed to the better performance
- High suspect towards the write()
- I will keep running in this configuration for some additional days. I will probably be less strict on "un interrupted". There are lot of things I want to modify to the functional part my application. I will probably stop the application and do updates. But keep running the application including stress conditions when I'm not programming. Over time it will built enough run-hours to get confident.
- Once I feel confident that configuration is stable I will "backwards troubleshoot" by "step-by-step" going back to failing configuration. 1st test will be to bring back the write() call.