Well....
I'm running without crash for 24hrs now:- TC17.1 vanilla from download area
- running full blown application
- running stress program that brings cpu-load to 80% and networkload (Nic) to 90%
Too often I have cried victory only to find out that things got into a crash after multipel days, or,... after 5 weeks last august.
But.....
Before updating my application I got 5 crashes in a row with <9hr runtime.
So...
It sure looks promising!To get here I changed my application:
1/
more strict control of data structures. extra checks to avoid writing more data into strings than the string size allows
2/
proper handling of all network calls that return an error
3/
no longer use "select()" to check whether socket is ready for writing but "just write" and check error code from write().
Note: I basically have a state machine that calls the network module every 200ms. After connect() I return to the main program. After 200ms the module gets called again and select() succeeds 99.99999% of the time towards "ready to write".
By removing the select() I also remove a network call with potential erroneous behavior.4/
replaced write() bij send(..., ..., ..., MSG_NOSIGNAL)
If I understand well, in case of lost connection write() can trigger a SIGPIPE that could terminate the program while send(... MSG_NOSIGNAL) only generates an errocode on that.5/
For all of above exceptions: add a syslog command to write to logging "which exception got triggered"
The purpose of this: "in case the program does now NOT crash I can see which exceptions naturally happen during execution of my program and conclude that improper handling of them could have caused the crashing before"
=========
So..... It has run flawless under stress conditions for 24 hr...
Remarkable enough: the added syslog hardly gives logging.
- zero data-size violations (I would have expected application crash on that anyways so its not really surprising)
- also no network errors as long as host is not in service interval (I had expected much more)
The only errors I see (and only during that service interval):
send(): 111 connection refused
read(): 104 SSL_ERROR_SYSCALL, socket reset during call
So..... I tend to conclude:
- either the (now removed) select() call was the trouble maker
- or the replacement of write() by send() fixed things
(or I will see crashes in the coming days)
Furthermore,... I found this article:
https://medium.com/@sagarmadala/how-write-system-call-works-d133921dce32This is about "write() to disk" not internet socket.
But... the commonality is:
- dated march 2026, that is about the same linux version
- kernel crash on write()
- in this case diagnosed as "write() returns no error when writing to disconnected disk and kernel crashes when buffer overflows". The author concludes that he should have used fsynk() more often but personally I would have been understanding towards application crashes but not really to kernel crashes.
Note... my program flow is roughly (many details not included):
Before:
- create and configure socket in non blocking mode
- connect() without looking at error (in nonblocking mode it always returns with not connected error)
- return and get recalled after 200ms
- select() to check socket is ready to write. If not do few return/200ms recalls until maximally 15 seconds. in practice this always succeeds.
- write(), abort in case return <0; no check on return==0 which is not correct because 0 is unspecified behavior
- return and get recalled after 200ms
- read(), abort in case return <0
- close connection
After:
- create and configure socket in non blocking mode
- connect(), abort on any error other than "in progress" which is the expected outcome in nonblocking mode
- return and get recalled after 200ms
- send(), return/recall after 200ms in case EAGAIN or EWOULDBLOCK until max 15seconds; abort in case return <=0; (so ==0 check included now)
- return and get recalled after 200ms
- read(), abort in case return <0
- close connection
Conclusion, Next steps:- Given all observations I feel the removal of select() is less likely to have attributed to the better performance
- High suspect towards the write()
- I will keep running in this configuration for some additional days. I will probably be less strict on "un interrupted". There are lot of things I want to modify to the functional part my application. I will probably stop the application and do updates. But keep running the application including stress conditions when I'm not programming. Over time it will built enough run-hours to get confident.
- Once I feel confident that configuration is stable I will "backwards troubleshoot" by "step-by-step" going back to failing configuration. 1st test will be to bring back the write() call.
Welcome
FAQ
Downloads
Wiki
