Find cause for crashes on my VIA EDEN 500MHz 1core 32bit cpu

[Stefann]

back from vacation and an update.
Summary:
- the good thing: no crash over 8 days
- the bad thing: I made a mistake with configurations... It's now unclear again.

In detail:
I reverted the Realtek 8139 driver patch ands did rebuilt bzImage (vmlinuz)
However... I did not realize that the driver is not part of bzImage but part of modules.gz
So... today I found out that bzImage (vmlinuz) was never updated, I effectively tested with the one I built on august-22 as described this post: https://forum.tinycorelinux.net/index.php?topic=28078.msg183601#msg183601
Basically TC17.1 source from TC site:

Code: [Select]

-rw-r--r--    1 tc       staff    159566696 Aug 21 16:54 linux-6.18.35-patched.tar.xzCustom built vmlinuz with 1 modification: "include serial usb driver"

I need to recheck things a bit to be absolutely sure but it looks like I now have:

TC17.1 custom built vmlinuz:

Code: [Select]

-rw-r--r--    1 tc       staff      6128128 Aug 22 14:19 bzImage- crashing within 1 or 2 days when using usb-serial and/or select() statements (tested august-22)
- stable over 8+ days after NOT using usb-serial and after code simplification to not use select() and have write() catch all errors

TC17.1 downloaded vmlinuz:

Code: [Select]

-rw-rw-r--    1 tc       staff      6115840 Aug 23 14:21 vmlinuz171C- crashing within 2 hours after code simplification to not use select() but have write() catch all errors, even without using usb-serial

Damn... "why is this all so vague....."

Anyway... not sure how to proceed.
I think I will:
- keep current 6.1.35 built directory as that "apparently" has something good.
- redownload linux-6.18.35-patched.tar.xz and built vmlinuz without any modification
- test whether that new vmlinuz is stable as well

either its stable --> which make one wonder why the downloaded vmlinuz is not
or it isn't --> which urges  to search for differences with the stable built

To keep things simple I will test without usb-serial and without select() but only write() that now has proper error-handling.

There is actually still a possibility the issue is in the Realtek 8139 driver. It may be a rare race-condition that gets triggered by minute timing differences resulting form "quite unrelated built differences". On the one hand thats a bit unlikely, but on the other hand the whole thing is quite mysterious.

[Stefann]

4days since I last posted, time for an update. I'll keep it summarized:

- no crash with customer built kernel with intended network driver fix over 8 days, but the driver fix was not in the kernel (its part of modules) so no fix
- same kernel did crash in 18 hrs when I used "ignore pipe event" in application
- so: although crash frequency varies between minutes and weeks, it seems all configurations I have tested ultimately fail.

Conclusions I tend to get:
- the crash mechanism is extremely sensitive to minute config changes. It's very likely a race condition.
- As my 2nd system is also running TC17 without any crash over  past 8 month I generally I feel TC17 is absolutely fine also on my ancient VIA EDEN processors
- I expect some very specific HW configuration to be the rootcause
- And on that aspect the network driver is my highest suspect candidate: it's an old Realtek chip using old 8139too.c driver

I'm now using chatGPT to help me. On "logic of doing things" I often have a different opinion, but its actually a great help for reason that its knowledge about syntax is very superior to mine and it has access to all linux sources and can quickly find fix-history.

At this moment I'm running a self built kernel including an updated 8139too.c driver that now includes lots of pr_emerg() logging. I have also installed persistent ram that hopefully survives a crash. That part is not fully working yet though.

An other path I'm exploring is to install a watchdog such that at least my system reboots on crash. I found watchdog-6.18.35-tynicore.tcz that should do that.

[Stefann]

small update, I think I get to some sort of conclusion:

Summary:
- Doing non-logic-changing modifications to the kernel changes Like custom built instead of download built or adding diagnostics to a custom built can change the crash behavior from minutes to weeks or verse versa.
- This makes me conclude the driving mechanism is a very rare race condition.
- I donot think this is in the "most commonly used part of the kernel" because "although rare", frequency is way to high to be a common problem.
- For that reason this is more likely to be a driver-specific problem and most likely in a "not so widely used driver".
- My biggest suspect is the 8139too.c Realtek network driver. This driver is used for some ancient 8139 network chips. More modern 8139 network chips use the 8139cp.c driver which seems to be coded a bit more robust. My [10ec:8139 rev 10] chip however cannot handle that driver.

My current configuration:
- custom built TC171 kernel with persistant ram configuration (otherwise standard).
- self built modified 8139too.c driver as kernel extension. Only diagnostics is added, logic has not been changed so this should crash just as hard as before. Diagnostics is added to a driver level tx timeout function. This function is almost never executed, I have not seen a single execution of this function over past 44 hours. So.. even if this function is as bad as "always crash" it could explain the rare crash observations.
- I have confirmation the driver is operational. I included a message during initialization to demonstrate that.
- my home application is using write() and using "ignore pipe signal" to avoid that the application crashes on pipe errors from write(). Standard write behavior is to send pipe signal to terminate the application in case of lost pipe but linux should not crash than. This code has been crashing within 2 hours multiple times with other kernel configurations. So... although the code is really correct, it can crash the kernel.
- dedicated stress program to drive cpu load to about 80% and network load to about 90%
- watchdog.tcz and wdt bootcode to reboot linux in case of crash. This actually is a really really good addition. As long as crashing is not so often it is inconvenient but not fatal. As this is controlling my full house as home automation "having the system in lockup for multiple hours" is however very bad. It messes my heating, solar power, battery and car charging.
- save of messages from persistent ram at every reboot

I'm now running without crash for 44hours.
I have no illusion that I have somehow magically fixed things "by not adding any fix" but at least:
- I'm in a workable condition due to the watchdog timer
- I have added diagnostics to the most suspicious driver code
- I have persistant ram that ensures I can check that logging after a crash

watchdog timer and persistant ram are tested and work.

So.....
Don't expect quick followups
But I'm "in the game" again.
The real good thing... I really feel I can now safely abandon TC15 and with that dare to use new functionality, new tcz from TC17.

itself 

rokespecially a driver for so
can made to crashes anywhere between few minutes and few weeks by doing non-logic-changes like: "do custom