Find cause for crashes on my VIA EDEN 500MHz 1core 32bit cpu

[Stefann]

back from vacation and an update.
Summary:
- the good thing: no crash over 8 days
- the bad thing: I made a mistake with configurations... It's now unclear again.

In detail:
I reverted the Realtek 8139 driver patch ands did rebuilt bzImage (vmlinuz)
However... I did not realize that the driver is not part of bzImage but part of modules.gz
So... today I found out that bzImage (vmlinuz) was never updated, I effectively tested with the one I built on august-22 as described this post: https://forum.tinycorelinux.net/index.php?topic=28078.msg183601#msg183601
Basically TC17.1 source from TC site:

Code: [Select]

-rw-r--r--    1 tc       staff    159566696 Aug 21 16:54 linux-6.18.35-patched.tar.xzCustom built vmlinuz with 1 modification: "include serial usb driver"

I need to recheck things a bit to be absolutely sure but it looks like I now have:

TC17.1 custom built vmlinuz:

Code: [Select]

-rw-r--r--    1 tc       staff      6128128 Aug 22 14:19 bzImage- crashing within 1 or 2 days when using usb-serial and/or select() statements (tested august-22)
- stable over 8+ days after NOT using usb-serial and after code simplification to not use select() and have write() catch all errors

TC17.1 downloaded vmlinuz:

Code: [Select]

-rw-rw-r--    1 tc       staff      6115840 Aug 23 14:21 vmlinuz171C- crashing within 2 hours after code simplification to not use select() but have write() catch all errors, even without using usb-serial

Damn... "why is this all so vague....."

Anyway... not sure how to proceed.
I think I will:
- keep current 6.1.35 built directory as that "apparently" has something good.
- redownload linux-6.18.35-patched.tar.xz and built vmlinuz without any modification
- test whether that new vmlinuz is stable as well

either its stable --> which make one wonder why the downloaded vmlinuz is not
or it isn't --> which urges  to search for differences with the stable built

To keep things simple I will test without usb-serial and without select() but only write() that now has proper error-handling.

There is actually still a possibility the issue is in the Realtek 8139 driver. It may be a rare race-condition that gets triggered by minute timing differences resulting form "quite unrelated built differences". On the one hand thats a bit unlikely, but on the other hand the whole thing is quite mysterious.

[Stefann]

4days since I last posted, time for an update. I'll keep it summarized:

- no crash with customer built kernel with intended network driver fix over 8 days, but the driver fix was not in the kernel (its part of modules) so no fix
- same kernel did crash in 18 hrs when I used "ignore pipe event" in application
- so: although crash frequency varies between minutes and weeks, it seems all configurations I have tested ultimately fail.

Conclusions I tend to get:
- the crash mechanism is extremely sensitive to minute config changes. It's very likely a race condition.
- As my 2nd system is also running TC17 without any crash over  past 8 month I generally I feel TC17 is absolutely fine also on my ancient VIA EDEN processors
- I expect some very specific HW configuration to be the rootcause
- And on that aspect the network driver is my highest suspect candidate: it's an old Realtek chip using old 8139too.c driver

I'm now using chatGPT to help me. On "logic of doing things" I often have a different opinion, but its actually a great help for reason that its knowledge about syntax is very superior to mine and it has access to all linux sources and can quickly find fix-history.

At this moment I'm running a self built kernel including an updated 8139too.c driver that now includes lots of pr_emerg() logging. I have also installed persistent ram that hopefully survives a crash. That part is not fully working yet though.

An other path I'm exploring is to install a watchdog such that at least my system reboots on crash. I found watchdog-6.18.35-tynicore.tcz that should do that.

[Stefann]

small update, I think I get to some sort of conclusion:

Summary:
- Doing non-logic-changing modifications to the kernel changes Like custom built instead of download built or adding diagnostics to a custom built can change the crash behavior from minutes to weeks or verse versa.
- This makes me conclude the driving mechanism is a very rare race condition.
- I donot think this is in the "most commonly used part of the kernel" because "although rare", frequency is way to high to be a common problem.
- For that reason this is more likely to be a driver-specific problem and most likely in a "not so widely used driver".
- My biggest suspect is the 8139too.c Realtek network driver. This driver is used for some ancient 8139 network chips. More modern 8139 network chips use the 8139cp.c driver which seems to be coded a bit more robust. My [10ec:8139 rev 10] chip however cannot handle that driver.

My current configuration:
- custom built TC171 kernel with persistant ram configuration (otherwise standard).
- self built modified 8139too.c driver as kernel extension. Only diagnostics is added, logic has not been changed so this should crash just as hard as before. Diagnostics is added to a driver level tx timeout function. This function is almost never executed, I have not seen a single execution of this function over past 44 hours. So.. even if this function is as bad as "always crash" it could explain the rare crash observations.
- I have confirmation the driver is operational. I included a message during initialization to demonstrate that.
- my home application is using write() and using "ignore pipe signal" to avoid that the application crashes on pipe errors from write(). Standard write behavior is to send pipe signal to terminate the application in case of lost pipe but linux should not crash than. This code has been crashing within 2 hours multiple times with other kernel configurations. So... although the code is really correct, it can crash the kernel.
- dedicated stress program to drive cpu load to about 80% and network load to about 90%
- watchdog.tcz and wdt bootcode to reboot linux in case of crash. This actually is a really really good addition. As long as crashing is not so often it is inconvenient but not fatal. As this is controlling my full house as home automation "having the system in lockup for multiple hours" is however very bad. It messes my heating, solar power, battery and car charging.
- save of messages from persistent ram at every reboot

I'm now running without crash for 44hours.
I have no illusion that I have somehow magically fixed things "by not adding any fix" but at least:
- I'm in a workable condition due to the watchdog timer
- I have added diagnostics to the most suspicious driver code
- I have persistant ram that ensures I can check that logging after a crash

watchdog timer and persistant ram are tested and work.

So.....
Don't expect quick followups
But I'm "in the game" again.
The real good thing... I really feel I can now safely abandon TC15 and with that dare to use new functionality, new tcz from TC17.

itself 

rokespecially a driver for so
can made to crashes anywhere between few minutes and few weeks by doing non-logic-changes like: "do custom

[Stefann]

small update for those still interested.

System was running for about 4 days and than got an other crash.
The assumption was that it would do a warm reboot triggered by the hardware watchdog and than I would have diagnostics in the persistant ram that I configured for that.
Sadly "no"... no reboot by the watchdog.

After some troubleshooting I found out that the hardware watchdog was not running. for some reason wdt boot option is not recognized. --> I posted that in bug-topic: https://forum.tinycorelinux.net/index.php?topic=28312.0

I fixed that by configuring the watchdog with some commands in boot local.sh.
Tested and seems to work (but cannot test with rock solid hard lockup).
So... an other run....
I sure expect an other crash in few days and than hope that the hardware watchdog does a warm reboot and I catch diagnostics.

[gadget42]

@Stefann, thanks for continuing the threads and updates. right now the thread shows 2770 views so there are some people following(and no doubt a bunch of bots).

[Stefann]

@gadget42, thanks! It really helps to get some feedback from time to time.

small update...
I was rather busy last week with other things. But.... system did run so I have information:
2x crash over past week > this is not surprising. Crashes occurred with frequency similar to earlier observations on used conditions.

In both cases: hardware watchdog recovered system >> VERY GOOD! My application has a pretty good recovery mechanism so..... Although I lose some logging, my house (being controlled by the home control application on this system) is not suffering from the crashes. A major step forward!

Persistent ram is functional for console messages and dmesg output. These are successfully tested to survive watchdog reboots. However... sadly no messages related to the crash.

There is some sadness in that:
- I have included lot of pr_emerg() functions in 8139too.c network driver around the timeout function that was patched between TC15 and TC17. But although I tested that these functions "are functional", I do not see anything from them in the persistant ram of console & dmesg.
Either the lockup happened before handling from pr_emerg to persistant ram was ready OR the fault mechanisme is not related to 8139too.c

I now try to activate CONFIG_FTRACE=y and see whether that brings something.

Note,
I guess step by step I'm getting pretty advanced. , I'm actually doing debugging on linux kernel . I'm successful in custom kernel built and custom modules built. That means that my toolbox is "quite well filled".
With that said... "I can certainly use a golden tip to catch logging before the lockup".

[gadget42]

you're not alone in the "strange events" department.

recent experience with a newer(circa2022) fancy space heater(even came with a handy remote control) ($150ish range)

naturally it goes without saying that heaters shouldn't be used unattended...

anywho, this crazy heater would sometimes "play dead" until it was unplugged and plugged back in. other times, as soon as it was plugged in it would start the heating elements without running the fan(very bad). and the tip-over switch didn't work either. this thing was so bad that when the owner/purchaser tried to return it to the store they immediately gave a complete refund and refused to accept the heater at all(they didn't come right out and say to destroy it because then that would essentially be admitting some sort of fault/liability/etc).

it has been terminated, permanently. one wonders how many poorly designed devices such as these units have resulted in property losses, injuries, and deaths. sigh.

[Stefann]

@gadget42: thanks for the fun comment here

Update:
In short:
- not really solved
- I kind of give up. I go forward without extra stress program with "lowest crash frequency" configuration to see whether that is workable.
- hardware watchdog is functional, so if this crashes only 1x/month I will live with it
- also got 2 crashes on my 2nd system while that was under heavy load building a linux kernel

In more detail:
- I have a well tested hardware watchdog that reboots my system in case of crash. That is a real good addition because although it will loose some monitor data the home control system restarts nicely without collateral damage.
- also installed ramoops that keeps dmesg and console data if that gets generated with a crash. This is also tested and indeed survives a crash with watchdog-reboot but none of the crashes generates any logging
- biggest suspect was the 8139too.c realtek network driver. That driver had a nontested update in 2025 in the timeout path. That driver is only used by very old hardware and timeouts are very rare so "it could have been a kernel bug that got unnoticed". I recompiled the driver with lots of pr_emerg() statements even before the napi_lock part but none of the crashes did show any message (while the extra message I did put in the init_function was visible). Either this is not the problem or the crash is before the message gets into ramoops. After testing multiple times I think it gets unlikely that the pr_emerg() statements get hit without reaching ramoops. I guess this is not it.
- I have been using my 2nd system (via eden 1GHz 2core 64bit system running 32bit TC17) a lot to do kernel builds. It did also crash 2x. That also makes the 8139too.c driver less suspicious (as this has a totally different network card).
- tested with "noapic" and "8139too.use_io=1" bootcode including stress test. This made the time to crash considerable longer but ultimately crashed as well.

By now I tend to conclude...
- it's not in the 8139too.c network driver but in the generic kernel. I consider that hopeful because that is more likely to get a fix with a new kernel update than something in the very specific very old driver.
- I don't think I will find the bug
- Next is to see how useful the configuration is under a combination of "least crashing findings". So... I just rebooted with "8139too.use_io=1 bootcode", hardware watchdog, ramoops, "using send() in application in stead of write()" and no additional stress program. With a little luck this will crash only rarely which would be acceptable as the hardware watchdog prevents collateral damage.

- Additionally I will setup my 2nd system also with ramoops and watchdog and have it continuously do linux builds to see whether I can crash it and maybe get some logging. It has 2 cpu's so maybe if one stalls the other can write something.
 
All in all this also answers the question that @gadget42 asked few month ago: "what do you do if you cannot find it"

I keep you guys updated but expect very infrequent updates

[Stefann]

small update.
system did run for 4.5 days without crash. At that time I had to manually reboot for unrelated reason.
It's now running again running for 1 day.

It's running:
- without additional stress test
- with hardware watchdog that reboots system in case of lockup (such that my homecontrol system keeps functioning).
- with added diagnostics to 8139too.c network driver. logic flow is unmodified.
- with persistent ram that preserves dmesg and console messages over a reboot.
- kernel is self build but apart from above diagnostics unmodified
- with kernel option: "8139too.use_io=1". This option forces the network card to communicate over an IO pin instead of over the pci-bus. The rationale is that the pci-bus is heavily driving a usb-based 1wire controller. It does 800 interrupts/second for that. By using "8139too.use_io=1" I divert networked traffic from that bus. As I already concluded that intensifying network traffic makes lockups more frequent this "could help" and not just by coincidence but "for a reason".

So... looks like much better than before.
Rootcause likely has to do with "heavy interrupt traffic over pci-bus" generating a race-condition. This may or may not have been in earlier kernels but simply never surfaced "by change".

I'm also testing my 2nd system but no conclusive data from that yet.

I hope I donot have to manually reboot again in coming days.. weeks.. Lets hope the setup can do sufficiently good.