Find cause for crashes on my VIA EDEN 500MHz 1core 32bit cpu

[Stefann]

back from vacation and an update.
Summary:
- the good thing: no crash over 8 days
- the bad thing: I made a mistake with configurations... It's now unclear again.

In detail:
I reverted the Realtek 8139 driver patch ands did rebuilt bzImage (vmlinuz)
However... I did not realize that the driver is not part of bzImage but part of modules.gz
So... today I found out that bzImage (vmlinuz) was never updated, I effectively tested with the one I built on august-22 as described this post: https://forum.tinycorelinux.net/index.php?topic=28078.msg183601#msg183601
Basically TC17.1 source from TC site:

Code: [Select]

-rw-r--r--    1 tc       staff    159566696 Aug 21 16:54 linux-6.18.35-patched.tar.xzCustom built vmlinuz with 1 modification: "include serial usb driver"

I need to recheck things a bit to be absolutely sure but it looks like I now have:

TC17.1 custom built vmlinuz:

Code: [Select]

-rw-r--r--    1 tc       staff      6128128 Aug 22 14:19 bzImage- crashing within 1 or 2 days when using usb-serial and/or select() statements (tested august-22)
- stable over 8+ days after NOT using usb-serial and after code simplification to not use select() and have write() catch all errors

TC17.1 downloaded vmlinuz:

Code: [Select]

-rw-rw-r--    1 tc       staff      6115840 Aug 23 14:21 vmlinuz171C- crashing within 2 hours after code simplification to not use select() but have write() catch all errors, even without using usb-serial

Damn... "why is this all so vague....."

Anyway... not sure how to proceed.
I think I will:
- keep current 6.1.35 built directory as that "apparently" has something good.
- redownload linux-6.18.35-patched.tar.xz and built vmlinuz without any modification
- test whether that new vmlinuz is stable as well

either its stable --> which make one wonder why the downloaded vmlinuz is not
or it isn't --> which urges  to search for differences with the stable built

To keep things simple I will test without usb-serial and without select() but only write() that now has proper error-handling.

There is actually still a possibility the issue is in the Realtek 8139 driver. It may be a rare race-condition that gets triggered by minute timing differences resulting form "quite unrelated built differences". On the one hand thats a bit unlikely, but on the other hand the whole thing is quite mysterious.

[Stefann]

4days since I last posted, time for an update. I'll keep it summarized:

- no crash with customer built kernel with intended network driver fix over 8 days, but the driver fix was not in the kernel (its part of modules) so no fix
- same kernel did crash in 18 hrs when I used "ignore pipe event" in application
- so: although crash frequency varies between minutes and weeks, it seems all configurations I have tested ultimately fail.

Conclusions I tend to get:
- the crash mechanism is extremely sensitive to minute config changes. It's very likely a race condition.
- As my 2nd system is also running TC17 without any crash over  past 8 month I generally I feel TC17 is absolutely fine also on my ancient VIA EDEN processors
- I expect some very specific HW configuration to be the rootcause
- And on that aspect the network driver is my highest suspect candidate: it's an old Realtek chip using old 8139too.c driver

I'm now using chatGPT to help me. On "logic of doing things" I often have a different opinion, but its actually a great help for reason that its knowledge about syntax is very superior to mine and it has access to all linux sources and can quickly find fix-history.

At this moment I'm running a self built kernel including an updated 8139too.c driver that now includes lots of pr_emerg() logging. I have also installed persistent ram that hopefully survives a crash. That part is not fully working yet though.

An other path I'm exploring is to install a watchdog such that at least my system reboots on crash. I found watchdog-6.18.35-tynicore.tcz that should do that.

[Stefann]

small update, I think I get to some sort of conclusion:

Summary:
- Doing non-logic-changing modifications to the kernel changes Like custom built instead of download built or adding diagnostics to a custom built can change the crash behavior from minutes to weeks or verse versa.
- This makes me conclude the driving mechanism is a very rare race condition.
- I donot think this is in the "most commonly used part of the kernel" because "although rare", frequency is way to high to be a common problem.
- For that reason this is more likely to be a driver-specific problem and most likely in a "not so widely used driver".
- My biggest suspect is the 8139too.c Realtek network driver. This driver is used for some ancient 8139 network chips. More modern 8139 network chips use the 8139cp.c driver which seems to be coded a bit more robust. My [10ec:8139 rev 10] chip however cannot handle that driver.

My current configuration:
- custom built TC171 kernel with persistant ram configuration (otherwise standard).
- self built modified 8139too.c driver as kernel extension. Only diagnostics is added, logic has not been changed so this should crash just as hard as before. Diagnostics is added to a driver level tx timeout function. This function is almost never executed, I have not seen a single execution of this function over past 44 hours. So.. even if this function is as bad as "always crash" it could explain the rare crash observations.
- I have confirmation the driver is operational. I included a message during initialization to demonstrate that.
- my home application is using write() and using "ignore pipe signal" to avoid that the application crashes on pipe errors from write(). Standard write behavior is to send pipe signal to terminate the application in case of lost pipe but linux should not crash than. This code has been crashing within 2 hours multiple times with other kernel configurations. So... although the code is really correct, it can crash the kernel.
- dedicated stress program to drive cpu load to about 80% and network load to about 90%
- watchdog.tcz and wdt bootcode to reboot linux in case of crash. This actually is a really really good addition. As long as crashing is not so often it is inconvenient but not fatal. As this is controlling my full house as home automation "having the system in lockup for multiple hours" is however very bad. It messes my heating, solar power, battery and car charging.
- save of messages from persistent ram at every reboot

I'm now running without crash for 44hours.
I have no illusion that I have somehow magically fixed things "by not adding any fix" but at least:
- I'm in a workable condition due to the watchdog timer
- I have added diagnostics to the most suspicious driver code
- I have persistant ram that ensures I can check that logging after a crash

watchdog timer and persistant ram are tested and work.

So.....
Don't expect quick followups
But I'm "in the game" again.
The real good thing... I really feel I can now safely abandon TC15 and with that dare to use new functionality, new tcz from TC17.

itself 

rokespecially a driver for so
can made to crashes anywhere between few minutes and few weeks by doing non-logic-changes like: "do custom

[Stefann]

small update for those still interested.

System was running for about 4 days and than got an other crash.
The assumption was that it would do a warm reboot triggered by the hardware watchdog and than I would have diagnostics in the persistant ram that I configured for that.
Sadly "no"... no reboot by the watchdog.

After some troubleshooting I found out that the hardware watchdog was not running. for some reason wdt boot option is not recognized. --> I posted that in bug-topic: https://forum.tinycorelinux.net/index.php?topic=28312.0

I fixed that by configuring the watchdog with some commands in boot local.sh.
Tested and seems to work (but cannot test with rock solid hard lockup).
So... an other run....
I sure expect an other crash in few days and than hope that the hardware watchdog does a warm reboot and I catch diagnostics.

[gadget42]

@Stefann, thanks for continuing the threads and updates. right now the thread shows 2770 views so there are some people following(and no doubt a bunch of bots).

[Stefann]

@gadget42, thanks! It really helps to get some feedback from time to time.

small update...
I was rather busy last week with other things. But.... system did run so I have information:
2x crash over past week > this is not surprising. Crashes occurred with frequency similar to earlier observations on used conditions.

In both cases: hardware watchdog recovered system >> VERY GOOD! My application has a pretty good recovery mechanism so..... Although I lose some logging, my house (being controlled by the home control application on this system) is not suffering from the crashes. A major step forward!

Persistent ram is functional for console messages and dmesg output. These are successfully tested to survive watchdog reboots. However... sadly no messages related to the crash.

There is some sadness in that:
- I have included lot of pr_emerg() functions in 8139too.c network driver around the timeout function that was patched between TC15 and TC17. But although I tested that these functions "are functional", I do not see anything from them in the persistant ram of console & dmesg.
Either the lockup happened before handling from pr_emerg to persistant ram was ready OR the fault mechanisme is not related to 8139too.c

I now try to activate CONFIG_FTRACE=y and see whether that brings something.

Note,
I guess step by step I'm getting pretty advanced. , I'm actually doing debugging on linux kernel . I'm successful in custom kernel built and custom modules built. That means that my toolbox is "quite well filled".
With that said... "I can certainly use a golden tip to catch logging before the lockup".

[gadget42]

you're not alone in the "strange events" department.

recent experience with a newer(circa2022) fancy space heater(even came with a handy remote control) ($150ish range)

naturally it goes without saying that heaters shouldn't be used unattended...

anywho, this crazy heater would sometimes "play dead" until it was unplugged and plugged back in. other times, as soon as it was plugged in it would start the heating elements without running the fan(very bad). and the tip-over switch didn't work either. this thing was so bad that when the owner/purchaser tried to return it to the store they immediately gave a complete refund and refused to accept the heater at all(they didn't come right out and say to destroy it because then that would essentially be admitting some sort of fault/liability/etc).

it has been terminated, permanently. one wonders how many poorly designed devices such as these units have resulted in property losses, injuries, and deaths. sigh.