Re: Tiny Core v17.0 upgrade issues

[Stefann]

Update,
Its running as I said in previous post

- Life system running on TC17 core.gz & vmlinuz but non-upgraded .tcz applications
- disabled rsyslog
- /sbin/syslogd -O /sdb1/logging -s 1000 -b 20 -l 5
So this writes to external usb, not to built-in compact flash
- test system running fully on TC17 and also running the application but without peripherals
- test system ALSO running crontabs for php scripts that get weather data, energy data, and solar panel readout

I notice that the syslog reports every crontab entry execution with a cron.err. The entries are correctly executed so I dont understand why it reports this as an "error" as it basically is more a "notice" but maybe thats just syntax. For now I ignore it. scripts are correctly executed so "no issue"
To my relieve the logging from syslog is "not overly big". No immediate worry that I break the usb drive within an hour. Breaking it is not a problem but that would stop the logging and that would bring me nowhere.

wait and see.......

[Rich]

Hi Stefann

... To my relieve the logging from syslog is "not overly big". No immediate worry that I break the usb drive within an hour. Breaking it is not a problem but that would stop the logging and that would bring me nowhere. ...

The commands I listed limit the space the logs will occupy so they can not fill your disk.

The total size from syslog will be capped at around 6 Mbytes.
It first creates  /sdb1/logging.
When that grows to 1 Mbyte, it moves /sdb1/logging to /sdb1/logging.0
The next time /sdb1/logging grows to 1 Mbyte:
        it moves /sdb1/logging.0 to /sdb1/logging.1 and /sdb1/logging to /sdb1/logging.0
Eventually you will have 6 files:
        logging, logging.0, logging.1, logging.2, logging.3, and logging.4.
When logging again reaches 1 Mbyte, syslog rotates the logging files like this:
        remove logging.4, move logging.3 to logging.4, move logging.2 to logging.3,
        move logging.1 to logging.2, move logging.0 to logging.1, and move logging to logging.0.
This logging file rotation will occur every time the logging file grows to 1 Mbyte in size.

The script I showed that executes the free command every 5 minutes also limits file size.
Every time that log file reaches a length of 2460 lines, the oldest 60 lines get deleted.
This limits the size of that file to about 100 Kbytes.

[Stefann]

update,
"live system" crashed again after about 17 hour runtime.
- running on TC17 core.gz and vmlinuz but with non-upgraded .tcz applications. Note, 1 exception, usb-to-serial kernel extension IS upgraded
- my home automation application running with several php & python scripts.

"damn"...
I used syslog to write to external usb: /sbin/syslogd -O /sdb1/logging -s 1000 -b 20 -l 5
But...
After restart de logging was gone. System started a new "logging" file.
In hindsight I should probably have removed the usb and read it with an other system.


Going forward..
- I remove the syslog start from boot local and mount and start it manual
- that ensures that after a reboot the usb is "untouched"

It "may" be that the logging file is not overwritten by the new syslog but had disappeared / got corrupted at the crash. We'll find out I guess.
[edit] a... I understand... I forgot to remount the drive before writing to it > fixed


On statistics:
I thought system could run more than a day but that is not true:
- Tuesday crash after about 12 hours
- Wednesday crash after about 13 hrs
- today crash after 17 hours
So, in all cases below 24hr

note..
The "test system" still runs. Running the application without peripherals. On full TC17 with full upgraded applications.

[GNUser]

Hi Stefann. I'm sorry to hear about your unexplained crashes.

Twice in the past I had problems with crashes after a TCL upgrade. Both times, the crashes were in the laptop that I repurposed as a wireless router. I'll share my experience with those problematic upgrades in case they are helpful to you.

1. One of the problematic upgrades (when I upgraded the wireless router from TCL13 x86_64 to TCL14 x86_64) turned out to be a regression in the mac80211 kernel module, which affected my specific wireless hardware. It is possible that your problem is not due to TCL17 applications, but due to a regression in linux 6.18.2--either the kernel itself or a module--that is affecting your hardware.

2. The second time I experienced wireless router crashes after a TCL version upgrade (I don't remember which TCL version), the crashes stopped after I recompiled the workhorse applications in my router, namely hostapd and dnsmasq. Maybe it would be helpful for you to make a list of the services that are always running on your machine, then check the "Current:" field in the .info file to see if the relevant extensions were created after TCL17 was released. If one or more of your workhorse applications/services were built before TCL17 was released, I would try rebuilding those extensions on TCL17.

I hope the above is helpful. Those were the only two times I had crashes after upgrading TCL to a new version.

[GNUser]

Hi Stefann. One way to explore possibility #1 from my post above (Reply #18) is to try booting TCL17 with a different/older kernel and modules copied from another distro. A while back I found a hacky way to do it. It's ugly as heck and I'm not proud of it, but here it is: https://forum.tinycorelinux.net/index.php/topic,26071.msg181848.html#msg181848

A less hacky way might be to use core.gz and applications from TCL17 but vmlinuz and foo-KERNEL extensions from the last TCL that worked for you without any problems (if I read your posts correctly, that was TCL15). Note that some kernel modules are provided by the base system, so you would have to add /lib/modules/TCL15KERNELVERSION to the TCL17 core.gz.

[Stefann]

Hi @GNUser
Thanks for this info!!!! It helps!!

I was already thinking a bit in that direction, your experience underlines it.
I would find it a bit weird that the standard linux has a problem. I would expect that to have surfaced way earlier.
I'm using 1 kernel extension: usb-serial-#-#-#-tinycore.tcz
This extension allows to connect a RS232 connection over USB port. It's connected to my energy meter where it reads data every second.
This extension is likely way less thoroughly tested during the TC17 beta-phase so it would make sense that a bug on this slipped through. And it connects with your experience.
At this moment I'm again running the application and I keep it running to see whether I can catch some logging at next crash. It's running now for 7 hours so "statistically" it should crash in about an other 7 hrs.

Thanks for the info.. because this motivates the next test to be:
- "remove the kernel extension and not use the peripheral" for next testrun.
I can actually bring this peripheral to my "test system" and see whether it works there.
"test system" is now running with full TC17 for over 24hrs now. It has the same kernel extension installed but not used.
Would be a very clear pointer if I can get the "test system" to crash with this usb-serial peripheral.

[GNUser]

- "remove the kernel extension and not use the peripheral" for next testrun.
...
Would be a very clear pointer if I can get the "test system" to crash with this usb-serial peripheral.

Hi Stefann. Having a log capture an error at the time of the crash would be the best, of course, but the above also sounds like a excellent strategy.

If you can poinpoint  usb-serial-KERNEL.tcz  as the cause of your crashes on TCL17, maybe the kernel mailing list or other users will point you to a workaround or fix/patch. (When I had my problem with the mac80211 kernel module in the wireless-KERNEL.tcz extension, it was actually other knowledgeable users--over at https://github.com/morrownr/USB-WiFi --who pointed me to a patch. I don't know if there is a similar waterhole for users who know everything about usb.)

[Stefann]

Update..
It’s still running after about 22 hours. That’s much longer than previous runs.
That’s actually “not good” because it’s clearly unreliable and it needs fix.

@GNUser, yes I try to catch error. That’s why I keep it running.
But when it crashes I will restart without the usb-serial kernel extension and let that extension run on my test system. Not alone will that isolate the error, it will also bring my “live system” in a more reliable state as it is indeed live controlling my house. Thursday morning system had crashed and heating had been off all night.
I should have error logging than.

With that said..
I more and more see reasons that it’s indeed de usb-serial kernel extension as it is a bit vulnerable.
It reads data from my energy meter but it opens the connection and keeps reading without ever closing. So apart from improving the kernel extension I could modify my code to close the connection after each data block.
I will try to catch the error but if it’s pinpointed to this than that’s what I will  do.
So far it was never unreliable though.
The energy read using serial read over usb has been opererational on DSL from 2009 to 2024 and than on TC15 when I moved to that in summer 2024. System runs 24/7 and after initial debugging it never crashed in at least last 10 years.

[Rich]

Hi Stefann

... It reads data from my energy meter but it opens the connection and keeps reading without ever closing. ...

That makes me wonder if any USB devices are being put to sleep.

When a USB device is idle for 2 seconds, the kernel suspends it to save power.

... This extension allows to connect a RS232 connection over USB port. It's connected to my energy meter where it reads data every second. ...

Yeah, I saw that too. But this isn't a real time kernel. So it's possible that if the
system becomes busy enough, every once in a while that read may be delayed
long enough that the kernel sees it as idle.

The easiest way to test this is to disable  usbcore.autosuspend.
Add this boot code to your boot loaders config file:

Code: [Select]

usbcore.autosuspend=-1
Since  usbcore  is built into the kernel, you can not unload it and
then reload it with autosuspend disabled.

you can disable it in a running kernel like this:

Code: [Select]

echo -1 | sudo tee /sys/module/usbcore/parameters/autosuspendBut in order for it to take effect on existing devices, you would probably
need to tell udev to re-scan the system:

Code: [Select]

sudo udevadm trigger
Personally, I feel the boot code option is the better way to test this.

[Stefann]

@rich,
Great great great!
Thanks!

To be more precise,
The energy meter sends a continuous stream of “telegrams”. It just sends them, you don’t have to request them. It’s a broadcast. It’s called the “DSMR” protocol (Dutch Smart Reader Requirements).
Until march-4 I had a dsmr 2.2 meter that spits out data every 10 seconds. It got replaced on march 4 by a dsmr 5 meter that spits out data every second.
As said, I’m interfacing with this already sinds 2009 so it’s real strange it stops working now.

Your comment however makes perfect sense!
Although the data was transmitted every 10 seconds and now every second,  I’m polling the usb about every 300ms, well within the suspend timeout. This however is NOT a real-time method. It’s just an endless service-loop for my home automation. It is very well possible that “occasioanllly” some other service takes 1 or 2 seconds.

So yes… this could be it.
With that said….
IF it’s indeed this interface (which is not yet proven), than it’s certainly Linux should not hang completely. I would totally understand the application to crash, but not the full OS including ssh, Apache, VNC, samba.

But… nothing is proven yet. At this moment it’s a strong but non-proven theory.
The system is still running. After 3 crashes that all took place after about 14 hrs it’s now already running for more than 24hrs. I keep letting it run. I have setup your syslog “trap” and really want to see whether I can catch some logging.

Thanks for the method to disable suspend. I will certainly make that part of my test-approach. I will use the boot-option.

[Stefann]

Hi @rich,

I'm using extlinux.conf
Is this how I should modify it? (added last line, other lines is current)
Or should it be part of the "APPEND" string? or different?

Code: [Select]

DEFAULT gui17

LABEL gui17
KERNEL /tce/boot/vmlinuz17
INITRD /tce/boot/core17.gz
APPEND quiet  host=huis cron tz=CET-1CEST,M3.5.0,M10.5.0/3 waitusb=5:UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113" tce=UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113"
usbcore.autosuspend=-1
Thanks

[patrikg]

On the long APPEND string. Not on it's own line.
You can also see the kernel command line when booted with:

Code: [Select]

cat /proc/cmdline

[Rich]

Hi Stefann
Insert it right right between  quiet  and  host=huis.
The contents of  APPEND  must be 1 continuous line.

You can also see the kernel command line with:

Code: [Select]

showbootcodes

[Stefann]

Thanks @Rich & @patrikg.
Will do that.

Note, I start getting the feeling that 1st 3 crashes where not "typically around after 14hr duration" but "typically around 24:00"
Which connects to the usb-suspend theory because at 24:00 the application does some housekeeping which can cause a larger loop-time.
Thank "last night it missed that slot" but that is also not strange,. It's not 100% deterministic.
So.. Will see whether I have a crash upcoming midnight.

[Stefann]

Update,
This morning the system was still running.
In itself "I like that it runs", but as it had failed 3 times earlier I actually DON't like it because this thing is supposed to be robust enough to run for multiple month without hiccup.

But anyways...
- I manually stopped it
- I brought the tce/optional folder to full TC17 status so system is now 100% TC17
- I removed the external usb that I used for syslog to write to
- I reconnected my X10 peripheral to that usb-port as I had to remove that few days aago to make room for the usb stick.
- I reduced the amount of cron_actions to bare minimum for reason every cron_action triggers the syslog and I want to minimise write wear of my compact flashdrive
- I did setup the syslog logging as suggested by @Rich to write to internal compact flash in stead of external usb.

So...
- I'm actually in full operational mode now
- but WITH logging enabled to persistent storage such that I can catch error logging in case of crash

Note: I did NOT insert "usbcore.autosuspend=-1" in boot line yet. I try to let it crash and collect logging before I do that.