Re: Tiny Core v17.0 upgrade issues

[Stefann]

Update,
Its running as I said in previous post

- Life system running on TC17 core.gz & vmlinuz but non-upgraded .tcz applications
- disabled rsyslog
- /sbin/syslogd -O /sdb1/logging -s 1000 -b 20 -l 5
So this writes to external usb, not to built-in compact flash
- test system running fully on TC17 and also running the application but without peripherals
- test system ALSO running crontabs for php scripts that get weather data, energy data, and solar panel readout

I notice that the syslog reports every crontab entry execution with a cron.err. The entries are correctly executed so I dont understand why it reports this as an "error" as it basically is more a "notice" but maybe thats just syntax. For now I ignore it. scripts are correctly executed so "no issue"
To my relieve the logging from syslog is "not overly big". No immediate worry that I break the usb drive within an hour. Breaking it is not a problem but that would stop the logging and that would bring me nowhere.

wait and see.......

[Rich]

Hi Stefann

... To my relieve the logging from syslog is "not overly big". No immediate worry that I break the usb drive within an hour. Breaking it is not a problem but that would stop the logging and that would bring me nowhere. ...

The commands I listed limit the space the logs will occupy so they can not fill your disk.

The total size from syslog will be capped at around 6 Mbytes.
It first creates  /sdb1/logging.
When that grows to 1 Mbyte, it moves /sdb1/logging to /sdb1/logging.0
The next time /sdb1/logging grows to 1 Mbyte:
        it moves /sdb1/logging.0 to /sdb1/logging.1 and /sdb1/logging to /sdb1/logging.0
Eventually you will have 6 files:
        logging, logging.0, logging.1, logging.2, logging.3, and logging.4.
When logging again reaches 1 Mbyte, syslog rotates the logging files like this:
        remove logging.4, move logging.3 to logging.4, move logging.2 to logging.3,
        move logging.1 to logging.2, move logging.0 to logging.1, and move logging to logging.0.
This logging file rotation will occur every time the logging file grows to 1 Mbyte in size.

The script I showed that executes the free command every 5 minutes also limits file size.
Every time that log file reaches a length of 2460 lines, the oldest 60 lines get deleted.
This limits the size of that file to about 100 Kbytes.

[Stefann]

update,
"live system" crashed again after about 17 hour runtime.
- running on TC17 core.gz and vmlinuz but with non-upgraded .tcz applications. Note, 1 exception, usb-to-serial kernel extension IS upgraded
- my home automation application running with several php & python scripts.

"damn"...
I used syslog to write to external usb: /sbin/syslogd -O /sdb1/logging -s 1000 -b 20 -l 5
But...
After restart de logging was gone. System started a new "logging" file.
In hindsight I should probably have removed the usb and read it with an other system.


Going forward..
- I remove the syslog start from boot local and mount and start it manual
- that ensures that after a reboot the usb is "untouched"

It "may" be that the logging file is not overwritten by the new syslog but had disappeared / got corrupted at the crash. We'll find out I guess.
[edit] a... I understand... I forgot to remount the drive before writing to it > fixed


On statistics:
I thought system could run more than a day but that is not true:
- Tuesday crash after about 12 hours
- Wednesday crash after about 13 hrs
- today crash after 17 hours
So, in all cases below 24hr

note..
The "test system" still runs. Running the application without peripherals. On full TC17 with full upgraded applications.

[GNUser]

Hi Stefann. I'm sorry to hear about your unexplained crashes.

Twice in the past I had problems with crashes after a TCL upgrade. Both times, the crashes were in the laptop that I repurposed as a wireless router. I'll share my experience with those problematic upgrades in case they are helpful to you.

1. One of the problematic upgrades (when I upgraded the wireless router from TCL13 x86_64 to TCL14 x86_64) turned out to be a regression in the mac80211 kernel module, which affected my specific wireless hardware. It is possible that your problem is not due to TCL17 applications, but due to a regression in linux 6.18.2--either the kernel itself or a module--that is affecting your hardware.

2. The second time I experienced wireless router crashes after a TCL version upgrade (I don't remember which TCL version), the crashes stopped after I recompiled the workhorse applications in my router, namely hostapd and dnsmasq. Maybe it would be helpful for you to make a list of the services that are always running on your machine, then check the "Current:" field in the .info file to see if the relevant extensions were created after TCL17 was released. If one or more of your workhorse applications/services were built before TCL17 was released, I would try rebuilding those extensions on TCL17.

I hope the above is helpful. Those were the only two times I had crashes after upgrading TCL to a new version.

[GNUser]

Hi Stefann. One way to explore possibility #1 from my post above (Reply #18) is to try booting TCL17 with a different/older kernel and modules copied from another distro. A while back I found a hacky way to do it. It's ugly as heck and I'm not proud of it, but here it is: https://forum.tinycorelinux.net/index.php/topic,26071.msg181848.html#msg181848

A less hacky way might be to use core.gz and applications from TCL17 but vmlinuz and foo-KERNEL extensions from the last TCL that worked for you without any problems (if I read your posts correctly, that was TCL15). Note that some kernel modules are provided by the base system, so you would have to add /lib/modules/TCL15KERNELVERSION to the TCL17 core.gz.

[Stefann]

Hi @GNUser
Thanks for this info!!!! It helps!!

I was already thinking a bit in that direction, your experience underlines it.
I would find it a bit weird that the standard linux has a problem. I would expect that to have surfaced way earlier.
I'm using 1 kernel extension: usb-serial-#-#-#-tinycore.tcz
This extension allows to connect a RS232 connection over USB port. It's connected to my energy meter where it reads data every second.
This extension is likely way less thoroughly tested during the TC17 beta-phase so it would make sense that a bug on this slipped through. And it connects with your experience.
At this moment I'm again running the application and I keep it running to see whether I can catch some logging at next crash. It's running now for 7 hours so "statistically" it should crash in about an other 7 hrs.

Thanks for the info.. because this motivates the next test to be:
- "remove the kernel extension and not use the peripheral" for next testrun.
I can actually bring this peripheral to my "test system" and see whether it works there.
"test system" is now running with full TC17 for over 24hrs now. It has the same kernel extension installed but not used.
Would be a very clear pointer if I can get the "test system" to crash with this usb-serial peripheral.

[GNUser]

- "remove the kernel extension and not use the peripheral" for next testrun.
...
Would be a very clear pointer if I can get the "test system" to crash with this usb-serial peripheral.

Hi Stefann. Having a log capture an error at the time of the crash would be the best, of course, but the above also sounds like a excellent strategy.

If you can poinpoint  usb-serial-KERNEL.tcz  as the cause of your crashes on TCL17, maybe the kernel mailing list or other users will point you to a workaround or fix/patch. (When I had my problem with the mac80211 kernel module in the wireless-KERNEL.tcz extension, it was actually other knowledgeable users--over at https://github.com/morrownr/USB-WiFi --who pointed me to a patch. I don't know if there is a similar waterhole for users who know everything about usb.)

[Stefann]

Update..
It’s still running after about 22 hours. That’s much longer than previous runs.
That’s actually “not good” because it’s clearly unreliable and it needs fix.

@GNUser, yes I try to catch error. That’s why I keep it running.
But when it crashes I will restart without the usb-serial kernel extension and let that extension run on my test system. Not alone will that isolate the error, it will also bring my “live system” in a more reliable state as it is indeed live controlling my house. Thursday morning system had crashed and heating had been off all night.
I should have error logging than.

With that said..
I more and more see reasons that it’s indeed de usb-serial kernel extension as it is a bit vulnerable.
It reads data from my energy meter but it opens the connection and keeps reading without ever closing. So apart from improving the kernel extension I could modify my code to close the connection after each data block.
I will try to catch the error but if it’s pinpointed to this than that’s what I will  do.
So far it was never unreliable though.
The energy read using serial read over usb has been opererational on DSL from 2009 to 2024 and than on TC15 when I moved to that in summer 2024. System runs 24/7 and after initial debugging it never crashed in at least last 10 years.