Re: Tiny Core v17.0 upgrade issues

[Stefann]

Today 14:00 still running on TC15 after 3days.
I leave it running, go on 8day travel.

Note, I realised that instead of logging to usb-stick on 2nd system I can simply do that to the ram-disk of the 2nd system.
When logging on the live-system logging to ram-disk was not an option because it was not readable after a crash. Now I log via the network to the 2nd system there is no reason to avoid its ram-disk.
That means I can do logging at debug-level without causing excessive flashdisk writes when I re-setup the thing in "crash mode on TC17" when I'm back

For those interested, my full crashlog.
Initial log-entries are less precise maintained than later entries as at the beginning I did not know it would be this structural.

Code: [Select]

mon march 16: start around 12:00 //full TC17
Tue march 17: crashed in morning without yesterday data, so 24:00 latest
Tue march 17: start around 13:00 //full TC17
Wed march 18: crashed in morning without yesterday data, so 24:00 latest
Wed march 18: start around 23:00 //TC17 core & vmlinuz but no apps, add logging

Fri march 20: crashed in morning without yesterday data, so 24:00 latest // no logging due to mount mistake
Fri march 20: start around 14:00 //TC17 core & vmlinuz but  no apps, no X10 interface on usb because usb port used for logging on usb-stick
mon march 23: manual stop around 9:00 // no crash after 4 days
mon march 23: start with full TC17 and P1 & X10 interface 8:30 // X10 interface installed again, check that it still crashes
mon march 23: crash, no logging 22:00 - 23:00 // no logging because usb-slot used by X10 interface, not available for logging usb stick
tue march 24: restart with usbcore.autosuspend=-1 bootcode around 9:00 // usbcore.autosuspend=-1 bootcode
tue march 24: manual stop around 12:00
multiple manual start/stop due to install of home battery
Wed march 25: restart with usbcore.autosuspend=-1 bootcode around 20:00 // usbcore.autosuspend=-1 bootcode
Fri march 27: manual stop for maintenance 10:00; 38hr crash-free run
Fri march 27: restart with usbcore.autosuspend=-1 bootcode around 14:20 // usbcore.autosuspend=-1 bootcode
Sat march 28: crash around 9:00 //after 19hrs
Sat march 28: restart with usbcore.autosuspend=-1 bootcode around 13:30 // introduced serious logging, *.notice
Mon march 30: manual stop around 14:00 // 48hr run without issues
Mon march 30: restart under TC15, no extra bootcode around 17:00
Thu april 02: still running on TC15. Leaving for a 8 day travel

[Stefann]

OK... I'm back
System has been running on TC15 without any glitch for 12 days on TC15 without issue.
--> that proofs that it is TC related

Stats sofar:
TC15
- including usb-serial-6.6.8-tinycore.tcz
- "per second reading" of serial interface
- multiple month of running without issue
- rechecked last 2 weeks, 12days without issue ending with manual stop

TC17
- including usb-serial-6.18.2-tinycore.tcz
- "per second reading" of serial interface
3x crash after about 15hrs of runtime

TC17
- including usbcore.autosuspend=-1 bootcode
- including usb-serial-6.18.2-tinycore.tcz
- "per second reading" of serial interface
1x crash after about 19hrs of runtime
2x still running after >38hrs but manual stopped for reason of maintenance

I just started the thing:
TC17
- WITHOUT recompile or relink of application
- WITHOUT usbcore.autosuspend=-1 bootcode // try to get it to crash within a day
- including usb-serial-6.18.2-tinycore.tcz
- "per second reading" of serial interface
- rsyslog logging at debug level (*.*), writing over network to Ramdisk of 2nd computer (avoid flash-wear)

Note: on "logging" I'm a bit disappointed.
- I now see very detailed logging from my application
- but litterally zero kernel logging (maybe a bit at startup but that has been rotated away)
- I would have expected that "debug level" (which I think is included in *.*) would give me kernel things per second

So: any tips on getting more kernel logging are welcome
I'm writing to a Ramdisk so "overload of logging" is not a problem.

[Stefann]

Wel...
after 3 days of running in "crash mode" it did not crash.
I stopped the thing manually.

I had a tiny "hunge"... I did a full recompile of my application when I went back to TC15.
So... I grabbed the executable from the backup I made before upgrading and started that.

We'll see how this goes.
For the record, I did NOT reboot TC17

so... (recent) history:
sat april 11 TC17, 15:30
- WITHOUT recompile or relink of application
- WITHOUT usbcore.autosuspend=-1 bootcode // try to get it to crash within a day
- including usb-serial-6.18.2-tinycore.tcz
- "per second reading" of serial interface
- rsyslog logging at debug level (*.*), writing over network to Ramdisk of 2nd computer (avoid flash-wear)

tue april 14 16:20, manual stop, no crash over 3 days

tue april 14 16:23, start again under TC17
- using executeble as compiled on TC 15 march15, same executable that crashed earlier
- no reboot of TC17
- WITHOUT usbcore.autosuspend=-1 bootcode // try to get it to crash within a day
- including usb-serial-6.18.2-tinycore.tcz
- "per second reading" of serial interface
- rsyslog logging at debug level (*.*), writing over network to Ramdisk of 2nd computer (avoid flash-wear)

proof I donot have  usbcore.autosuspend=-1 bootcode

Code: [Select]

tc@huis:/krubo/work$ showbootcodes
BOOT_IMAGE=/tce/boot/vmlinuz17 quiet  host=huis cron tz=CET-1CEST,M3.5.0,M10.5.0/3 waitusb=5:UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113" tce=UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113"  initrd=/tce/boot/core17.gz
proof of release running:

Code: [Select]

tc@huis:/krubo/work$ uname -a
Linux huis 6.18.2-tinycore #1 SMP Sat Dec 20 21:06:22 UTC 2025 i686 GNU/Linux
proof of version of executable:

Code: [Select]

tc@huis:/krubo/work$ ls -l krubo
-rwxr-xr-x    1 tc       staff       242508 Mar 14 16:33 krubo

[Stefann]

Update.
Last Wednesday my MacBook broke down. Screen was flickering but still working on external monitor. So.. likely screen connection flex-cable.
I disassembled the MacBook, removed the screen, but failed to repair the flex cable. Got it reassembled but same problem.
I ended up buying a new MacBook. That was "not entirely bad", the macbook was 8 years old and I was already longing for an M-series computer so now I had a good excuse. Got the new model as released 1 month ago.

But... that's why I was a bit silent. "Not having the laptop" means "not logging in" on my tinycore system. I CAN login from my iPad but that's very inconvenient for doing serious work like this.

Having said that...
System did run without crash from April-14 16:23 until today, April 18 14:30, so 4 days.
On the one hand "great", on the other other hand "but why did it fail 3 time in a row after 15 hours when I upgrade tot TC17"?

So...
next try...
I recompiled my application under TC17.
It's a bit of a wild guess. But I know I recompiled parts of my application when the things were failing earlier.
When I originally upgraded from TC15 to TC17 I first got the thing running with TC15 compiled executable but "after I concluded that was running without problem" I made some modifications and recompiled.
So, I do not know whether the failing was from the TC15 compiled executable or from the TC17 compiled executable.
Note: I now did a full recompile which is different from my original sequence of events that had only a partial recompile. It becomes extremely difficult to exactly redo that

Any suggestions are welcome
Now this config is under test:

Code: [Select]

sat april 18 15:00, start again under TC17
- using full recompiled executable under TC17
- WITHOUT usbcore.autosuspend=-1 bootcode // try to get it to crash within a day
- including usb-serial-6.18.2-tinycore.tcz
- "per second reading" of serial interface
- rsyslog logging at debug level (kern.*), writing over network to Ramdisk of 2nd computer (avoid flash-wear)


note:
I changed the (*.*) Rsyslog to (kern.*).
By doing so I got rid of logging pollution from my own application. My application sends multiple logging messages per second so that was quite annoying.

I got lots of kernel messages on startup > so logging is working
But zero messages over past hour

[Stefann]

Bang,
Crash after about 4 hours.
It’s late now, I will look further tomorrow.
Quick conclusion.
It’s not the Linux kernel.
It’s a weak spot in my application that gets exposed by the new gcc
Unfortunately zero logging from rsyslog. Logging was active but did not give any output.

What probably happened during the tc15 > tc17 upgrade:
- upgraded tc15 to tc17 > all ok, ssh, samba, vnc, Apache all functional.
- ran application > also ok
Than…
As I was working on the system and seemed fine..
Did some changes to the code and did a recompile.
>> software still running. No suspicion

17 hour later crash > forgot about the recompile, blamed tc17.

At this moment this is my theory. Tomorrow I will see how to proceed.

[Stefann]

Ok..... new day...

I booted back to TC15
Did a complete recompile
than booted into TC17
Re-compiled only the top-level .c file
Restarted the application

I'm a bit under the assumption that the "issue" is in one of the driver or I/O related functions which I now did NOT compile on tc17.

If it does NOT crash after 24hr:
> I will recompile the .c file that drives the serial-over-usb port and try again

If it DOES crash after 24hr:
> I will revert to all compilation by tc15 and just do the linking in tc17 and try again

I also found my old vga monitor and keyboard at the attic and connected that. So I now have a direct terminal "not over ethernet"

We'll see

[Stefann]

Update
I noticed that the energy meter was not updating.
And that is suspicious because the energy meter is connected to the "serial over usb" port that uses the kernel extension "usb-serial-6.18.2-tinycore.tcz"

So..
I rechecked again...
And I changed the makefile such that I can relink the application without doing any compilation

boot to TC15:
compile all
run  > program runs, energy meter works

boot to TC17
run  > program runs, energy meter works (this configuration did run for 4 days April-14 >18)
delete executable
relink executable without any compilation
run  > program runs, everything seems to function but energy meter is not working
recompile serial over usb component
run  > program runs, energy meter works

I leave this running for a while

Along the way:
- I invested in making "boot different versions" super easy selectable from extlinux.conf by using dedicated tce directories while using same restore location
- I now have my "main.c" also as a separate .o object file. I used to have all underlying components compiled to .o but main.c compiled and linked together.
- system now has physical hardwired monitor & keyboard for testing (it's taking annoying amount of space)

[Stefann]

update,
definitely "something" is not right

Code: [Select]

19 april
15:25 Mem: 468928K used, 492592K free, 35852K shrd, 3844K buff, 283808K cached
15:38 Mem: 489316K used, 472204K free, 36036K shrd, 3852K buff, 283992K cached
15:41 Mem: 497508K used, 464012K free, 35900K shrd, 3852K buff, 283856K cached
16:00 Mem: 497464K used, 464056K free, 35832K shrd, 3868K buff, 283852K cached
16:16 Mem: 497672K used, 463848K free, 35880K shrd, 3868K buff, 283900K cached
16:53 Mem: 506816K used, 454704K free, 35876K shrd, 3868K buff, 283896K cached
17:02 Mem: 507040K used, 454480K free, 35956K shrd, 3868K buff, 283976K cached
17:43 Mem: 515232K used, 446288K free, 35912K shrd, 3884K buff, 283932K cached
18:02 Mem: 523640K used, 437880K free, 36028K shrd, 3884K buff, 284048K cached

looks like I'm loosing memory at a speed : 54712k over 2.7hr = 20MB/hr
My "serial over usb" port is reading blocks of 812byte every second = 3MB/hr (and I'm sure overhead will multiply this)

So... its entirely possible that the "serial over usb" kext is completely "not" releasing memory for any data it is receiving (until connection gets closed).
My application opens a serial file descriptor and keeps reading it for ever.
Note: using this connector is a configuration item.
I keep it running until it (hopefully) crashes, but after that I can rerun it without reading the serial port without recompilation.

[Rich]

Hi Stefann
Each time before checking for free memory, I suggest running this:

Code: [Select]

sync
sudo cache-clear
syncThis helps to insure other processes are not skewing the results.

[Stefann]

@rich...
good suggestion...
now I got a lot back:

Code: [Select]

NO CACHE CLEAR
Sun Apr 19 19:57:41 CEST 2026
 m              total        used        free      shared  buff/cache   available
Mem:         961520      228600      424736       35888      308184      559560
Swap:        226116           0      226116

WITH CASH CLEAR
Sun Apr 19 19:59:40 CEST 2026
 m              total        used        free      shared  buff/cache   available
Mem:         961520      258156      635764       35948       67600      578960
Swap:        226116           0      226116

But...
- did I now cleanup the analysis?
- or did I inject a solution? (doing a cleanup that the application or kext should have done)?

I think I better NOT do this at this point. Just observe whether it runs towards a crash if I do not interfere

[Rich]

Hi Stefann

... - did I now cleanup the analysis? ...

Yes you did.

... - or did I inject a solution? (doing a cleanup that the application or kext should have done)? ...

No, you did not. If you really have a memory leak, free
memory will continue to drop.

[Stefann]

1 day later, no crash.
memory is not going down the drain.

pfew.. its not easy.

I stopped the application & recompiled all under TC17. This was the "fastest crash situation" so far.
Last stretch it did run without gui and without vnc server for reason I have a text-terminal connected to it without mouse. That however also reduces the system load. So.. gui & vnc are active again.
I DO have the monitor attached. Not sure weather I will be able to type something but I will have at least "some" screen vision when it crashes.

Also monitoring memory use.

[patrikg]

Do you leak on your own app ?
Maybe valgrind time then.

[Stefann]

Do you leak on your own app ?
Maybe valgrind time then.

On the one hand: "everything is possible"
on the other hand: my app used to run uninterrupted for multiple month without issue on TC15

never used valgrind but my brother also mentioned it.
At this moment I try to have it crash in a configuration that did crash earlier.
I noticed that "any modification" changes the behavior which made me conclude I best "make the absolute minimum of modifications" per analysis.

At this moment I have a monitor connected to the system that normally runs headless. So I hope for a crash "with info"

[Stefann]

and an other crash...
after about 16hr of runtime with full application recompiled under tc17

Code: [Select]

mon Apr 20 16:15:18  // restart with full recompile under TC17
tue apr 21 8:50 //crashed
- hardwired connected monitor does not show anything, "no signal"
- kernel logging not present (Rsyslog logging kern.* messages)
- memory logging shows no issue:
Tue Apr 21 07:57:25 CEST 2026
 m              total        used        free      shared  buff/cache   available
Mem:         961520      278448      445316       42608      237756      503028
Tue Apr 21 08:12:25 CEST 2026
 m              total        used        free      shared  buff/cache   available
Mem:         961520      279016      444812       42544      237692      502524
Tue Apr 21 08:27:25 CEST 2026
 m              total        used        free      shared  buff/cache   available
Mem:         961520      279084      444812       42476      237624      502524
Tue Apr 21 08:42:25 CEST 2026
 m              total        used        free      shared  buff/cache   available
Mem:         961520      279028      444812       42532      237680      502524
>> 8 minutes later it had crashed
I now restarted:
- exactly same executable
- but changed setting: it now no longer reads /dev/ttyUSB0

any tips on getting more logging?