Re: Tiny Core v17.0 upgrade issues

[Stefann]

Update,
yes.. crash again. In about 14 hours.
Apart from having the application .tcz files updated I also reconnected the X10 peripherals over usb.
I have no idea how that interface looks like as I made that in 2009. I DO remember that is was a bit of a "dirty from the internet code". There was no GitHub at that time.

Unfortunately there is no logging of the crash.
Here you see the logging I haver. Its basically just logging the crontab
crash somewhere between 22:00 and 23:00

Code: [Select]

Mar 23 19:31:00 huis cron.err crond[3358]: USER tc pid 4738 cmd php /home/tc/php/knmi.php
Mar 23 20:01:00 huis cron.err crond[3358]: USER tc pid 4760 cmd /krubo/work/krubo_watchdog
Mar 23 20:31:00 huis cron.err crond[3358]: USER tc pid 4783 cmd /krubo/work/krubo_watchdog
Mar 23 20:31:00 huis cron.err crond[3358]: USER tc pid 4784 cmd php /home/tc/php/knmi.php
Mar 23 21:01:00 huis cron.err crond[3358]: USER tc pid 4803 cmd /krubo/work/krubo_watchdog
Mar 23 21:31:00 huis cron.err crond[3358]: USER tc pid 4823 cmd /krubo/work/krubo_watchdog
Mar 23 21:31:00 huis cron.err crond[3358]: USER tc pid 4824 cmd php /home/tc/php/knmi.php
Mar 23 22:01:00 huis cron.err crond[3358]: USER tc pid 4844 cmd /krubo/work/krubo_watchdog
used this command as suggested by @rich for logging:

Code: [Select]

/sbin/syslogd -O /krubo/work/logging.txt -s 1000 -b 20 -l 5

Anyway.. I just restarted the application.
Will do with different bootcode tomorrow.
Sleeping time now.

[edit] could not resist and did a quick peek into the x10 code. Its basically using the libusb1 library for interfacing.

[Rich]

Hi Stefann

... Unfortunately there is no logging of the crash.
Here you see the logging I haver. Its basically just logging the crontab ...

If you want to stop crond from spamming your log file:

Code: [Select]

tc@E310:~$ crond --help
BusyBox v1.29.3 (2018-12-19 15:29:37 UTC) multi-call binary.

Usage: crond -fbS -l N -d N -L LOGFILE -c DIR

        -f      Foreground
        -b      Background (default)
        -S      Log to syslog (default)
        -l N    Set log level. Most verbose 0, default 8
        -d N    Set log level, log to stderr
        -L FILE Log to FILE
        -c DIR  Cron dir. Default:/var/spool/cron/crontabstry starting crond with the  -L  option and point it to another file, or
to  /dev/null  if you prefer to send those messages to a black hole.

used this command as suggested by @rich for logging:

Code: [Select]

/sbin/syslogd -O /krubo/work/logging.txt -s 1000 -b 20 -l 5

Then you should be able to reduce the number of log files kept
back to 5 by changing  -b 20  back to  -b 5.

[Stefann]

All,
New day, new chances.

So,... the system did run through the night. As the crash was yesterday between 22:00 & 23:00 it's not related to "24:00 maintenance".
The kind of strange thing is that the system did run for 3 days without crash and now it again crashes after about 14hr runtime.
Only differences:
- removed usbstick for external logging
- reconnected X10 USB interface that uses libusb-1.0 library form my C-language application
- upgraded .tcz applications to TC17 level again

I "kind of feel without evidence" that it's not the .tcz applications. I still suspect the usb stuff

Anyways, restarted the system with "usbcore.autosuspend=-1" boot option.

Here is my applicable extlinux.conf section

Code: [Select]

LABEL gui17tst
KERNEL /tce/boot/vmlinuz17
INITRD /tce/boot/core17.gz
APPEND quiet usbcore.autosuspend=-1 host=huis cron tz=CET-1CEST,M3.5.0,M10.5.0/3 waitusb=5:UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113" tce=UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113"
showbootcodes:

Code: [Select]

tc@huis:/krubo/work$ showbootcodes
BOOT_IMAGE=/tce/boot/vmlinuz17 quiet usbcore.autosuspend=-1 host=huis cron tz=CET-1CEST,M3.5.0,M10.5.0/3 waitusb=5:UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113" tce=UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113"  initrd=/tce/boot/core17.gz
I will likely have to manually halt the system a few times in upcoming 2 days for reason I get a home-battery installed.
I will definitely keep this thread updated but probably no news in upcoming 2..4 days.

@Rich, thanks for the directions on avoiding cron-logging. I will for now not do that for reason I have minimised the entries towards low flash-wear anyways. And having some logging helps to get timestamps for start&crash. Very likely I do something like that at a later stage.

With that said... "any tips to get better crash logging"?? I basically have zero logging from last crash. I probably need to set -l to 6 instead of 5.
Will try that later. Lets see how the "usbcore.autosuspend=-1" works out for now.

    [Edit]: Adjusted first showbootcodes code tag.  Rich

[Rich]

Hi Stefann

... With that said... "any tips to get better crash logging"?? I basically have zero logging from last crash. I probably need to set -l to 6 instead of 5. ...

Not at the moment. you seem to have it figured out for now.

I initially went with  -l 5  because it seemed like a reasonable
compromise between message severity and log activity. It's
possible you may have to turn up that setting for more info.

[Stefann]

Small update.
I had several manual stops over past few days for reason I got a home battery installed.
However..
WITH the “disabled usb suspend” boot code and fully on TC17 the system did run for 38hrs from Wednesday 20:00 until this morning (Friday) 10:00.
I had to manually stop the system this morning for intended maintenance.

So…
- it’s not a complete proof yet.
- but given that all 3 earlier runs in this exact same configuration earlier crashed after about 15hrs it’s is a very positive sign.

At this moment I did not add additional logging for reason I donot want to stress my compact flash drive. Earlier I modified things to write to a usb stick but that required me to remove a peripheral to free-up a port and the system got more stable. So… “the observation changed the behavior”.
I COULD include a usb-hub to allow both the usb-stick and the peripheral bit that would again change the configuration.
So…
For sure I will do such testing later but for now I just want see whether the “disabled usb suspend” boot code solves the issue.
So… I will try to run it for at least a week. Likely 2 weeeks because I will be traveling 2-10 April.

With that all said……
Does anybody know where I can find the source code of usb-serial-6.18.2-tinycore.tcz ??
I would like to compare that to usb-serial-6.6.8-tinycore.tcz as that was running fine without “disable usb suspend” boot code on TC15

That an application crashes for reason of timeouts is kind of “acceptable behavior”.
That Linux core crashes is “not really”.

[Rich]

Hi Stefann

... Does anybody know where I can find the source code of usb-serial-6.18.2-tinycore.tcz ?? ...

That's a kernel driver.

You would have to look under  drivers/usb/serial/  in the kernel source package.

[Stefann]

Hi Stefann

... Does anybody know where I can find the source code of usb-serial-6.18.2-tinycore.tcz ?? ...

That's a kernel driver.

You would have to look under  drivers/usb/serial/  in the kernel source package.

Thanks.
pfew....  that is not simple....
I guess I focus on getting some logging first.
 

[Rich]

Hi Stefann

... pfew....  that is not simple....

Yeah, there are 80 source files in there. I would have mentioned that
sooner, but I didn't want to spoil the surprise.

[Paul_123]

usb/serial is not so bad, Only a few files in that directory are related to your specific chipset.  Did you ever say which kernel module was being loaded?

Your best bet would be to connect to the kernel git (https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git) or one of the github clones. and then look for driver updates or compare between branches.  USB is a convoluted monster,  you have usb/serial for the endpoint driver, but its also going to have drivers/usb/core /drivers/usb/dwc2 (USB2) or drivers/usb/dwc3 (USB3)  For the USB host adapter and core.

But yes, logging is important to see if you can find which rabbit hole to start looking.  Did you ever tell us?

Which USB serial driver is being used (Also the USB IDs)
What the USB host adapter is (USB2 or USB3) and USB IDs of it.
IS there a USB hub involved?

Everything related to USB in dmesg would be useful.  (And turn up the KERNEL Logging level in cmdline.txt )  and of course anything syslog picks up from the kernel from that logging.

[Stefann]

@rich,
no worry.
Not sure whether I will be able to "do something usefull" on finding a bug,
But at least I get my feet wet.

Note....
Reminds me of when I started my homecontrol project in 2008.
I have a background as "control engineer" and I liked the idea of designing my own intelligent control system.
I than learned the hard way that "control logic" is about 2% of the whole thing.
Next to all hardware, electronics but also plumbing, I did choose the "maxim 1wire components" for switches and temperature sensors.
They came with a SDK that included all usb-to-1wire drivers so I thought I would have a nice time.
However..... the sensors behaved 10x to 50x slower than harware specifications had promised.
So.... I started to look into the driver software and discovered a real nightmare of "bad code".
I ended up rewriting the USB drivers.......
Successfully, I got it to HW promised speed level.

Soooooo in theory I should know "a little bit" about this.
In practice this knowledge is about 17 years old and not maintained.... 

Anyway... it's a hobby project... so I like to explore.

[Stefann]

usb/serial is not so bad, Only a few files in that directory are related to your specific chipset.  Did you ever say which kernel module was being loaded?

Your best bet would be to connect to the kernel git (https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git) or one of the github clones. and then look for driver updates or compare between branches.  USB is a convoluted monster,  you have usb/serial for the endpoint driver, but its also going to have drivers/usb/core /drivers/usb/dwc2 (USB2) or drivers/usb/dwc3 (USB3)  For the USB host adapter and core.

But yes, logging is important to see if you can find which rabbit hole to start looking.  Did you ever tell us?

Which USB serial driver is being used (Also the USB IDs)
What the USB host adapter is (USB2 or USB3) and USB IDs of it.
IS there a USB hub involved?

Everything related to USB in dmesg would be useful.  (And turn up the KERNEL Logging level in cmdline.txt )  and of course anything syslog picks up from the kernel from that logging.

THANKS!!!!
I'm not sure I understand all you say...
But at least some answers...

- not using a hub
- using  usb-serial-6.18.2-tinycore.tcz
- it's basically a RS232 protocol connected to a usb by a serial-to-usb cable
- USB2 (maybe 1??) it's on a 500MHz via Eden single core cpu from 2008.
- Probably to make it very clear: I get all serial data in my terminal when I simply type:

Code: [Select]

cat /dev/ttyUSB0- The data is send as broadcast by my energy meter (using DSMR5 standard, Dutch Smart Meter Requirements). You donot have to request it. It just spits out a report every second.

As said... For now I will just keep the thing running and see whether the "usbcore.autosuspend=-1" bootcode fixes it. After that I will spend time in trying to catch the error with logging.

[Rich]

Hi Stefann
If you look at:
http://tinycorelinux.net/17.x/x86/tcz/usb-serial-6.18.2-tinycore.tcz.list
it provides many drivers. Paul_123 was asking which one(s) get loaded.

This should help in gathering more information:

Code: [Select]

tce-load -wi lsusb
lsusb > lsusb.txt
lsusb -t >> lsusb.txt
dmesg | grep -iE "usb|serial" > dmesgusb.txt
lsmod | head -n 1 > lsmod.txt
lsmod | tail -n +2 | sort >> lsmod.txt
Then post the contents of  lsusb.txt, dmesgusb.txt, and lsmod.txt.

[Stefann]

Update,
Sadly...  this morning crashed again. After less than 12hr :-(
No logging, I was basically testing whether the "usbcore.autosuspend=-1" bootcode would fix the problem with system running in normal state. So... it did not.

Thanks @Rich for directions on lsusb.tcz. However... it's not in the distribution.
I checked http://repo.tinycorelinux.net/17.x/x86/tcz/ until http://repo.tinycorelinux.net/6.x/x86/tcz/ and could not find it (I skipped a few releases)

Anyway...
I think I will setup a syslog that writes to a network connected usb on my test system. By doing so I feel comfortable logging massive data and potentially frying the usb-stick.
I need to figure out how to get this all operational but i think I will manage.

From tomorrow or Monday onwards I will revert back to TC15 however. I will be travelling April 2-10 and over that time the system should be rock stable. So I want to check that few days before I leave. That will (hopefully) also give confirmation that it's really upgrade related and not "something else that by change happened during the upgrade"

[Zhe]

lsusb in usbutils.tcz

[Rich]

Hi Stefann
Sorry, my mistake. Zhe is correct. I should have said:

Code: [Select]

tce-load -wi usbutils
lsusb > lsusb.txt
lsusb -t >> lsusb.txt