Tiny Core Linux
Tiny Core Base => TCB Talk => Topic started by: Stefann on March 15, 2026, 12:15:12 PM
-
Well,
I updated my Via Eden 500MHz single core system.
I succeeded the upgrade and my homecontrol application is running agin.
- including samba, ssh and vnc so that is nice.
Some things....
1/
After updating the applications I got this error:
error while loading shared libraries: libusb-1.0.so.0: cannot open shared object file: No such file or directoryI was able to fix that by adding libusb.tcz to the onboot.lst
The system is coming from TC15. with TC15 I only had libusb-compat-dev.tcz which apparently did load libusb.tcz as a dependancy but now it does not do that anymore.
Anyways... I got it fixed
[edit] NO... its not working. need to dig deeper
2/ rsyslog is not working
I'm starting it from bootlocal.sh and I tried to run it manual but than it complains its already running:
tc@huis:/opt$ sudo /usr/local/sbin/rsyslogd
rsyslogd: pidfile '/var/run/rsyslogd.pid' and pid 4789 already exist.However... nothing gets logged.
I did set debugging to level 2 in the rsyslog.conf file
$DebugFile /var/log/log_debug.txt
$DebugLevel 2 # 0=off, 2 = full debugAfter a reboot I indeed get an enormous /var/log/log_debug.txt file
However... nothing obviously wrong
But no logging
My rsyslog.conf file:
$WorkDirectory /var/log
$ModLoad imudp
$UDPServerRun 514
$ModLoad imtcp
$InputTCPServerRun 514
$ModLoad imklog
$ModLoad imuxsock
$template MyFormat,"%timegenerated:1:19% %msg:::drop-last-lf%\n"
$template FullFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"
$DebugFile /var/log/log_debug.txt
$DebugLevel 2 # 0=off, 2 = full debug
#example
# $outchannel log_rotation,/var/log/log_rotation.log, 52428800,/home/me/./log_rotation_script
# activate the channel and log everything to it
# *.* :omfile:$log_rotation
#
# /home/me/log_rotation_script:
# mv -f /var/log/log_rotation.log /var/log/log_rotation.log.1
# ------------- User section -------------------
# max 100k debugfile = 1000 lines x 100chars
# Need script: /var/log/rotate:
# tail -n 500 /var/log/${1} > /var/log/${1}.tmp
# cat /var/log/${1}.tmp > /var/log/${1}
# rm -f /var/log/${1}.tmp
# Heat Control: HeatPump and CV
$outchannel heat_log, /var/log/heatlog.txt, 100000, /var/log/rotate heatlog.txt
local1.=info :omfile:$heat_log;MyFormat
# Solar Thermal
$outchannel solar_log,/var/log/solarlog.txt, 100000, /var/log/rotate solarlog.txt
local1.=notice :omfile:$solar_log;MyFormat
# Main (normal main logging)
$outchannel main_log, /var/log/mainlog.txt, 100000, /var/log/rotate mainlog.txt
local1.=debug :omfile:$main_log;MyFormat
# EVSE
$outchannel evse_log, /var/log/evselog.txt, 100000, /var/log/rotate evselog.txt
local2.=notice :omfile:$evse_log;MyFormat
# Weather
$outchannel weather_log, /var/log/weatherlog.txt, 100000, /var/log/rotate weatherlog.txt
local3.=notice :omfile:$weather_log;MyFormat
# PV, enphase and solaredge
$outchannel PV_log, /var/log/PVlog.txt, 100000, /var/log/rotate PVlog.txt
local3.=info :omfile:$PV_log;MyFormat
# Debug, catch all that have not been catched before
$outchannel debug_log,/var/log/debuglog.txt, 100000, /var/log/rotate debuglog.txt
Any tips??
[edit]
I reverted back to original /mnt/sda1/tce/optional folder
So running TC17 updated core.gz and vmlinuz only. no tcz extension updates.
==> this runs fine!
libusb and rsyslog both working
-
Note,
on the libusb...
I have the feeling the file "may be there", in /usr/local/lib
But can probably not be found in runtime.
I checked and the libusb tcz applications did not change (did they?)
So its probably a search path
I cannot check this assumption at this moment because I reverted back for reason the system is running my home automation and I cannot leave that in a non-functional state. I will need to find time to continue tomorrow.
But..... anyone who triggers on this and has some hint towards "search path" on /usr/local/lib libraries is welcome.
Note... I cannot imagine it's the gcc compiler as I'm not compiling anything. I'm just running an executable that can suddenly not find the libusb-1.0.so.0
-
Ref:
cat /etc/ld.so.conf
/usr/local/lib
..tells the loader to look in /usr/local/lib as well as /lib and /usr/lib, which are searched by default.
If a library has been manually added the following command is required before the library will be found: sudo ldconfig
-
Thanks,
It’s strange because “just upgrading core.gz and vmlinuz” works. It derails with the application update.
I will have to go step by step tomorrow.
Takes quite some time because after each step I need to reboot and carefully test.
A full application update is also not superfast. Took about 20 mins today.
I will post progress.
-
SOLVED!!
[edit] almost... usb-serial is not working now but it worked earlier
sooo.....
My target system is a Via Eden 500MHz single core very low power 32bit cpu running my home automation.
I also have a dual core Via Eden 1GHz cpu system that I normally use as test system.
I had TC17 already running on that test system but had not verified libUSB and rsyslog.
So what I did today:
- bring latest version of my application to the test system and try to run it on TC17
- it appeared to work with both libUSB and rsyslog.
==> Now I got suspicious towards update corruption on my target system
I did a compare of onboot.lst between the systems -> they where identical
I did a compare of the tce/optional folder between systems -> that did show major differences
I copied tce/optional folder from test system to target system
I did a reboot of target system
BANG.... all working. Application fully runs. rsyslog works. libUSB works
So...
What I suspect...
- To do the application update on the target system I used the gui over VNC as I donot feel very comfortable with the command line commands for apps-maintenance.
- the gui over vnc however already takes 40% cpu load of this "not so powerful 500MHz single core cpu"
- running the apps update took about 20mins during which the gui was completely frozen
- Afterwards I did a recheck on update and "it showed no lost items"
But....
- I expect the upgrade process had issues
- and although the gui for apps-maintenance is handy, with this slow processor it does not allow proper checking of logging on "whether all went ok"
- issues I wold consider "not super strange". I basically ran the system on close to 100% cpu use for 20mins. Disk being a compact flash card.
Anyways... result...
I can confirm TC17 runs on Via Eden 500MHz 32bit single core cpu
- including samba
- including ssh
- including vnc
- including libUSB
- including gcc (yes I tested a recompile of some part)
- including rsyslog
[edit]
still problem with usb-serial-6.18.2-tinycore.tcz kernel extension
I used to get text-data but when I run
cat /dev/ttyUSB0
I now get binary characters.
I guess it has something to do with settings
It's a bit strange as this 6.18.2 version DID work on TC17 core.gz and vmlinuz before I did upgrade all other apps.
I will figure this out I think
---------
Thanks for the replies & reading,.. I was a bit stuck yesterday.
Upgrading is always kind of a headache as its is a quite "big" thing
Very beneficial about upgrading of tinycore is that a simple rename of core.gz, vmlinuz, tce/optional and tce/onboot.lst allows you to switch between upgrade and original. VERY convenient!!!!
going forward..
- On a next update I will use the command line tools.
- That will give me better logging
- and NOT consume 40% cpu for vnc
-
I can no longer edit the previous post
but.. usb-serial-6.18.2-tinycore.tcz kernel extension now also works
for some reason the baudrate setting got lost
Its a bit strange because that is set on every start of my application and I did restart it multiple times
But anyways... It now runs
Thats probably an "application weakness"
ALL WORKS
and thanks for the reading & helping
-
Hi Stefann
... - On a next update I will use the command line tools. ...
Tinycore has a single command that handles that. It's called:
update-everything
I can no longer edit the previous post ...
That's right. Years ago we added a 30 minute time limit after
which you can no longer edit a post.
The issue was twofold:
1. When you post, it shows up as a new post. When you edit a post
someone already read, there is no obvious indication it was changed.
2. We had individuals changing information in posts that made replies
to that post look irrelevant.
If you need to update information, add a new post.
If you have an important correction, for example, a command line typo that will
cause the command to fail or do something bad, add a new post requesting it
be corrected and it will be handled.
Either way, readers will remain informed.
-
A...
I can now use some serious help....
As said in above posts, I succeeded to upgrade my Via eden 500MHz single core cpu 32bit system to TC17.
Everything works. But...
- After about 2 days it crashed. Application crashed, ssh crashed, VNC crashed. It's headless so the only thing I could do is a power cycle.
That "could have been a glitch"... but 2 days later it crashed again. I did just reboot as well.
Until last weekend the system did run on TC15. It could run multiple month without problem. In fact, I have not seen a crash since I started using TC 2 years ago.
So.... It very very very likely has something to do with the upgrade
(apart form TC17 this could also be human error or flash-card issues because I suddenly created big backup files).
The system runs my home automation.
- C program
- some PHP scripts
- some python scripts
- apache server
- vnc server (but currently no connected clients)
- samba
- ssh
- serial port over USB interface (with kernel extension)
Let's refer this as the "live system"
Apart from this system, I have a 2nd system that I use as a test setup. This system is a Via Eden 1GHz 2core cpu.
It's also upgrade to TC17. It already runs TC17 for some weeks without any glitch.
But..... it does not run the application.
Lets refer this to be the "test system"
I need to think of a clever plan to trouble shoot this:
- run the application on the "test system". It will not be able to connect to all peripherals but it will at least run code.
- step by step revert back the "live system". probably 1st run with TC17 core and vmlinuz but still non-upgraded applications.
any tips on how I can put logging in place?
- problem is that everything runs in ram. So I have zero diagnostics in case of a crash.
I probably have to connect a monitor ??? to give me some hint in case of crash?
Any other trouble shoot ideas?
(I'm not expecting anyone to have an idea on rootcause at this stage, just tips on how to approach this would be very helpful)
-
Hi Stefann
... any tips on how I can put logging in place?
- problem is that everything runs in ram. So I have zero diagnostics in case of a crash. ...
Yes, but your extensions are on persistent storage, right?
So send any log messages to /etc/sysconfig/tcedir/SomeFileName.log
-
With everything running in RAM and you backups suddenly getting bigger, maybe you're running out of memory? Does the test system have more memory than the live system?
-
Hi Stefann
... any tips on how I can put logging in place?
- problem is that everything runs in ram. So I have zero diagnostics in case of a crash. ...
Yes, but your extensions are on persistent storage, right?
So send any log messages to /etc/sysconfig/tcedir/SomeFileName.log
True… but “what should I log”?
My application does log but just progress things
With everything running in RAM and you backups suddenly getting bigger, maybe you're running out of memory? Does the test system have more memory than the live system?
No, backups are not suddenly bigger.
When going to tc17 I made copy of tce/optional folder. That occupies about 280M diskspace. But still 3G available.
-
Oh by the way @rich…
I’m normally logging to ramdisk.
I normally do 1 single status diskwrite per day
For reason I have flash disk. I want to keep that healthy.
A bit extra temporal disk logging is ok.
But I don’t want mass logging on flash
-
For those interested, output of “top”.
I don’t see anything suspicious.
My home automation application is: /krubo/work/krubo /krubo/work/1wire.def
Anyway…
It’s late now, I restarted the application and have a sleep.
Tomorrow I will undo the apps update but keep the tc17 core.gz and vmlinuz.
I will also start the application on my test system without pheripherals
Than run that for few days to see whether it holds.
If it fails I will go back to tc15 to see whether it still runs on tc15.
It has been running for multiple month on tc15 without a glitch but all the activity in upgrading may have degraded some old hardware (maybe the flash card isn’t healthy anymore?)
If tc15 works and tc17 does not, I will test tc16.
Mem: 410040K used, 551480K free, 34800K shrd, 4412K buff, 204960K cached
CPU: 5.4% usr 6.4% sys 0.0% nic 88.1% idle 0.0% io 0.0% irq 0.0% sirq
Load average: 0.31 0.23 0.16 2/197 4900
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
4130 1 tc S 11572 1.2 0 10.6 /krubo/work/krubo /krubo/work/1wire.def
3481 1 root S 54508 5.6 0 0.3 /usr/local/sbin/rsyslogd
4880 4849 tc R 3600 0.3 0 0.3 top
4014 1 root S 8192 0.8 0 0.2 x0vncserver -PasswordFile=/home/tc/.vnc/passwd
4848 4844 tc S 7968 0.8 0 0.2 sshd-session: tc@pts/0
4017 3993 tc S 442m 47.0 0 0.0 /usr/local/sbin/httpd -k start
4019 3993 tc S 395m 41.9 0 0.0 /usr/local/sbin/httpd -k start
4050 3993 tc S 391m 41.6 0 0.0 /usr/local/sbin/httpd -k start
4314 3993 tc S 382m 40.6 0 0.0 /usr/local/sbin/httpd -k start
3993 1 root S 162m 17.2 0 0.0 /usr/local/sbin/httpd -k start
3995 1 root S 44968 4.6 0 0.0 Xvesa -br -screen 1024x768x32 -shadow -2button -mouse /dev/i
3821 3798 root S 18868 1.9 0 0.0 /usr/local/sbin/smbd -D
3798 1 root S 18812 1.9 0 0.0 /usr/local/sbin/smbd -D
4120 1 tc S 9852 1.0 0 0.0 wbar
4003 1 tc S 9628 1.0 0 0.0 flwm
4844 3855 root S 7564 0.7 0 0.0 sshd-session: tc [priv]
3855 1 root S 7312 0.7 0 0.0 sshd: /usr/local/sbin/sshd [listener] 0 of 10-100 startups
3641 1 root S 6056 0.6 0 0.0 /usr/local/sbin/nmbd -D
-
Hi Stefann
... True… but “what should I log”? ...
Well, I would start with syslog to capture any system errors. If you included
the syslog boot code in your boot loaders config file, remove it.
Add this to /opt/bootlocal.sh:
/sbin/syslogd -O /etc/sysconfig/tcedir/messages -s 1000 -b 5 -l 5-O The log files will be messages, messages.0, ..., messages.4.
-s Each file will be limited to 1000K (1 Mbyte) before rotating to the next log file.
-b There will be 5 rotated logs kept for a total of 6Mbytes.
-l Log only messages of Warning or above.
These are the syslog message levels:
Level Severity Description Example Use Case
0 Emergency System is unusable Hardware failure causing a full system crash
1 Alert Immediate action required Database corruption detected
2 Critical Critical conditions Firewall dropping all traffic unexpectedly
3 Error Error conditions Authentication failures
4 Warning Potential issues High CPU or memory usage
5 Notice Normal but significant events Configuration changes applied
6 Informational General system activity User logins, completed backups
7 Debug Debugging information Function traces, variable dumps
You could also run free once every 5 minutes to keep a log of memory and swap usage.
Create a file called FreeMem.sh under .local/bin in your home directory and make it
executable (chmod 755 .local/bin/FreeMem.sh).
Place the following in that file:
#!/bin/sh
Dest="/etc/sysconfig/tcedir/FreeMem.log"
while true
do
echo -ne "$(date)\n$(free -m)\n\n" >> "$Dest"
# Limit log file to last 2400 lines (400 entries) if it is greater or equal to 2460 lines (410 entries).
[ $(wc -l "$Dest" | cut -d " " -f1) -ge 2460 ] && echo "$(tail -n 2400 $Dest)" > "$Dest"
sleep 300
doneThe log file will hold about 33 hours worth of entries and be about 100 Kbytes in size.
To launch it, enter this into /opt/bootlocal.sh:
/home/tc/.local/bin/FreeMem.sh &This assumes you are running as user tc. Change tc to your user name if required.
[Edit]: Changed messages.1, ..., messages.5 to messages.0, ..., messages.4. Rich
-
Thanks for the suggestions. It helps.
It’s now 7am here in the Netherlands.
When I got up the system had crashed again. After about 8 hours.
Everything halted. Apache server not reachable, ssh not reachable.
So..
- Either the system is completely crashed
- Or the network has crashed
Some “non network peripherals” are also dead which indicates that the application is frozen. As far as I have tested earlier the application keeps running if network drops so I guess most changes are that system has completely crashed.
So.. it’s a bit chilly here now because the heatpump has not been operational this night.
Although the whole thing could be hardware related as the TC upgrade process required me to do a lot more disk actions that this system normally does, my big suspecting is that it is TC17 related (or TC16 as it was running TC15).
That I expect makes it a valid entry for this thread (if I’m polluting something feel free to correct me on that).
What I will do:
- run the thing on TC17 core.gz and vmlinuz but with all TC15 applications (.tcz files).
- include syslog logging as suggested by @rich. Thanks for that! I’m normally using rsyslog and use the logging for application logging. I will however revert to the original syslog and use the commands as you suggested. I will plug-in an external usb and write to that. By doing so I will avoid write-stress on my compact flashdrive. I will pause all application logging for a while. That’s no big deal, I only need that for application development and that’s on halt now anyway.
- in addition start running the application without peripherals on my “test system” on TC17. I will also run the python and php “helper programs” on that. Those helper programs grab data from internet api’s (get weather info, get energy prices, read my car status). As this interaction is in a “less controlled environment” it could be that the issue is in those. It’s still very very very strange that any application failure can completely crash the OS.
- find and connect my old monitor and keyboard somewhere in the mess at the attic. I kept these for especially this reason (Hope I can find them). If the OS is not crashed but only the network that should allow me to operate the system after crash.
Guess that should bring me somewhere.
Again especially thanks @rich.
I had to think on “how to log without wearing out my compact flashdrive” but your suggestion is very key to “not being blind”.
Any additional suggestions are welcome.
-
Update,
Its running as I said in previous post
- Life system running on TC17 core.gz & vmlinuz but non-upgraded .tcz applications
- disabled rsyslog
- /sbin/syslogd -O /sdb1/logging -s 1000 -b 20 -l 5
So this writes to external usb, not to built-in compact flash
- test system running fully on TC17 and also running the application but without peripherals
- test system ALSO running crontabs for php scripts that get weather data, energy data, and solar panel readout
I notice that the syslog reports every crontab entry execution with a cron.err. The entries are correctly executed so I dont understand why it reports this as an "error" as it basically is more a "notice" but maybe thats just syntax. For now I ignore it. scripts are correctly executed so "no issue"
To my relieve the logging from syslog is "not overly big". No immediate worry that I break the usb drive within an hour. Breaking it is not a problem but that would stop the logging and that would bring me nowhere.
wait and see.......
-
Hi Stefann
... To my relieve the logging from syslog is "not overly big". No immediate worry that I break the usb drive within an hour. Breaking it is not a problem but that would stop the logging and that would bring me nowhere. ...
The commands I listed limit the space the logs will occupy so they can not fill your disk.
The total size from syslog will be capped at around 6 Mbytes.
It first creates /sdb1/logging.
When that grows to 1 Mbyte, it moves /sdb1/logging to /sdb1/logging.0
The next time /sdb1/logging grows to 1 Mbyte:
it moves /sdb1/logging.0 to /sdb1/logging.1 and /sdb1/logging to /sdb1/logging.0
Eventually you will have 6 files:
logging, logging.0, logging.1, logging.2, logging.3, and logging.4.
When logging again reaches 1 Mbyte, syslog rotates the logging files like this:
remove logging.4, move logging.3 to logging.4, move logging.2 to logging.3,
move logging.1 to logging.2, move logging.0 to logging.1, and move logging to logging.0.
This logging file rotation will occur every time the logging file grows to 1 Mbyte in size.
The script I showed that executes the free command every 5 minutes also limits file size.
Every time that log file reaches a length of 2460 lines, the oldest 60 lines get deleted.
This limits the size of that file to about 100 Kbytes.
-
update,
"live system" crashed again after about 17 hour runtime.
- running on TC17 core.gz and vmlinuz but with non-upgraded .tcz applications. Note, 1 exception, usb-to-serial kernel extension IS upgraded
- my home automation application running with several php & python scripts.
"damn"...
I used syslog to write to external usb: /sbin/syslogd -O /sdb1/logging -s 1000 -b 20 -l 5
But...
After restart de logging was gone. System started a new "logging" file.
In hindsight I should probably have removed the usb and read it with an other system.
Going forward..
- I remove the syslog start from boot local and mount and start it manual
- that ensures that after a reboot the usb is "untouched"
It "may" be that the logging file is not overwritten by the new syslog but had disappeared / got corrupted at the crash. We'll find out I guess.
[edit] a... I understand... I forgot to remount the drive before writing to it > fixed
On statistics:
I thought system could run more than a day but that is not true:
- Tuesday crash after about 12 hours
- Wednesday crash after about 13 hrs
- today crash after 17 hours
So, in all cases below 24hr
note..
The "test system" still runs. Running the application without peripherals. On full TC17 with full upgraded applications.
-
Hi Stefann. I'm sorry to hear about your unexplained crashes.
Twice in the past I had problems with crashes after a TCL upgrade. Both times, the crashes were in the laptop that I repurposed as a wireless router. I'll share my experience with those problematic upgrades in case they are helpful to you.
1. One of the problematic upgrades (when I upgraded the wireless router from TCL13 x86_64 to TCL14 x86_64) turned out to be a regression in the mac80211 kernel module, which affected my specific wireless hardware. It is possible that your problem is not due to TCL17 applications, but due to a regression in linux 6.18.2--either the kernel itself or a module--that is affecting your hardware.
2. The second time I experienced wireless router crashes after a TCL version upgrade (I don't remember which TCL version), the crashes stopped after I recompiled the workhorse applications in my router, namely hostapd and dnsmasq. Maybe it would be helpful for you to make a list of the services that are always running on your machine, then check the "Current:" field in the .info file to see if the relevant extensions were created after TCL17 was released. If one or more of your workhorse applications/services were built before TCL17 was released, I would try rebuilding those extensions on TCL17.
I hope the above is helpful. Those were the only two times I had crashes after upgrading TCL to a new version.
-
Hi Stefann. One way to explore possibility #1 from my post above (Reply #18) is to try booting TCL17 with a different/older kernel and modules copied from another distro. A while back I found a hacky way to do it. It's ugly as heck and I'm not proud of it, but here it is: https://forum.tinycorelinux.net/index.php/topic,26071.msg181848.html#msg181848
A less hacky way might be to use core.gz and applications from TCL17 but vmlinuz and foo-KERNEL extensions from the last TCL that worked for you without any problems (if I read your posts correctly, that was TCL15). Note that some kernel modules are provided by the base system, so you would have to add /lib/modules/TCL15KERNELVERSION to the TCL17 core.gz.
-
Hi @GNUser
Thanks for this info!!!! It helps!!
I was already thinking a bit in that direction, your experience underlines it.
I would find it a bit weird that the standard linux has a problem. I would expect that to have surfaced way earlier.
I'm using 1 kernel extension: usb-serial-#-#-#-tinycore.tcz
This extension allows to connect a RS232 connection over USB port. It's connected to my energy meter where it reads data every second.
This extension is likely way less thoroughly tested during the TC17 beta-phase so it would make sense that a bug on this slipped through. And it connects with your experience.
At this moment I'm again running the application and I keep it running to see whether I can catch some logging at next crash. It's running now for 7 hours so "statistically" it should crash in about an other 7 hrs.
Thanks for the info.. because this motivates the next test to be:
- "remove the kernel extension and not use the peripheral" for next testrun.
I can actually bring this peripheral to my "test system" and see whether it works there.
"test system" is now running with full TC17 for over 24hrs now. It has the same kernel extension installed but not used.
Would be a very clear pointer if I can get the "test system" to crash with this usb-serial peripheral.
-
- "remove the kernel extension and not use the peripheral" for next testrun.
...
Would be a very clear pointer if I can get the "test system" to crash with this usb-serial peripheral.
Hi Stefann. Having a log capture an error at the time of the crash would be the best, of course, but the above also sounds like a excellent strategy.
If you can poinpoint usb-serial-KERNEL.tcz as the cause of your crashes on TCL17, maybe the kernel mailing list or other users will point you to a workaround or fix/patch. (When I had my problem with the mac80211 kernel module in the wireless-KERNEL.tcz extension, it was actually other knowledgeable users--over at https://github.com/morrownr/USB-WiFi --who pointed me to a patch. I don't know if there is a similar waterhole for users who know everything about usb.)
-
Update..
It’s still running after about 22 hours. That’s much longer than previous runs.
That’s actually “not good” because it’s clearly unreliable and it needs fix.
@GNUser, yes I try to catch error. That’s why I keep it running.
But when it crashes I will restart without the usb-serial kernel extension and let that extension run on my test system. Not alone will that isolate the error, it will also bring my “live system” in a more reliable state as it is indeed live controlling my house. Thursday morning system had crashed and heating had been off all night.
I should have error logging than.
With that said..
I more and more see reasons that it’s indeed de usb-serial kernel extension as it is a bit vulnerable.
It reads data from my energy meter but it opens the connection and keeps reading without ever closing. So apart from improving the kernel extension I could modify my code to close the connection after each data block.
I will try to catch the error but if it’s pinpointed to this than that’s what I will do.
So far it was never unreliable though.
The energy read using serial read over usb has been opererational on DSL from 2009 to 2024 and than on TC15 when I moved to that in summer 2024. System runs 24/7 and after initial debugging it never crashed in at least last 10 years.
-
Hi Stefann
... It reads data from my energy meter but it opens the connection and keeps reading without ever closing. ...
That makes me wonder if any USB devices are being put to sleep.
When a USB device is idle for 2 seconds, the kernel suspends it to save power.
... This extension allows to connect a RS232 connection over USB port. It's connected to my energy meter where it reads data every second. ...
Yeah, I saw that too. But this isn't a real time kernel. So it's possible that if the
system becomes busy enough, every once in a while that read may be delayed
long enough that the kernel sees it as idle.
The easiest way to test this is to disable usbcore.autosuspend.
Add this boot code to your boot loaders config file:
usbcore.autosuspend=-1
Since usbcore is built into the kernel, you can not unload it and
then reload it with autosuspend disabled.
you can disable it in a running kernel like this:
echo -1 | sudo tee /sys/module/usbcore/parameters/autosuspendBut in order for it to take effect on existing devices, you would probably
need to tell udev to re-scan the system:
sudo udevadm trigger
Personally, I feel the boot code option is the better way to test this.
-
@rich,
Great great great!
Thanks!
To be more precise,
The energy meter sends a continuous stream of “telegrams”. It just sends them, you don’t have to request them. It’s a broadcast. It’s called the “DSMR” protocol (Dutch Smart Reader Requirements).
Until march-4 I had a dsmr 2.2 meter that spits out data every 10 seconds. It got replaced on march 4 by a dsmr 5 meter that spits out data every second.
As said, I’m interfacing with this already sinds 2009 so it’s real strange it stops working now.
Your comment however makes perfect sense!
Although the data was transmitted every 10 seconds and now every second, I’m polling the usb about every 300ms, well within the suspend timeout. This however is NOT a real-time method. It’s just an endless service-loop for my home automation. It is very well possible that “occasioanllly” some other service takes 1 or 2 seconds.
So yes… this could be it.
With that said….
IF it’s indeed this interface (which is not yet proven), than it’s certainly Linux should not hang completely. I would totally understand the application to crash, but not the full OS including ssh, Apache, VNC, samba.
But… nothing is proven yet. At this moment it’s a strong but non-proven theory.
The system is still running. After 3 crashes that all took place after about 14 hrs it’s now already running for more than 24hrs. I keep letting it run. I have setup your syslog “trap” and really want to see whether I can catch some logging.
Thanks for the method to disable suspend. I will certainly make that part of my test-approach. I will use the boot-option.
-
Hi @rich,
I'm using extlinux.conf
Is this how I should modify it? (added last line, other lines is current)
Or should it be part of the "APPEND" string? or different?
DEFAULT gui17
LABEL gui17
KERNEL /tce/boot/vmlinuz17
INITRD /tce/boot/core17.gz
APPEND quiet host=huis cron tz=CET-1CEST,M3.5.0,M10.5.0/3 waitusb=5:UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113" tce=UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113"
usbcore.autosuspend=-1
Thanks
-
On the long APPEND string. Not on it's own line.
You can also see the kernel command line when booted with:
cat /proc/cmdline
-
Hi Stefann
Insert it right right between quiet and host=huis.
The contents of APPEND must be 1 continuous line.
You can also see the kernel command line with:
showbootcodes
-
Thanks @Rich & @patrikg.
Will do that.
Note, I start getting the feeling that 1st 3 crashes where not "typically around after 14hr duration" but "typically around 24:00"
Which connects to the usb-suspend theory because at 24:00 the application does some housekeeping which can cause a larger loop-time.
Thank "last night it missed that slot" but that is also not strange,. It's not 100% deterministic.
So.. Will see whether I have a crash upcoming midnight.
-
Update,
This morning the system was still running.
In itself "I like that it runs", but as it had failed 3 times earlier I actually DON't like it because this thing is supposed to be robust enough to run for multiple month without hiccup.
But anyways...
- I manually stopped it
- I brought the tce/optional folder to full TC17 status so system is now 100% TC17
- I removed the external usb that I used for syslog to write to
- I reconnected my X10 peripheral to that usb-port as I had to remove that few days aago to make room for the usb stick.
- I reduced the amount of cron_actions to bare minimum for reason every cron_action triggers the syslog and I want to minimise write wear of my compact flashdrive
- I did setup the syslog logging as suggested by @Rich to write to internal compact flash in stead of external usb.
So...
- I'm actually in full operational mode now
- but WITH logging enabled to persistent storage such that I can catch error logging in case of crash
Note: I did NOT insert "usbcore.autosuspend=-1" in boot line yet. I try to let it crash and collect logging before I do that.
-
Update,
yes.. crash again. In about 14 hours.
Apart from having the application .tcz files updated I also reconnected the X10 peripherals over usb.
I have no idea how that interface looks like as I made that in 2009. I DO remember that is was a bit of a "dirty from the internet code". There was no GitHub at that time.
Unfortunately there is no logging of the crash.
Here you see the logging I haver. Its basically just logging the crontab
crash somewhere between 22:00 and 23:00
Mar 23 19:31:00 huis cron.err crond[3358]: USER tc pid 4738 cmd php /home/tc/php/knmi.php
Mar 23 20:01:00 huis cron.err crond[3358]: USER tc pid 4760 cmd /krubo/work/krubo_watchdog
Mar 23 20:31:00 huis cron.err crond[3358]: USER tc pid 4783 cmd /krubo/work/krubo_watchdog
Mar 23 20:31:00 huis cron.err crond[3358]: USER tc pid 4784 cmd php /home/tc/php/knmi.php
Mar 23 21:01:00 huis cron.err crond[3358]: USER tc pid 4803 cmd /krubo/work/krubo_watchdog
Mar 23 21:31:00 huis cron.err crond[3358]: USER tc pid 4823 cmd /krubo/work/krubo_watchdog
Mar 23 21:31:00 huis cron.err crond[3358]: USER tc pid 4824 cmd php /home/tc/php/knmi.php
Mar 23 22:01:00 huis cron.err crond[3358]: USER tc pid 4844 cmd /krubo/work/krubo_watchdog
used this command as suggested by @rich for logging:
/sbin/syslogd -O /krubo/work/logging.txt -s 1000 -b 20 -l 5
Anyway.. I just restarted the application.
Will do with different bootcode tomorrow.
Sleeping time now.
[edit] could not resist and did a quick peek into the x10 code. Its basically using the libusb1 library for interfacing.
-
Hi Stefann
... Unfortunately there is no logging of the crash.
Here you see the logging I haver. Its basically just logging the crontab ...
If you want to stop crond from spamming your log file:
tc@E310:~$ crond --help
BusyBox v1.29.3 (2018-12-19 15:29:37 UTC) multi-call binary.
Usage: crond -fbS -l N -d N -L LOGFILE -c DIR
-f Foreground
-b Background (default)
-S Log to syslog (default)
-l N Set log level. Most verbose 0, default 8
-d N Set log level, log to stderr
-L FILE Log to FILE
-c DIR Cron dir. Default:/var/spool/cron/crontabstry starting crond with the -L option and point it to another file, or
to /dev/null if you prefer to send those messages to a black hole.
used this command as suggested by @rich for logging:
/sbin/syslogd -O /krubo/work/logging.txt -s 1000 -b 20 -l 5
Then you should be able to reduce the number of log files kept
back to 5 by changing -b 20 back to -b 5.
-
All,
New day, new chances.
So,... the system did run through the night. As the crash was yesterday between 22:00 & 23:00 it's not related to "24:00 maintenance".
The kind of strange thing is that the system did run for 3 days without crash and now it again crashes after about 14hr runtime.
Only differences:
- removed usbstick for external logging
- reconnected X10 USB interface that uses libusb-1.0 library form my C-language application
- upgraded .tcz applications to TC17 level again
I "kind of feel without evidence" that it's not the .tcz applications. I still suspect the usb stuff
Anyways, restarted the system with "usbcore.autosuspend=-1" boot option.
Here is my applicable extlinux.conf section
LABEL gui17tst
KERNEL /tce/boot/vmlinuz17
INITRD /tce/boot/core17.gz
APPEND quiet usbcore.autosuspend=-1 host=huis cron tz=CET-1CEST,M3.5.0,M10.5.0/3 waitusb=5:UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113" tce=UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113"
showbootcodes:
tc@huis:/krubo/work$ showbootcodes
BOOT_IMAGE=/tce/boot/vmlinuz17 quiet usbcore.autosuspend=-1 host=huis cron tz=CET-1CEST,M3.5.0,M10.5.0/3 waitusb=5:UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113" tce=UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113" initrd=/tce/boot/core17.gz
I will likely have to manually halt the system a few times in upcoming 2 days for reason I get a home-battery installed.
I will definitely keep this thread updated but probably no news in upcoming 2..4 days.
@Rich, thanks for the directions on avoiding cron-logging. I will for now not do that for reason I have minimised the entries towards low flash-wear anyways. And having some logging helps to get timestamps for start&crash. Very likely I do something like that at a later stage.
With that said... "any tips to get better crash logging"?? I basically have zero logging from last crash. I probably need to set -l to 6 instead of 5.
Will try that later. Lets see how the "usbcore.autosuspend=-1" works out for now.
[Edit]: Adjusted first showbootcodes code tag. Rich
-
Hi Stefann
... With that said... "any tips to get better crash logging"?? I basically have zero logging from last crash. I probably need to set -l to 6 instead of 5. ...
Not at the moment. you seem to have it figured out for now.
I initially went with -l 5 because it seemed like a reasonable
compromise between message severity and log activity. It's
possible you may have to turn up that setting for more info.
-
Small update.
I had several manual stops over past few days for reason I got a home battery installed.
However..
WITH the “disabled usb suspend” boot code and fully on TC17 the system did run for 38hrs from Wednesday 20:00 until this morning (Friday) 10:00.
I had to manually stop the system this morning for intended maintenance.
So…
- it’s not a complete proof yet.
- but given that all 3 earlier runs in this exact same configuration earlier crashed after about 15hrs it’s is a very positive sign.
At this moment I did not add additional logging for reason I donot want to stress my compact flash drive. Earlier I modified things to write to a usb stick but that required me to remove a peripheral to free-up a port and the system got more stable. So… “the observation changed the behavior”.
I COULD include a usb-hub to allow both the usb-stick and the peripheral bit that would again change the configuration.
So…
For sure I will do such testing later but for now I just want see whether the “disabled usb suspend” boot code solves the issue.
So… I will try to run it for at least a week. Likely 2 weeeks because I will be traveling 2-10 April.
With that all said……
Does anybody know where I can find the source code of usb-serial-6.18.2-tinycore.tcz ??
I would like to compare that to usb-serial-6.6.8-tinycore.tcz as that was running fine without “disable usb suspend” boot code on TC15
That an application crashes for reason of timeouts is kind of “acceptable behavior”.
That Linux core crashes is “not really”.
-
Hi Stefann
... Does anybody know where I can find the source code of usb-serial-6.18.2-tinycore.tcz ?? ...
That's a kernel driver.
You would have to look under drivers/usb/serial/ in the kernel source package.
-
Hi Stefann
... Does anybody know where I can find the source code of usb-serial-6.18.2-tinycore.tcz ?? ...
That's a kernel driver.
You would have to look under drivers/usb/serial/ in the kernel source package.
Thanks.
pfew.... that is not simple....
I guess I focus on getting some logging first.
-
Hi Stefann
... pfew.... that is not simple....
Yeah, there are 80 source files in there. I would have mentioned that
sooner, but I didn't want to spoil the surprise. :)
-
usb/serial is not so bad, Only a few files in that directory are related to your specific chipset. Did you ever say which kernel module was being loaded?
Your best bet would be to connect to the kernel git (https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git) or one of the github clones. and then look for driver updates or compare between branches. USB is a convoluted monster, you have usb/serial for the endpoint driver, but its also going to have drivers/usb/core /drivers/usb/dwc2 (USB2) or drivers/usb/dwc3 (USB3) For the USB host adapter and core.
But yes, logging is important to see if you can find which rabbit hole to start looking. Did you ever tell us?
Which USB serial driver is being used (Also the USB IDs)
What the USB host adapter is (USB2 or USB3) and USB IDs of it.
IS there a USB hub involved?
Everything related to USB in dmesg would be useful. (And turn up the KERNEL Logging level in cmdline.txt ) and of course anything syslog picks up from the kernel from that logging.
-
@rich,
no worry.
Not sure whether I will be able to "do something usefull" on finding a bug,
But at least I get my feet wet.
Note....
Reminds me of when I started my homecontrol project in 2008.
I have a background as "control engineer" and I liked the idea of designing my own intelligent control system.
I than learned the hard way that "control logic" is about 2% of the whole thing.
Next to all hardware, electronics but also plumbing, I did choose the "maxim 1wire components" for switches and temperature sensors.
They came with a SDK that included all usb-to-1wire drivers so I thought I would have a nice time.
However..... the sensors behaved 10x to 50x slower than harware specifications had promised.
So.... I started to look into the driver software and discovered a real nightmare of "bad code".
I ended up rewriting the USB drivers.......
Successfully, I got it to HW promised speed level.
Soooooo in theory I should know "a little bit" about this.
In practice this knowledge is about 17 years old and not maintained....
Anyway... it's a hobby project... so I like to explore.
-
usb/serial is not so bad, Only a few files in that directory are related to your specific chipset. Did you ever say which kernel module was being loaded?
Your best bet would be to connect to the kernel git (https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git) or one of the github clones. and then look for driver updates or compare between branches. USB is a convoluted monster, you have usb/serial for the endpoint driver, but its also going to have drivers/usb/core /drivers/usb/dwc2 (USB2) or drivers/usb/dwc3 (USB3) For the USB host adapter and core.
But yes, logging is important to see if you can find which rabbit hole to start looking. Did you ever tell us?
Which USB serial driver is being used (Also the USB IDs)
What the USB host adapter is (USB2 or USB3) and USB IDs of it.
IS there a USB hub involved?
Everything related to USB in dmesg would be useful. (And turn up the KERNEL Logging level in cmdline.txt ) and of course anything syslog picks up from the kernel from that logging.
THANKS!!!!
I'm not sure I understand all you say...
But at least some answers...
- not using a hub
- using usb-serial-6.18.2-tinycore.tcz
- it's basically a RS232 protocol connected to a usb by a serial-to-usb cable
- USB2 (maybe 1??) it's on a 500MHz via Eden single core cpu from 2008.
- Probably to make it very clear: I get all serial data in my terminal when I simply type:
cat /dev/ttyUSB0- The data is send as broadcast by my energy meter (using DSMR5 standard, Dutch Smart Meter Requirements). You donot have to request it. It just spits out a report every second.
As said... For now I will just keep the thing running and see whether the "usbcore.autosuspend=-1" bootcode fixes it. After that I will spend time in trying to catch the error with logging.
-
Hi Stefann
If you look at:
http://tinycorelinux.net/17.x/x86/tcz/usb-serial-6.18.2-tinycore.tcz.list
it provides many drivers. Paul_123 was asking which one(s) get loaded.
This should help in gathering more information:
tce-load -wi lsusb
lsusb > lsusb.txt
lsusb -t >> lsusb.txt
dmesg | grep -iE "usb|serial" > dmesgusb.txt
lsmod | head -n 1 > lsmod.txt
lsmod | tail -n +2 | sort >> lsmod.txt
Then post the contents of lsusb.txt, dmesgusb.txt, and lsmod.txt.
-
Update,
Sadly... this morning crashed again. After less than 12hr :-(
No logging, I was basically testing whether the "usbcore.autosuspend=-1" bootcode would fix the problem with system running in normal state. So... it did not.
Thanks @Rich for directions on lsusb.tcz. However... it's not in the distribution.
I checked http://repo.tinycorelinux.net/17.x/x86/tcz/ until http://repo.tinycorelinux.net/6.x/x86/tcz/ and could not find it (I skipped a few releases)
Anyway...
I think I will setup a syslog that writes to a network connected usb on my test system. By doing so I feel comfortable logging massive data and potentially frying the usb-stick.
I need to figure out how to get this all operational but i think I will manage.
From tomorrow or Monday onwards I will revert back to TC15 however. I will be travelling April 2-10 and over that time the system should be rock stable. So I want to check that few days before I leave. That will (hopefully) also give confirmation that it's really upgrade related and not "something else that by change happened during the upgrade"
-
lsusb in usbutils.tcz
-
Hi Stefann
Sorry, my mistake. Zhe is correct. I should have said:
tce-load -wi usbutils
lsusb > lsusb.txt
lsusb -t >> lsusb.txt
-
@Zhe, Thanks! found it.
And @rich: dont worry, I appreciate all help!
tc@huis:/$ lsusb -t
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M
|__ Port 1: Dev 2, If 0, Class=(Defined at Interface level), Driver=usbfs, 1.5M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M
|__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=usbfs, 12M
|__ Port 2: Dev 3, If 0, Class=Vendor Specific Class, Driver=ftdi_sio, 12M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/4p, 480M
tc@huis:/$ dmesg | grep -iE "usb|serial"
Kernel command line: BOOT_IMAGE=/tce/boot/vmlinuz17 quiet usbcore.autosuspend=-1 host=huis cron tz=CET-1CEST,M3.5.0,M10.5.0/3 waitusb=5:UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113" tce=UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113" initrd=/tce/boot/core17.gz
Unknown kernel command line parameters "cron host=huis tz=CET-1CEST,M3.5.0,M10.5.0/3 waitusb=5:UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113" tce=UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113"", will be passed to user space.
ACPI: bus type USB registered
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
ehci-pci 0000:00:10.4: new USB bus registered, assigned bus number 1
ehci-pci 0000:00:10.4: USB 2.0 started, EHCI 1.00
hub 1-0:1.0: USB hub found
uhci_hcd 0000:00:10.0: new USB bus registered, assigned bus number 2
hub 2-0:1.0: USB hub found
uhci_hcd 0000:00:10.1: new USB bus registered, assigned bus number 3
hub 3-0:1.0: USB hub found
usbcore: registered new interface driver uas
usbcore: registered new interface driver usb-storage
usbcore: registered new interface driver ums-alauda
usbcore: registered new interface driver ums-cypress
usbcore: registered new interface driver ums-datafab
usbcore: registered new interface driver ums_eneub6250
usbcore: registered new interface driver ums-freecom
usbcore: registered new interface driver ums-isd200
usbcore: registered new interface driver ums-jumpshot
usbcore: registered new interface driver ums-karma
usbcore: registered new interface driver ums-onetouch
usbcore: registered new interface driver ums-sddr09
usbcore: registered new interface driver ums-sddr55
usbcore: registered new interface driver ums-usbat
usbcore: registered new interface driver appletouch
usbcore: registered new interface driver bcm5974
usbcore: registered new interface driver synaptics_usb
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
usb 2-1: new full-speed USB device number 2 using uhci_hcd
waitusb=5:UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113"
usb 2-2: new full-speed USB device number 3 using uhci_hcd
usb 3-1: new low-speed USB device number 2 using uhci_hcd
usbcore: registered new interface driver usbserial_generic
usbserial: USB Serial support registered for generic
usbcore: registered new interface driver ftdi_sio
usbserial: USB Serial support registered for FTDI USB Serial Device
ftdi_sio 2-2:1.0: FTDI USB Serial Device converter detected
usb 2-2: Detected FT232R
usb 2-2: FTDI USB Serial Device converter now attached to ttyUSB0
tc@huis:/$ lsmod | head -n 1 && lsmod | tail -n +2 | sort
Module Size Used by Not tainted
8139cp 20480 0
8139too 20480 0
cifs 454656 2
cifs_md4 12288 1 cifs
cpufreq_conservative 12288 0
cpufreq_powersave 12288 0
cpufreq_userspace 12288 0
ftdi_sio 36864 1
loop 20480 314
mii 12288 2 8139too,8139cp
netfs 36864 1 cifs
nls_ucs2_utils 8192 1 cifs
pcspkr 12288 0
serio_raw 12288 0
squashfs 36864 157
usbserial 20480 3 ftdi_sio
Next to that....
I made some steps to set this up a bit more structural
- system is now running in fully normal state on TC17
- I disabled the syslog that @rich defined for me
- I re-enabled the rsyslog that I am usually using. Now I have my normal logging back
- I included a line for system logging. It's now logging on "notice" level:
# debug tc17 crash
*.notice :omfile:$tc17_log;MyFormat- I made this write to my network connected test-system where it writes to an old usb-stick I had lying around. it's a 200M stick. So I can fully overload and fry that with data. don't care if that breaks.
- proper rotation is in place.
Now... wait...
As said, by latest Monday I will revert to TC15 as I need to have a robust run between April 2-10 when I'm travelling
-
Hi Stefann
A couple of items:
... tce-load -wi usbutils
lsusb > lsusb.txt
lsusb -t >> lsusb.txt
You forgot to run the first version of the command.
The first version shows VID and PID numbers but not how devices are connected.
The second version shows how devices are connected but not VID and PID numbers.
... ----- Snip -----
lsmod | head -n 1 > lsmod.txt
lsmod | tail -n +2 | sort >> lsmod.txt ...
These two commands were not meant to be run in isolation. When run as
shown, the resulting file contains the header immediately followed by a
sorted list of loaded modules. I corrected the result in your post to show
how it should have looked.
This would have worked too:
lsmod | head -n 1 && lsmod | tail -n +2 | sort
-
You forgot to run the first version of the command.
oops. here you go:
tc@huis:~$ lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 002: ID 0bc7:0001 X10 Wireless Technology, Inc. ActiveHome (ACPI-compliant)
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 003: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC
Bus 002 Device 002: ID 04fa:2490 Dallas Semiconductor DS1490F 2-in-1 Fob, 1-Wire adapter
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 002: ID 0bc7:0001 X10 Wireless Technology, Inc. ActiveHome (ACPI-compliant)
This is a usb connected peripheral I control from my application with libusb
Bus 002 Device 002: ID 04fa:2490 Dallas Semiconductor DS1490F 2-in-1 Fob, 1-Wire adapter
Also a usb connected peripheral I control from my application with libusb
Bus 002 Device 003: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC
This seems to be the serial device. I don't have info. Its a black interface cable I bought 17years ago.
-
Well, there is alot there to unpack. But any of that could be your issue. But at least we have a clear understanding of drivers involved. Could be any of them
-
Hi Stefann
... tc@huis:/$ lsusb -t
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M <-------------------- Slow port
|__ Port 1: Dev 2, If 0, Class=(Defined at Interface level), Driver=usbfs, 1.5M <-- X10 Wireless Technology, Inc. ActiveHome (ACPI-compliant)
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M <-------------------- Slow port
|__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=usbfs, 12M <---------- Dallas Semiconductor DS1490F 2-in-1 Fob, 1-Wire adapter
|__ Port 2: Dev 3, If 0, Class=Vendor Specific Class, Driver=ftdi_sio, 12M <------- Future Technology Devices International, Ltd FT232 Serial (UART) IC
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/4p, 480M <------------------- Fast port ...
I don't know if it's an issue, but your peripherals appear to be plugged
into slow speed (12 Mbits/sec) ports.
If I'm reading it correctly:
The X10 Wireless device is connecting at 1.5 Mbits/sec, which sounds fine.
The other 2 devices are connecting at 12 Mbits/sec, which may be taxing that slow speed bus.
The faster (480 Mbits/sec) port might be a better choice if it
provides external USB ports.
-
Hi Stefann
... tc@huis:/$ lsusb -t
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M <-------------------- Slow port
|__ Port 1: Dev 2, If 0, Class=(Defined at Interface level), Driver=usbfs, 1.5M <-- X10 Wireless Technology, Inc. ActiveHome (ACPI-compliant)
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M <-------------------- Slow port
|__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=usbfs, 12M <---------- Dallas Semiconductor DS1490F 2-in-1 Fob, 1-Wire adapter
|__ Port 2: Dev 3, If 0, Class=Vendor Specific Class, Driver=ftdi_sio, 12M <------- Future Technology Devices International, Ltd FT232 Serial (UART) IC
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/4p, 480M <------------------- Fast port ...
I don't know if it's an issue, but your peripherals appear to be plugged
into slow speed (12 Mbits/sec) ports.
If I'm reading it correctly:
The X10 Wireless device is connecting at 1.5 Mbits/sec, which sounds fine.
The other 2 devices are connecting at 12 Mbits/sec, which may be taxing that slow speed bus.
The faster (480 Mbits/sec) port might be a better choice if it
provides external USB ports.
Well…it’s a low power low spec computer from 2008. 500MHz single core. So it’s definitely not fast.
There is not much choice on ports. There are only 3 on the outside and I’m using all 3.
The 480M port is NOT available on the outside. Could this be the compact flash drive?
The system does not have harddisk but a compact flash card as persistent at storage.
Speed however is not a big thing.
- the serial port reads my energy meter that spits out a data block of 500 bytes each second.
- the X10 interface controls the lights in the house. That’s blocks of about 20 bytes each time a light gets switched.
- the 1 wire is the only time critical thing. It is a constant stream of data. 1 wire protocol allows bit-length to be 75us so that is still only 13kHz. Keeping that thing at full speed is important. It currently takes 250ms to Do a “all sensor read” of the house. That should not get longer.
System still running since start yesterday afternoon.
-
For those interested,
manual for this 18yr old computer is still online (amazing!):
https://ftp.emacinc.com/LegacyProducts/EmbeddedServers/ebox-4300/Manual/ebox_4300_manual.pdf
I have version with 1G ram.
I have it currently with 8G compact flash as persistent storage.
It runs on a low power (1 watt) 500MHz 32bit single core Via Eden x86 based processor
I donot have the (optional) serial port
-
Hi Stefann
... Speed however is not a big thing. ...
Agreed. I does not appear like any of those devices would tax the USB ports.
... - the serial port reads my energy meter that spits out a data block of 500 bytes each second. ...
You haven't mention the baud rate you are using and whether your
USB to RS-232 cable contains a transmit FIFO, so I decided to do a
little math.
Assuming:
8 data bits, 1 parity bit, 1 start bit, 2 stop bits (12 bits).
Baud rate of 9600 (~104 uSecs/bit).
Data bursts of 500 bytes.
12 bits * 0.000104 secs = 0.001248 secs
500 bytes * 0.001248 secs = 0.624 secs
This assumes zero gap time between transmitted bytes.
The RS-232 spec does not specify min/max gap times.
-
Hi Stefann
Is the USB to RS-232 cable you are using listed here:
https://ftdichip.com/product-category/products/cables/usb-rs232-cable-series/
-
Hi,
its difficult to say as I bought the cable 17 years ago.
I just went through my notes.
I cannot find a literal order, but my notes indicate that I was intending to buy: "FTDI TTL-232R (5V)".
From the website you shared that is close to this one:
https://ftdichip.com/products/usb-rs232-we-5000-bt_5-0/
But my usb-connector is black, not transparant.
I can see I saved screenshots from "FTDI FT prog" that configures the cable. So it looks like I kind of have that one.
For the record: it's still running. About 24hrs now.
logging is coming in. But to be honest, not much interesting. Just crontabs getting fired.
It's nice that the logging is now on a separate network connected system. That also means that in case of a crash I can check the logging before starting the crashed system.
-
On configuration of the cable,
Probably easiest if I just share my "init code".
It's probably a bit dirty. I was learning the technology while doing the job.
It's a combination of code I grabbed from the internet with personal additions.
so.. baudrate is 11520
After this initialisation I just read the output.
The "temp" is a data structure I use for temporary results.
int start_meter(char * dev)
{ struct termios pts = { 0 }; /* termios settings on port */
int fd;
/* some things we want to set arbitrarily */
pts.c_lflag &= ~ICANON;
pts.c_lflag &= ~(ECHO | ECHOCTL | ECHONL);
pts.c_cflag |= HUPCL;
pts.c_cc[VMIN] = 1;
pts.c_cc[VTIME] = 0;
/* Standard CR/LF handling: this is a dumb terminal.
* Do no translation:
* no NL -> CR/NL mapping on output, and
* no CR -> NL mapping on input.
*/
pts.c_oflag |= ONLCR;
pts.c_iflag &= ~ICRNL;
/* set hardware flow control by default */
pts.c_cflag |= CRTSCTS;
pts.c_iflag &= ~(IXON | IXOFF | IXANY);
/* set 9600 bps speed by default */
// cfsetospeed(&pts, B9600);
// cfsetispeed(&pts, B9600);
cfsetospeed(&pts, B115200);
cfsetispeed(&pts, B115200);
temp.E_night = -1.0;
temp.E_day = -1.0;
temp.E_rnight = -1.0;
temp.E_rday = -1.0;
temp.E_power = -1.0;
temp.E_rpower = -1.0;
temp.E_id = -1.0;
temp.gaz_time_sec = -1.0;
temp.gaz_count = -1.0;
fd = open(dev, O_RDWR);
if (fd>=0)
tcsetattr(fd, TCSANOW, &pts);
return fd;
}
-
Update,
After 48hrs it was still running under TC17 with extra bootcode "usbcore.autosuspend=-1"
Logging at *.notice level but not much in logging. Only some application logging I made myself and some crontab-launches.
I stopped all manual.
I will be travelling April 2-10 during which period I cannot risk a crash.
So.. I reverted back to TC15 and restarted.
That gives me a few days of run hopefully without crash.
The thing did run for multipel month without crash before I started my upgrade work so in principle it should run without crash now.
However... there is always the possibility that during the upgrade I did something "TC17 unrelated" that is causing all this. So in itself this should also give confirmation that TC15 is stable.
Target to run until April 10...11.
Note...
On the positive side..
It's extremely convenient that going back to earlier release is as simple as:
- modify tce/boot/extlinux/extlinux.conf to pick the older core.gz and vmlinuz (I did not overwrite them but kept old version as core15.gz and vmlinuz15)
- rename tce/optional directory to get the previous one back (yes I made a backup :-)
- modify tce/onboot.lst to include correct kernel extension for usb-serial
try that with windows.......
unless I get a crash: dont expect to hear form me until April-11
in the mean time: any smart idea is welcome. I will not have access to my tinycore system but travel with laptop so will check this forum regularly.
-
Hickup when I restarted on TC15,
I thought it was running (all gets started automatically all seemed up) but I noticed that the state did not change.
Better checking revealed:
- yes TC15 was running
- but my application was not
Starting it from the command line gave me:
tc@huis:/krubo/work$ /krubo/work/krubo /krubo/work/1wire.def
/krubo/work/krubo: /lib/libc.so.6: version `GLIBC_ABI_GNU_TLS' not found (required by /krubo/work/krubo)
/krubo/work/krubo: /lib/libc.so.6: version `GLIBC_2.42' not found (required by /krubo/work/krubo)
What I did:
- completely recompiled my application on TC15
- restart
now all works.
Which makes me thinking:
- I probably need to do that on TC17: "completely recompile the application"
Anyway... I now want to run the thing "crash free" for upcoming days to allow me a headache-free travel April 2-10
-
We update Glibc every release, For the most part Glibc is backwards compatible. So there should be no reason to recompile when moving from 15->17.....but if you compiled under 17, then it will not work on 15.
Note that we keep referencing the Serial USB interface, but it could be any of them. Now that you have logging figured out, it might be useful to set it so that it does crash, to hopefully verify the reason for the crash.
-
Yes I will “set it up to crash”.
But first travel between April 2-10 running on TC15.
IF it crashes I will notice that as the system runs an Apache webserver that is available from outside.
My brother lives nearby and could than give it a manual reset.
The fricky thing is that “the full OS crashes”. No way to reset from outside.
My application has a watchdog around it. If that crashes it gets a restart after 15 minutes.
When I get back to “let it crash” I will also search my vga-monitor and keyboard that I have somewhere at the attic for exactly this reason.
-
Today 14:00 still running on TC15 after 3days.
I leave it running, go on 8day travel.
Note, I realised that instead of logging to usb-stick on 2nd system I can simply do that to the ram-disk of the 2nd system.
When logging on the live-system logging to ram-disk was not an option because it was not readable after a crash. Now I log via the network to the 2nd system there is no reason to avoid its ram-disk.
That means I can do logging at debug-level without causing excessive flashdisk writes when I re-setup the thing in "crash mode on TC17" when I'm back
For those interested, my full crashlog.
Initial log-entries are less precise maintained than later entries as at the beginning I did not know it would be this structural.
mon march 16: start around 12:00 //full TC17
Tue march 17: crashed in morning without yesterday data, so 24:00 latest
Tue march 17: start around 13:00 //full TC17
Wed march 18: crashed in morning without yesterday data, so 24:00 latest
Wed march 18: start around 23:00 //TC17 core & vmlinuz but no apps, add logging
Fri march 20: crashed in morning without yesterday data, so 24:00 latest // no logging due to mount mistake
Fri march 20: start around 14:00 //TC17 core & vmlinuz but no apps, no X10 interface on usb because usb port used for logging on usb-stick
mon march 23: manual stop around 9:00 // no crash after 4 days
mon march 23: start with full TC17 and P1 & X10 interface 8:30 // X10 interface installed again, check that it still crashes
mon march 23: crash, no logging 22:00 - 23:00 // no logging because usb-slot used by X10 interface, not available for logging usb stick
tue march 24: restart with usbcore.autosuspend=-1 bootcode around 9:00 // usbcore.autosuspend=-1 bootcode
tue march 24: manual stop around 12:00
multiple manual start/stop due to install of home battery
Wed march 25: restart with usbcore.autosuspend=-1 bootcode around 20:00 // usbcore.autosuspend=-1 bootcode
Fri march 27: manual stop for maintenance 10:00; 38hr crash-free run
Fri march 27: restart with usbcore.autosuspend=-1 bootcode around 14:20 // usbcore.autosuspend=-1 bootcode
Sat march 28: crash around 9:00 //after 19hrs
Sat march 28: restart with usbcore.autosuspend=-1 bootcode around 13:30 // introduced serious logging, *.notice
Mon march 30: manual stop around 14:00 // 48hr run without issues
Mon march 30: restart under TC15, no extra bootcode around 17:00
Thu april 02: still running on TC15. Leaving for a 8 day travel
-
OK... I'm back
System has been running on TC15 without any glitch for 12 days on TC15 without issue.
--> that proofs that it is TC related
Stats sofar:
TC15
- including usb-serial-6.6.8-tinycore.tcz
- "per second reading" of serial interface
- multiple month of running without issue
- rechecked last 2 weeks, 12days without issue ending with manual stop
TC17
- including usb-serial-6.18.2-tinycore.tcz
- "per second reading" of serial interface
3x crash after about 15hrs of runtime
TC17
- including usbcore.autosuspend=-1 bootcode
- including usb-serial-6.18.2-tinycore.tcz
- "per second reading" of serial interface
1x crash after about 19hrs of runtime
2x still running after >38hrs but manual stopped for reason of maintenance
I just started the thing:
TC17
- WITHOUT recompile or relink of application
- WITHOUT usbcore.autosuspend=-1 bootcode // try to get it to crash within a day
- including usb-serial-6.18.2-tinycore.tcz
- "per second reading" of serial interface
- rsyslog logging at debug level (*.*), writing over network to Ramdisk of 2nd computer (avoid flash-wear)
Note: on "logging" I'm a bit disappointed.
- I now see very detailed logging from my application
- but litterally zero kernel logging (maybe a bit at startup but that has been rotated away)
- I would have expected that "debug level" (which I think is included in *.*) would give me kernel things per second
So: any tips on getting more kernel logging are welcome
I'm writing to a Ramdisk so "overload of logging" is not a problem.
-
Wel...
after 3 days of running in "crash mode" it did not crash.
I stopped the thing manually.
I had a tiny "hunge"... I did a full recompile of my application when I went back to TC15.
So... I grabbed the executable from the backup I made before upgrading and started that.
We'll see how this goes.
For the record, I did NOT reboot TC17
so... (recent) history:
sat april 11 TC17, 15:30
- WITHOUT recompile or relink of application
- WITHOUT usbcore.autosuspend=-1 bootcode // try to get it to crash within a day
- including usb-serial-6.18.2-tinycore.tcz
- "per second reading" of serial interface
- rsyslog logging at debug level (*.*), writing over network to Ramdisk of 2nd computer (avoid flash-wear)
tue april 14 16:20, manual stop, no crash over 3 days
tue april 14 16:23, start again under TC17
- using executeble as compiled on TC 15 march15, same executable that crashed earlier
- no reboot of TC17
- WITHOUT usbcore.autosuspend=-1 bootcode // try to get it to crash within a day
- including usb-serial-6.18.2-tinycore.tcz
- "per second reading" of serial interface
- rsyslog logging at debug level (*.*), writing over network to Ramdisk of 2nd computer (avoid flash-wear)
proof I donot have usbcore.autosuspend=-1 bootcode
tc@huis:/krubo/work$ showbootcodes
BOOT_IMAGE=/tce/boot/vmlinuz17 quiet host=huis cron tz=CET-1CEST,M3.5.0,M10.5.0/3 waitusb=5:UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113" tce=UUID="a6f362cf-6c77-4fb7-bb05-76724d3b8113" initrd=/tce/boot/core17.gz
proof of release running:
tc@huis:/krubo/work$ uname -a
Linux huis 6.18.2-tinycore #1 SMP Sat Dec 20 21:06:22 UTC 2025 i686 GNU/Linux
proof of version of executable:
tc@huis:/krubo/work$ ls -l krubo
-rwxr-xr-x 1 tc staff 242508 Mar 14 16:33 krubo
-
Update.
Last Wednesday my MacBook broke down. Screen was flickering but still working on external monitor. So.. likely screen connection flex-cable.
I disassembled the MacBook, removed the screen, but failed to repair the flex cable. Got it reassembled but same problem.
I ended up buying a new MacBook. That was "not entirely bad", the macbook was 8 years old and I was already longing for an M-series computer so now I had a good excuse. Got the new model as released 1 month ago.
But... that's why I was a bit silent. "Not having the laptop" means "not logging in" on my tinycore system. I CAN login from my iPad but that's very inconvenient for doing serious work like this.
Having said that...
System did run without crash from April-14 16:23 until today, April 18 14:30, so 4 days.
On the one hand "great", on the other other hand "but why did it fail 3 time in a row after 15 hours when I upgrade tot TC17"?
So...
next try...
I recompiled my application under TC17.
It's a bit of a wild guess. But I know I recompiled parts of my application when the things were failing earlier.
When I originally upgraded from TC15 to TC17 I first got the thing running with TC15 compiled executable but "after I concluded that was running without problem" I made some modifications and recompiled.
So, I do not know whether the failing was from the TC15 compiled executable or from the TC17 compiled executable.
Note: I now did a full recompile which is different from my original sequence of events that had only a partial recompile. It becomes extremely difficult to exactly redo that
Any suggestions are welcome
Now this config is under test:
sat april 18 15:00, start again under TC17
- using full recompiled executable under TC17
- WITHOUT usbcore.autosuspend=-1 bootcode // try to get it to crash within a day
- including usb-serial-6.18.2-tinycore.tcz
- "per second reading" of serial interface
- rsyslog logging at debug level (kern.*), writing over network to Ramdisk of 2nd computer (avoid flash-wear)
note:
I changed the (*.*) Rsyslog to (kern.*).
By doing so I got rid of logging pollution from my own application. My application sends multiple logging messages per second so that was quite annoying.
I got lots of kernel messages on startup > so logging is working
But zero messages over past hour
-
Bang,
Crash after about 4 hours.
It’s late now, I will look further tomorrow.
Quick conclusion.
It’s not the Linux kernel.
It’s a weak spot in my application that gets exposed by the new gcc
Unfortunately zero logging from rsyslog. Logging was active but did not give any output.
What probably happened during the tc15 > tc17 upgrade:
- upgraded tc15 to tc17 > all ok, ssh, samba, vnc, Apache all functional.
- ran application > also ok
Than…
As I was working on the system and seemed fine..
Did some changes to the code and did a recompile.
>> software still running. No suspicion
17 hour later crash > forgot about the recompile, blamed tc17.
At this moment this is my theory. Tomorrow I will see how to proceed.
-
Ok..... new day...
I booted back to TC15
Did a complete recompile
than booted into TC17
Re-compiled only the top-level .c file
Restarted the application
I'm a bit under the assumption that the "issue" is in one of the driver or I/O related functions which I now did NOT compile on tc17.
If it does NOT crash after 24hr:
> I will recompile the .c file that drives the serial-over-usb port and try again
If it DOES crash after 24hr:
> I will revert to all compilation by tc15 and just do the linking in tc17 and try again
I also found my old vga monitor and keyboard at the attic and connected that. So I now have a direct terminal "not over ethernet"
We'll see
-
Update
I noticed that the energy meter was not updating.
And that is suspicious because the energy meter is connected to the "serial over usb" port that uses the kernel extension "usb-serial-6.18.2-tinycore.tcz"
So..
I rechecked again...
And I changed the makefile such that I can relink the application without doing any compilation
boot to TC15:
compile all
run > program runs, energy meter works
boot to TC17
run > program runs, energy meter works (this configuration did run for 4 days April-14 >18)
delete executable
relink executable without any compilation
run > program runs, everything seems to function but energy meter is not working
recompile serial over usb component
run > program runs, energy meter works
I leave this running for a while
Along the way:
- I invested in making "boot different versions" super easy selectable from extlinux.conf by using dedicated tce directories while using same restore location
- I now have my "main.c" also as a separate .o object file. I used to have all underlying components compiled to .o but main.c compiled and linked together.
- system now has physical hardwired monitor & keyboard for testing (it's taking annoying amount of space)
-
update,
definitely "something" is not right
19 april
15:25 Mem: 468928K used, 492592K free, 35852K shrd, 3844K buff, 283808K cached
15:38 Mem: 489316K used, 472204K free, 36036K shrd, 3852K buff, 283992K cached
15:41 Mem: 497508K used, 464012K free, 35900K shrd, 3852K buff, 283856K cached
16:00 Mem: 497464K used, 464056K free, 35832K shrd, 3868K buff, 283852K cached
16:16 Mem: 497672K used, 463848K free, 35880K shrd, 3868K buff, 283900K cached
16:53 Mem: 506816K used, 454704K free, 35876K shrd, 3868K buff, 283896K cached
17:02 Mem: 507040K used, 454480K free, 35956K shrd, 3868K buff, 283976K cached
17:43 Mem: 515232K used, 446288K free, 35912K shrd, 3884K buff, 283932K cached
18:02 Mem: 523640K used, 437880K free, 36028K shrd, 3884K buff, 284048K cached
looks like I'm loosing memory at a speed : 54712k over 2.7hr = 20MB/hr
My "serial over usb" port is reading blocks of 812byte every second = 3MB/hr (and I'm sure overhead will multiply this)
So... its entirely possible that the "serial over usb" kext is completely "not" releasing memory for any data it is receiving (until connection gets closed).
My application opens a serial file descriptor and keeps reading it for ever.
Note: using this connector is a configuration item.
I keep it running until it (hopefully) crashes, but after that I can rerun it without reading the serial port without recompilation.
-
Hi Stefann
Each time before checking for free memory, I suggest running this:
sync
sudo cache-clear
syncThis helps to insure other processes are not skewing the results.
-
@rich...
good suggestion...
now I got a lot back:
NO CACHE CLEAR
Sun Apr 19 19:57:41 CEST 2026
m total used free shared buff/cache available
Mem: 961520 228600 424736 35888 308184 559560
Swap: 226116 0 226116
WITH CASH CLEAR
Sun Apr 19 19:59:40 CEST 2026
m total used free shared buff/cache available
Mem: 961520 258156 635764 35948 67600 578960
Swap: 226116 0 226116
But...
- did I now cleanup the analysis?
- or did I inject a solution? (doing a cleanup that the application or kext should have done)?
I think I better NOT do this at this point. Just observe whether it runs towards a crash if I do not interfere
-
Hi Stefann
... - did I now cleanup the analysis? ...
Yes you did.
... - or did I inject a solution? (doing a cleanup that the application or kext should have done)? ...
No, you did not. If you really have a memory leak, free
memory will continue to drop.
-
1 day later, no crash.
memory is not going down the drain.
pfew.. its not easy.
I stopped the application & recompiled all under TC17. This was the "fastest crash situation" so far.
Last stretch it did run without gui and without vnc server for reason I have a text-terminal connected to it without mouse. That however also reduces the system load. So.. gui & vnc are active again.
I DO have the monitor attached. Not sure weather I will be able to type something but I will have at least "some" screen vision when it crashes.
Also monitoring memory use.
-
Do you leak on your own app ?
Maybe valgrind time then. :)
-
Do you leak on your own app ?
Maybe valgrind time then. :)
On the one hand: "everything is possible"
on the other hand: my app used to run uninterrupted for multiple month without issue on TC15
never used valgrind but my brother also mentioned it.
At this moment I try to have it crash in a configuration that did crash earlier.
I noticed that "any modification" changes the behavior which made me conclude I best "make the absolute minimum of modifications" per analysis.
At this moment I have a monitor connected to the system that normally runs headless. So I hope for a crash "with info"
-
and an other crash...
after about 16hr of runtime with full application recompiled under tc17
mon Apr 20 16:15:18 // restart with full recompile under TC17
tue apr 21 8:50 //crashed
- hardwired connected monitor does not show anything, "no signal"
- kernel logging not present (Rsyslog logging kern.* messages)
- memory logging shows no issue:
Tue Apr 21 07:57:25 CEST 2026
m total used free shared buff/cache available
Mem: 961520 278448 445316 42608 237756 503028
Tue Apr 21 08:12:25 CEST 2026
m total used free shared buff/cache available
Mem: 961520 279016 444812 42544 237692 502524
Tue Apr 21 08:27:25 CEST 2026
m total used free shared buff/cache available
Mem: 961520 279084 444812 42476 237624 502524
Tue Apr 21 08:42:25 CEST 2026
m total used free shared buff/cache available
Mem: 961520 279028 444812 42532 237680 502524
>> 8 minutes later it had crashed
I now restarted:
- exactly same executable
- but changed setting: it now no longer reads /dev/ttyUSB0
any tips on getting more logging?
-
So.. still running at 15:00; that means 30hrs runtime without crash
So, to summarize....
TC15:
- including kext: usb-serial-6.6.8-tinycore.tcz
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- no crash over multiple month
TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- 4..6x tested, always crashes, mostly after about 15hrs; always within 24hrs
TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- SAME application, NO recompile
- run application with disabled read from /dev/ttyUSB0 by configuration constant
- 1x tested, no crash after 30hrs
Now...
This does NOT proof that the read-action is the rootcasue because earlier experiments did show that "more general load" makes the crashing earlier.
What I now have done:
- modify application code
- same "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- but close device and re-open after every 15minutes
- started at 16:00 European time
If this keeps running than I think its demonstrated that usb-serial-6.18.2-tinycore.tcz has an issue with "reading long time without intermediate closing"
fingers crossed
-
Have you consider the BIOS with USB energy saving.
-
I doubt my 2008 board has that.
But..... isn't that only applicable while idle? the computer is never idle but in 24/7 use.
-
Sad.... an other crash this night at 1:30 am; after 33.5 hours
So.....
On TC17 crash after 33.5 hours if constantly reading reading from serial-over-usb port AND close/open filedescriptor every 15mins.
But....
2 days ago I concluded that running without reading serial-over-usb was crash-free as I manually stopped after 30hr of crash-free run.
>> that conclusion may be premature
Still..
WITH constant reading of serial-over-usb crash is normally in about 15hrs, and always within 24 hours.
So... although there is variation... it still looks like "altering settings of serial-over-usb" influences the behavior.
I got a tip from my brother that "SELECT" is a quite complicated kernel call.
I'm using SELECT to check that the pipe is ready for read but I can just as well call read from it in nonblocking mode without checking.
So... I now changed that...
- removed the SELECT
- read in non-blocking mode
I also made using read, select, 15min_close/open configurable by a runtime settable variable such that I can test different configurations without recompile (to exclude impact of compilation on results)
Started at 8:22 am:
TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- new compiled executable with runtime configurable serial read modes
- run application with non-blocking read without SELECT
- "every second 812byte read from /dev/ttyUSB0; baudrate 11520" from /dev/ttyUSB0
Recap of results so far:
TC15:
- including kext: usb-serial-6.6.8-tinycore.tcz
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- no crash over multiple month
TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- 4..6x tested, always crashes, mostly after about 15hrs; always within 24hrs
TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- SAME application, NO recompile
- run application with disabled read from /dev/ttyUSB0 by configuration constant
- 1x tested, NO CRASH after 30hrs >> I thought this was good but as next config crashed after 33.5hrs I may not have been running long enough
TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- include every 15 minute close/open file descriptor for /dev/ttyUSB0
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- 1x tested, crash after 33.5hrs
-
try upgrade tc 16 ,maybe can shrink this problem
-
Hi Stefann
... I got a tip from my brother that "SELECT" is a quite complicated kernel call.
I'm using SELECT to check that the pipe is ready for read but I can just as well call read from it in nonblocking mode without checking.
So... I now changed that...
- removed the SELECT
- read in non-blocking mode ...
You mentioned in a previous post you were opening files without
closing them. If you were using those file descriptors with select(),
that could be a problem:
C_Programs/AutoCursor/AutoCursor.c: int fdmax=0; // Highest file descriptor select() should monitor.
C_Programs/AutoCursor/AutoCursor.c: select((fdmax + 1), &Dcursor, NULL, NULL, NULL);Select is limited to somewhere around 1024 file descriptors. If you keep
opening the same file over and over again, you will eventually reach
that limit and bad things things will happen. You can test for this fairly
easily. When you open a file for select() to monitor, compare the file
descriptor to FD_SETSIZE:
if(file_descriptor >= FD_SETSIZE)
{
print error message
exit program
}
-
Hi,
I appreciate a lot that you are reading my reports, I can sure use any help I'm getting.
Yes,...
I'm opening the file descriptor at the start of my program and than never close it:
fd = open("/dev/ttyUSB0", O_RDWR);
My home automation program (originally) simply keeps reading from it.
Before each read it did checks whether there was data with a select command.
The fd remains the same, that never changes, so as far as I know nothing increments and reaches a limit
call this as part of forever loop:
{ static char buf[BUFSIZE+1];
fd_set fd_check;
struct timeval wait = {0};
FD_ZERO(&fd_check);
FD_SET(fd, &fd_check);
select(fd+1, &fd_check, NULL, NULL, &wait);
if (FD_ISSET(fd, &fd_check) )
{ j = read(fd, &buf[i], BUFSIZE-i);
... process results...
}
other commands...
}
So.. I think in principle THIS should work, and actually it does on TC15.
However...
You are very right that "if I would open other pipes without closing" I could get in trouble.
Yes.. it's not very safely programmed, it does not include the range check.
I never intentionally keep opening files without closing when I no longer need them.
What I meant is that I open "this specific file" and keep reading without ever closing.
In itself "that is not strange", I'm actively using it.
But if "it collects some garbage over time", it never gets a fresh start.
Anyway, last experiment was to "close & reopen" every 15mins. Its seems that that about doubled the time to crash but it still crashes.
Thanks for the input, I can include a logging on number of fd's when I do a next compilation.
"never intentionally" is different from "absolutely sure I did not"
For now I keep it running. Its 13 hrs down the road now... And it's almost bedtime here. Tomorrow morning I will know whether it at least passed the "15hr average crash milestone"
-
try upgrade tc 16 ,maybe can shrink this problem
I thought about that but that would not help me.
Just to "not have crashes" I could simply stay on TC15. But.... Than I would miss out on upgrades.
Getting upgrades is the whole point of getting to TC to begin with. I started on DSL in 2008 but DSL quickly stopped doing upgrades.
2 years ago I really had to move to a newer OS. Without upgrades I had no https and missed lots of libraries on gcc, php, python.
So... getting OS updates is important.
A stepping stone on TC16 would not give other insights.
Only when its clear which component has the error it makes sense to figure out whether that component broke from 15 to 16 or from 16 to 17.
-
Through all this, you still have not managed to capture any sort of log or crash dump. You likely need a direct console connection on a monitor and local keyboard.
-
Through all this, you still have not managed to capture any sort of log or crash dump. You likely need a direct console connection on a monitor and local keyboard.
Before answering,..
System is still running with excluded select() call. It passed the “15hr mark”, running 22 hours now. I will try to get to 48.
Not managed to get crash logging…. Very true.
Is still not managed to get any logging.
And I sure tried.
At this moment:
- I have a monitor hardwired connected by vga (indeed a local console)
- a keyboard hardwired connected by ps2
- I write rsyslog logging at lowest level (kern.*) to a network connected 2nd system
But after a crash:
- monitor is still on “screensaver”, not able to wake up. If I press a button on the monitor it says “no signal”
- the logging on the network connected systems shows zero logging from the crash.
>> in fact, I donot get a lot of logging anyway. I see a big chunk of logging at booting of tinycore but after booting no kern.* logging is created.
Any suggestion to get logging is welcome.
In fact I should probably find a way to keep the signal on the monitor. I have no idea how.
It’s running fvwm.
When I boot in text mode (which would be more straightforward) the “time to crash” is also longer.
>> I’ll see whether I can find a setting on the monitor.
Any advice is welcome. I indeed had the objective to get some logging but “it’s not there”. It looks like the system “simply drops”.
-
Still running, 25hrs and counting
@Paul_123, I checked my monitor but did not find any timeout things.
screensave is probably a fvwm setting? or other setting?
It may be a bios setting, I can check that after this run.
@rich,
your comment on "not letting amount of open file descriptors grow" makes perfect sense.
I'm not using any malloc in my application so I thought I was relatively "save" on risk for memory leak.
However... repeatedly opening filedescriptors without closing COULD have been a leak.
I checked all source files, conclusion:
- No, no unlimited growing, I'm very clean in closing file descriptors after I opened them. As said, the usb-serial fd is 1x opened and never closed. But NOT incrementally opened without closing
- I DID see error in checking valid opening. I generally consider it a "open failed" by comparing to null. That should have been by checking on negative.
This can lead to:
- writing or reading from negative fd's > I guess that could potentially crash the application but linux should not really crash without any logging.
- 2nd open of the same device if it had opened with fd=0, but only 1x, only in the unlikely event that stdi is not connected to 0.
Not nice,
I will definitely fix
But I do not think its a reason for these crashes
-
Hi Stefann
... But after a crash:
- monitor is still on “screensaver”, not able to wake up. If I press a button on the monitor it says “no signal” ...
It's not the monitor doing that. It's the operating system disabling the
video because of no keyboard or mouse activity.
Add this boot code:
consoleblank=0
I don't remember, is there a GUI running?
If there is, run these commands just before starting your program:
xset dpms 0 0 0
xset s noblank
xset s off
xset -dpms
That should keep your monitors screen from blanking.
You'll know after about 10 minutes if that worked.
-
Thanks a lot!
I will do that.
Yes there is a gui and vnc server running.
I had switched to text mode during some earlier experiments but that made the "time to crash" much longer.
> the whole crash thing seems to have a "rootcause that triggers" and "load situation that depicts duration".
System is now running 32 hours.
> it passed the 15hr milestone: average crash duration in original state
> it is very close tot the 33.5hr milestone: crash with "every 15minute fd-close/reopen"
So.. fingers crossed.
If it does not crash I think I will keep running it for something like a week.
Because current configuration not only serves as an analysis but also can serve as a fix.
Reading without select by means of nonblocking read() statement can be a perfectly acceptable solution for my application
So... it is quite valuable to test whether this has the potential to be "crash free for multiple month". If it survives a week I get good feelings about that.
So...
- I will likely keep it running for a week if it does not crash
- if it DOES crash I will modify the monitor settings as you indicated. I will than run the original "likely crash in 15hr" configuration.
If it survives a week I will make a dedicated separate testprogram that calls select() in an endless loop with about 100x more calls/minute than my application. That would probably be able to get the system to crash in about an hour or so.
It still would make sense to find the rootcasue of select() shutting down the whole system. But that would likely take quite some time. I will definitely try to find or at least help to find the cause. But until than.... not using it can be a perfect solution for my application.
-
Hi Stefann
Thanks a lot!
I will do that.
Yes there is a gui and vnc server running. ...
After you do that, unplug the keyboard and mouse. Then check
the monitor after 15 minutes. If the screen is still visible, the
blanking has been disabled and you can reconnect the keyboard
and mouse. If the screen is blanked, check if the vnc server you
are using has any options to disable blanking the display.
-
Thanks,
I will do that.
Settings for the vnc server are not so relevant.
I now have a terminal hard wired connected.
Only reason to run it is to “keep the test setup consistent with load”
I donot have a mouse. There is no ps2 mouse connection and all usb ports are occupied.
Note..
Still running.
34 hr
A new record with serial over usb being full functional
-
small update:
- still running after 48 hour.
This is the first time I have more than 34 hours "full functional on TC17".
- on the one hand this is a stromg indication that select() is the root cause
- on the other hand "not using select() and doing nonblocking read" is a perfect fix for my application
I keep it running for 7 days, until next Friday. If it did not crash by than I call it "demonstrated OS/kext rootcause" and "demonstrated application fix/workaround".
So: No posts on this thread for coming 5 days unless there is a crash.
-
3x 24hr = 72hr crashfree continuous run and counting..
-
4x 24hr = 96hr crashfree continuous run and counting..
-
5x 24hr = 120hr crashfree continuous run and counting..
-
6x 24hr = 144hr crashfree continuous run and counting..
Tomorrow around this time I will stop it manual if it still did not crash.
A full week of crash free run I would consider "demonstrated".
That will give me at least a "fix/workaround to run my application under TC17".
I will however do a bit more work to zoom-in on rootcause...
I will get the screensaver deactivated by the instructions from @Rich
Than I will restart with select() back in but read() still on non-blocking (the select() will than not have any function but just be there to test whether it hurts).
The difference between original program and currently running version is not only that select() is no longer called, the read() is now called in non-blocking mode in stead of "select() gated blocking mode". So.... rootcause could be select() but it could also be that "blocking read() call" blocks forever even if there is no reason to do so. This would actually connect to the fact that zero logging gets produced at a crash. "The system just locks".
I'm running on a single core cpu. So if this single core gets stuck there is nothing else to take over (not sure it works like that, but it is at least a fact).
Current version of the application is made configurable on this by a user setting so I can run this configuration change without need for recompile. That is a benefit. This brings the amount of changes to an absolute minimum.
So....
- Any advice from a different timezone on "what to do with next run" is welcome.
- Again... although I'm logging rsyslog towards a network connected second computer, I get zero logging from rsyslog with a crash, even though I'm logging at kern.* level. Any advice to get more logging would also be welcome.
-
7x 24hr (-1hr, started early today) = 167hr crashfree continuous run
Good!
I just stopped the application manually.
I restarted the application with select(). The read() is still in nonblocking mode.
That means that the select() does not have a function. Its only there to see whether it crashes.
As said yesterday I originally planned to disable the screensaver to allow my monitor to show logging. But... I decide not to do that. I have earlier seen that addition changes bring different behavior.
I now did zero changes. I only started the application with a different config-file.
So... if the select() is the crash-cause it should crash within 24hr, likely around 15hr
If it does not crash the "non blocking nature" of the read() is probably the crash-cause.
fingers crossed again...
Recap of results so far:
TC15:
- including kext: usb-serial-6.6.8-tinycore.tcz
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- no crash over multiple month
TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- 4..6x tested, always crashes, mostly after about 15hrs; always within 24hrs
TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- SAME application, NO recompile
- run application with disabled read from /dev/ttyUSB0 by configuration constant
- 1x tested, NO CRASH after 30hrs >> I thought this was good but as next config crashed after 33.5hrs I may not have been running long enough
TC17:
- including kext: usb-serial-6.18.2-tinycore.tcz
- not use select()
- use read() in non-blocking mode
- recompile full application
- run application with "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- 1x tested, no crash, manually stopped after 7 days, 167hr.
SO:
- using the read() in non-blocking mode has all functionality I need, so to my application this is a very acceptable fix.
- Rootcasue for crash is likely either the select() OR the blocking nature of the read(), I keep zooming in to find out.
-
thanks for taking the time to relate your testing and subsequent results!
-
@gadget42, thanks for the warm comment.
It feels good. Assuming this ends in finding an OS-bug and not a “stupid stupid application mistake after all”, it’s my contribution to the community which is more than fair given all that I’m taken from it.
This morning, after 23 hours, still running.
So I’m happy that I did not change the screensaver setup as than I would have been doubting that to be the reason.
So far “faulty configurations” crash on average after 15 hours, max I have seen after 35 hours. So I keep it running until tomorrow morning to see whether it’s still alive after 48 hours. I have obligations today so no opportunity to do anything earlier anyways.
If this keeps running it would indicate that select() is NOT the cause.
It would than strongly point tot the blocking setting of read() to be the cause.
If it still runs tomorrow morning I will restart without select() and with read() in blocking mode.
I get data bursts every second so the blocking read() will pass regularly. This can NOT be a solution for my application because it basically freezes with 1 second gaps. But that’s ok. It’s testing.
-
Still running after 47hrs.
So far the longest it has been running in a faulty mode was 35hr so it looks like "non faulty" but I may need 7days to be fully sure.
- that means that the select() does not look like to cause trouble
- the only other change compared to crashing April 20 and april21 run was the use of non-blocking read()
So... I now started a run that uses blocking read() but NO select.
The serial data comes in as a continuous per second stream of 0.5sec datablocks. As the read() is no longer protected by select() this means the application is pausing for 0.5sec/second so it basically runs at half speed.
I had to adapt and recompile the program to make "blocking" a configuration item.
To summarize:
Just halted:
TC17: 47 hrs crash free run; start may-1 8:00 > may-3 7:00; MANUALLY STOPPED
- including kext: usb-serial-6.18.2-tinycore.tcz
- same executable as before
- configuration: non-blocking read() AND select()
- "every second 812byte read from /dev/ttyUSB0; baudrate 11520" from /dev/ttyUSB0
- 1x tested, 47hrs
Just started:
TC17: xxxx; start may-3 7:20 > xxxx
- including kext: usb-serial-6.18.2-tinycore.tcz
- recompiled executable with configuration option to call read() in blocking mode
- configuration: blocking read(), NO select()
- "every second 812byte read from /dev/ttyUSB0; baudrate 11520" from /dev/ttyUSB0
- 1x tested, xxxx
Note:
Few weeks ago I transferred the network connected pheripherals to my 2nd computer. It's not optimal but my home automation is "with less nice gui" operational being split over 2 computers during this test phase.
-
Oops, still running after 24hrs but I miscounted a day and manually stopped while I could better have let it run for an other day.
So:
- running with blocking read() without select() for 24hrs without crash
- accidentally manually stopped it
- restarted. I will let it run for 48hrs unless it crashes before that
For those interested, this is how the code looks likes
I use enable_meter as a global variable that is read from a configuration file to control what functions are active.
Selection bit masks:
#define meterREAD 1
#define meterFD 2
#define meterSELECT 4
#define meterREFRESH 8
#define meterBLOCKING 16
Initialize (call once):
int start_meter(tmeter *X)
{ int flags;
meterMODE = enable_meter;
/* some things we want to set arbitrarily */
(X->pts).c_lflag &= ~ICANON;
(X->pts).c_lflag &= ~(ECHO | ECHOCTL | ECHONL);
(X->pts).c_cflag |= HUPCL;
(X->pts).c_cc[VMIN] = 1;
(X->pts).c_cc[VTIME] = 0;
/* Standard CR/LF handling: this is a dumb terminal.
* Do no translation:
* no NL -> CR/NL mapping on output, and
* no CR -> NL mapping on input.
*/
(X->pts).c_oflag |= ONLCR;
(X->pts).c_iflag &= ~ICRNL;
/* set hardware flow control by default */
(X->pts).c_cflag |= CRTSCTS;
(X->pts).c_iflag &= ~(IXON | IXOFF | IXANY);
/* set 115200 bps speed by default */
cfsetospeed(&(X->pts), B115200);
cfsetispeed(&(X->pts), B115200);
X->fd = open(X->dev, O_RDWR);
if (X->fd>=0)
{ flags = fcntl(X->fd, F_GETFL);
if ( !(meterMODE & meterBLOCKING) )
fcntl(X->fd, F_SETFL, flags | O_NONBLOCK);
tcsetattr(X->fd, TCSANOW, &(X->pts));
X->SecLastRefresh = SECNOW();
}
return 1;
}
service (call every 250ms):
int do_readmeter(tmeter *X)
{ static int i=0;
static int j=0;
static int n=0;
static int k=0;
static char buf[BUFSIZE+1];
fd_set fd_check;
struct timeval wait = {0};
meterMODE = enable_meter;
if (meterMODE & meterFD)
{ FD_ZERO(&fd_check);
FD_SET(X->fd, &fd_check);
}
if (meterMODE & meterSELECT)
select(X->fd +1, &fd_check, NULL, NULL, &wait);
if (meterMODE & meterFD)
FD_ISSET(X->fd, &fd_check);
//if (FD_ISSET(X->fd, &fd_check) ) //this was originally there but removed for this investigation
if (meterMODE & meterREAD)
{ j = read(X->fd, &buf[i], BUFSIZE-i);
while (j>0)
{ buf[i] &=0x007F;
if ( buf[i]=='\n' )
{ do_parsemeter(buf, X);
i++; j--;
n=i;
i=0;
for (k=0; k<=j; k++)
buf[i+k]=buf[i+n+k];
}
else
{ i++; j--;
}
if (i >= BUFSIZE)
{ i=0; j=0;
}
}
}
return 1;
}
-
And... BINGO.... crash somewhere between 00:00 6:30 this morning.
I don't have time today so I just restarted. I will setup to rerun with disabled screensaver later.
So... what we have...
TC17
- including kext: usb-serial-6.18.2-tinycore.tcz
- continuously reading data from /dev/ttyUSB0 at baudrate 11520 getting "every second 812byte blocks"
- Without using select or any other port-related function:
---> manually stopped after 7 days and running without issue while using read() in nonblocking mode
---> crash after 40..46hr while using read() in blocking mode
zero other changes between these 2 runs, running fine on TC15
As said, no time to explore today, I will definitely followup later
=====================================
Summarizing:
TC17: 7day, 167hr NO crash; start apr-24 8:22 > manually stop may-1 07:30
- including kext: usb-serial-6.18.2-tinycore.tcz
- new compiled executable with runtime configurable serial read modes
- run application with non-blocking read without SELECT
- "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- 1x tested, no crash, manually stopped after 7 days, 167hr.
SO:
- using the read() in non-blocking mode has all functionality I need, so to my application this is a very acceptable fix.
- Rootcasue for crash is likely either the select() OR the blocking nature of the read(), I keep zooming in to find out.
TC17: 47 hrs crash free run; start may-1 8:00 > may-3 7:00; MANUALLY STOPPED
- including kext: usb-serial-6.18.2-tinycore.tcz
- same executable as before
- configuration: non-blocking read() AND select()
- "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- 1x tested, 47hrs
TC17: 25hr crahsfree run; start may-3 7:20 > may-4 8:30 MANUALLY STOPPED
TC17: crash after 40...46hr; start may-4 8:30 > may-6 00:00...6:30
- including kext: usb-serial-6.18.2-tinycore.tcz
- recompiled executable with configuration option to call read() in blocking mode
- configuration: blocking read(), NO select()
- "every second 812byte read from /dev/ttyUSB0; baudrate 11520"
- 1x tested, 25hr crashfree
- 1x tested, crash after 40..46hr
-
Short status update.
I did not have much time last 2 days so progress is limited.
- I succeeded in disabling the screensaver, so console will now be functional at a next crash . THANKS RICH I used the commands you gave me some days ago!
- I disabled (by configuration, no recompile needed) the read() in my application (it's a bit inconvenient but I found an alternative way to measure energy via my solar system)
- I started making a dedicated "crash program" that calls the read() from an endless loop with 2ms looptime. That is about 80x faster than the read() in my application which has a looptime of 250ms. The signal has data bursts/gaps of about 600ms/400ms. No acceleration will be present during the gaps.
- I was not able to give that a thorough test yet. I want to do that "attended" so I need to have few hours of time for that. With a little luck I find such testtime tomorrow.
to be continued....
-
OK.. update...
I made a small "crash program" > see code below.
It basically reads from the serial port over usb every 100us.
It reads in "blocking mode".
The serial port is getting "every second 812byte datablock on /dev/ttyUSB0; baudrate 11520"
My original program runs a continuous loop that calls several service functions. The read function is called every 250ms. However... when in blocking mode at least half of them gets blocked so "average looptime" would have been around 500ms.
This "crash" program has a looptime of 100us BUT is of cause blocked during the gaps in data availability.
logging shows I'm getting 151loops per minute. That's 397ms on average. Oops... that is not much faster.
I guess the 812 datablock gets read in about 3 calls after which the "end of the second" needs to be awaited until the next block.
In addition to the read() calls I have set an interval timer for 1ms (thank you brother for handing me this code). This timer interrupts the read(). Running on the assumption that the blocking call has difficulty handling interrupts this is intended to give that an extra stress.
Logging is written to a network connected 2nd computer so even if the computer crashes I will have the logging.
My main application is running in parallel. Without reading the serial port.
Console is working. screensaver no longer kicking in.
So...
It's running.
Now wait....
My crash test program:
#include <sys/time.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <time.h>
#include <termios.h>
#include <stddef.h>
#include <signal.h>
#include <fcntl.h>
#include <sys/times.h>
#include <sys/time.h>
#include <sys/stat.h>
#include <sys/select.h>
//#include <syslog.h>
#define usDELAYTIME 100
#define minLOGINTERVAL 10
#define meterREAD 1
#define meterFD 2
#define meterSELECT 4
#define meterREFRESH 8
#define meterBLOCKING 16
#define BUFSIZE 8191
int meterMODE = meterREAD | meterBLOCKING;
char logfile[128] = "/remote/home/tc/crashlog.txt";
struct termios pts; // interface settings
int fd;
void usDelay(int len)
{ struct timespec delay;
delay.tv_sec = 0;
delay.tv_nsec = len;
delay.tv_nsec *= 1000; //1000=us, set in us * 10e3
nanosleep(&delay, NULL);
}
static void dummy_func(int sig)
{
}
int start_meter()
{ int flags;
/* some things we want to set arbitrarily */
pts.c_lflag &= ~ICANON;
pts.c_lflag &= ~(ECHO | ECHOCTL | ECHONL);
pts.c_cflag |= HUPCL;
pts.c_cc[VMIN] = 1;
pts.c_cc[VTIME] = 0;
/* Standard CR/LF handling: this is a dumb terminal.
* Do no translation:
* no NL -> CR/NL mapping on output, and
* no CR -> NL mapping on input.
*/
pts.c_oflag |= ONLCR;
pts.c_iflag &= ~ICRNL;
/* set hardware flow control by default */
pts.c_cflag |= CRTSCTS;
pts.c_iflag &= ~(IXON | IXOFF | IXANY);
/* set 115200 bps speed by default */
cfsetospeed(&pts, B115200);
cfsetispeed(&pts, B115200);
fd = open("/dev/ttyUSB0", O_RDWR);
if (fd>=0)
{ flags = fcntl(fd, F_GETFL);
if ( !(meterMODE & meterBLOCKING) )
fcntl(fd, F_SETFL, flags | O_NONBLOCK);
tcsetattr(fd, TCSANOW, &pts);
}
return 1;
}
int main()
{ char buf[BUFSIZE+1];
fd_set fd_check;
struct tm *tm_now;
time_t now, tstamp;
struct timeval wait = {0};
FILE *f_log;
int j =0;
unsigned long long count =0;
unsigned long long kcount =0;
struct itimerval interval = { 0 };
interval.it_interval.tv_sec = 0;
interval.it_interval.tv_usec = 1000; // repeat interval after 1st trigger; 1000 = 1ms
interval.it_value.tv_sec = 0;
interval.it_value.tv_usec = 1000; // time until first trigger
start_meter( );
if (fd < 0)
{ printf("cannot open device");
exit(1);
}
setitimer(ITIMER_REAL, &interval, NULL); //set interval timer
signal(SIGALRM, dummy_func); // dummy interrupt call
while (1)
{ if (meterMODE & meterFD)
{ FD_ZERO(&fd_check);
FD_SET(fd, &fd_check);
}
if (meterMODE & meterSELECT)
select(fd +1, &fd_check, NULL, NULL, &wait);
if (meterMODE & meterFD)
FD_ISSET(fd, &fd_check);
if (meterMODE & meterREAD)
{ j = read(fd, buf, BUFSIZE); // read
}
usDelay(usDELAYTIME);
count++;
time(&now);
if (now > tstamp+60*minLOGINTERVAL )
{ tm_now = localtime(&now);
tstamp = now;
kcount+= count/1000;
count %= 1000;
f_log = fopen(logfile, "a");
if (f_log!=NULL)
fprintf(f_log, "mon day: %2d %2d | hh:mm:ss: %02d:%02d:%02d | Still alive after %lldk + %lld loops\n",
tm_now->tm_mon+1, tm_now->tm_mday, tm_now->tm_hour, tm_now->tm_min, tm_now->tm_sec, kcount, count);
fclose(f_log);
}
}
exit(0);
}
-
Hi Stefann
... The serial port is getting "every second 812byte datablock on /dev/ttyUSB0; baudrate 11520" ...
I'm guessing you meant 115200 there.
Assuming 8 data bits + 1 start bit + 1 stop bit + 2 bits of gap time
between each byte transmitted:
Bits per datablock = 12 * 812 = 9744 bit times.
Transmit time per datablock = 9744 / 115200 = 0.0846 seconds, or
just under 85ms.
-
Hi Stefann
... The serial port is getting "every second 812byte datablock on /dev/ttyUSB0; baudrate 11520" ...
I'm guessing you meant 115200 there.
Assuming 8 data bits + 1 start bit + 1 stop bit + 2 bits of gap time
between each byte transmitted:
Bits per datablock = 12 * 812 = 9744 bit times.
Transmit time per datablock = 9744 / 115200 = 0.0846 seconds, or
just under 85ms.
Ah!
Yes!
good catch.
I did set it correctly in the code but my calculation of busy/pause time was indeed a factor of 10 off.
That explains a lot.
Unless it crashes I leave it running for 48hrs now.
Its all "a bit vague" so instead of further speculation "lets just see what comes out now"
Thanks for the insight. That helps with defining a followup configuration.
With that said.....
I think im getting close to exploring the source code.
My brother will likely help me. He is a bit of a linux guru.
-
Boring update...
it still runs..
mon day: 5 8 | hh:mm:ss: 15:07:48 | Still alive after 0k + 1 loops
.............
mon day: 5 9 | hh:mm:ss: 15:09:47 | Still alive after 219k + 471 loops
-
Bingo! crashed!
After 45 hours
The screensaver was not active but it did not help me much
flwm was active but did simply show a frozen screen.
I have no mouse but keyboard did not give any response.
I just restart without too much intelligence for reason I need the automizations that my application is running.
I will think of proper followup later, no time now.
I will anyways start it in text mode. "maybe console shows more clues when crashing in text mode"
But.... this is first time I was able to get TC17 crashing with a very simple isolated "crash testcase program" (as published few posts back).
==> I would say... "this is definitely a kext usb-serial-6.18.2-tinycore.tcz TC17 issue"
Any advice is welcome
First logging: mon day: 5 8 | hh:mm:ss: 15:07:48 | Still alive after 0k + 1 loops
.............
Last logging: mon day: 5 10 | hh:mm:ss: 12:21:54 | Still alive after 414k + 476 loops
-
Hi Stefann
... The screensaver was not active but it did not help me much
flwm was active but did simply show a frozen screen.
I have no mouse but keyboard did not give any response. ...
Did you try hitting Ctrl-Alt-F1 ?
Normally that should switch you to the console so you can see messages.
Ctrl-Alt-F2 returns you to the GUI>
-
Hi Stefann
... The screensaver was not active but it did not help me much
flwm was active but did simply show a frozen screen.
I have no mouse but keyboard did not give any response. ...
Did you try hitting Ctrl-Alt-F1 ?
Normally that should switch you to the console so you can see messages.
Ctrl-Alt-F2 returns you to the GUI>
Ah.. no..
And I rebooted already.
I did not really have time to explore a lot.
Anyway..
It’s now running in textmode. Maybe it will crash as well.
At least “getting it to crash” is kind of predictable.
Both my application and this isolated crash program crashed after about 45 hrs.
I keep it running for 48 hrs again. If it does not crash I try again with gui and take your inputs.
Progress is slow but steady.
At some point I will also test TC16 to see whether it did break with TC16 or TC17
-
boring update... its still running..
mon day: 5 10 | hh:mm:ss: 13:49:56 | Still alive after 0k + 1 loops
.....
mon day: 5 12 | hh:mm:ss: 15:04:58 | Still alive after 453k + 670 loops
Its now running significantly longer than previous run that crashed after about 414k loops.
Configuration changes are:
- I removed the "interrupt stress inducing lines" from the code because that did not seem to add much. It crashed after 45hours while "with the read() as part of my application" without additional stress it also crashed after 45hrs
- I now booted into text mode instead of GUI
So...
- either its natural variation
- or the lower stress increases the "time to crash" (maybe to indefinite)
I leave it running for an other day. If it did not crash by than I bring it back to previous configuration and try to crash it and look into console using the CTR-ALT-F1 & F2 keys.
Note,... It would have been much handier to keep the system in crashed state and await suggestions on this forum BUT the system is actively running my home automation. The home automation crashes it goes into all kind of "safe fallback modes" but I donot want to have it stalled for to long time. Also depends on my agenda. If I'm home I can afford it to be down better than when I'm not home.