Re: Tiny Core v17.0 upgrade issues

[Stefann]

Well,
I updated my Via Eden 500MHz single core system.
I succeeded the upgrade and my homecontrol application is running agin.
- including samba, ssh and vnc so that is nice.

Some things....
1/
After updating the applications I got this error:

Code: [Select]

error while loading shared libraries: libusb-1.0.so.0: cannot open shared object file: No such file or directoryI was able to fix that by adding libusb.tcz to the onboot.lst
The system is coming from TC15. with TC15 I only had libusb-compat-dev.tcz which apparently did load libusb.tcz as a dependancy but now it does not do that anymore.
Anyways... I got it fixed

[edit] NO... its not working. need to dig deeper

2/ rsyslog is not working
I'm starting it from bootlocal.sh and I tried to run it manual but than it complains its already running:

Code: [Select]

tc@huis:/opt$ sudo /usr/local/sbin/rsyslogd
rsyslogd: pidfile '/var/run/rsyslogd.pid' and pid 4789 already exist.However... nothing gets logged.

I did set debugging to level 2 in the rsyslog.conf file

Code: [Select]

$DebugFile /var/log/log_debug.txt
$DebugLevel 2 # 0=off, 2 = full debugAfter a reboot I indeed get an enormous /var/log/log_debug.txt file
However... nothing obviously wrong
But no logging

My rsyslog.conf file:

Code: [Select]

$WorkDirectory /var/log

$ModLoad imudp
$UDPServerRun 514
$ModLoad imtcp
$InputTCPServerRun 514
$ModLoad imklog
$ModLoad imuxsock

$template MyFormat,"%timegenerated:1:19% %msg:::drop-last-lf%\n"
$template FullFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"

$DebugFile /var/log/log_debug.txt
$DebugLevel 2 # 0=off, 2 = full debug

#example
# $outchannel log_rotation,/var/log/log_rotation.log, 52428800,/home/me/./log_rotation_script
#  activate the channel and log everything to it
# *.* :omfile:$log_rotation
#
# /home/me/log_rotation_script:
# mv -f /var/log/log_rotation.log /var/log/log_rotation.log.1

# ------------- User section -------------------
# max 100k debugfile = 1000 lines x 100chars

# Need script: /var/log/rotate:
# tail -n 500 /var/log/${1} > /var/log/${1}.tmp
# cat /var/log/${1}.tmp > /var/log/${1}
# rm -f /var/log/${1}.tmp

# Heat Control: HeatPump and CV
$outchannel heat_log, /var/log/heatlog.txt,  100000, /var/log/rotate heatlog.txt
local1.=info   :omfile:$heat_log;MyFormat


# Solar Thermal
$outchannel solar_log,/var/log/solarlog.txt, 100000, /var/log/rotate solarlog.txt
local1.=notice :omfile:$solar_log;MyFormat

# Main (normal main logging)
$outchannel main_log, /var/log/mainlog.txt,  100000, /var/log/rotate mainlog.txt
local1.=debug  :omfile:$main_log;MyFormat

# EVSE
$outchannel evse_log, /var/log/evselog.txt,  100000, /var/log/rotate evselog.txt
local2.=notice  :omfile:$evse_log;MyFormat

# Weather
$outchannel weather_log, /var/log/weatherlog.txt,  100000, /var/log/rotate weatherlog.txt
local3.=notice  :omfile:$weather_log;MyFormat

# PV, enphase and solaredge
$outchannel PV_log, /var/log/PVlog.txt,  100000, /var/log/rotate PVlog.txt
local3.=info  :omfile:$PV_log;MyFormat

# Debug, catch all that have not been catched before
$outchannel debug_log,/var/log/debuglog.txt, 100000, /var/log/rotate debuglog.txt
Any tips??

[edit]
I reverted back to original /mnt/sda1/tce/optional folder
So running TC17 updated core.gz and vmlinuz only. no tcz extension updates.
==> this runs fine!
libusb and rsyslog both working

[Stefann]

Note,
on the libusb...
I have the feeling the file "may be there", in /usr/local/lib
But can probably not be found in runtime.
I checked and the libusb tcz applications did not change (did they?)
So its probably a search path

I cannot check this assumption at this moment because I reverted back for reason the system is running my home automation and I cannot leave that in a non-functional state. I will need to find time to continue tomorrow.

But..... anyone who triggers on this and has some hint towards "search path" on /usr/local/lib libraries is welcome.
Note... I cannot imagine it's the gcc compiler as I'm not compiling anything. I'm just running an executable that can suddenly not find the libusb-1.0.so.0

[Juanito]

Ref:

Code: [Select]

cat /etc/ld.so.conf
/usr/local/lib
..tells the loader to look in /usr/local/lib as well as /lib and /usr/lib, which are searched by default.

If a library has been manually added the following command is required before the library will be found:

Code: [Select]

sudo ldconfig

[Stefann]

Thanks,
It’s strange because “just upgrading core.gz and vmlinuz” works. It derails with the application update.
I will have to go step by step tomorrow.
Takes quite some time because after each step I need to reboot and carefully test.
A full application update is also not superfast. Took about 20 mins today.

I will post progress.

[Stefann]

SOLVED!!
[edit] almost... usb-serial is not working now but it worked earlier

sooo.....
My target system is a Via Eden 500MHz single core very low power 32bit cpu running my home automation.
I also have a dual core Via Eden 1GHz cpu system that I normally use as test system.

I had TC17 already running on that test system but had not verified libUSB and rsyslog.

So what I did today:
- bring latest version of my application to the test system and try to run it on TC17
- it appeared to work with both libUSB and rsyslog.
==> Now I got suspicious towards update corruption on my target system

I did a compare of onboot.lst between the systems -> they where identical
I did a compare of the tce/optional folder between systems -> that did show major differences

I copied tce/optional folder from test system to target system
I did a reboot of target system
BANG.... all working. Application fully runs. rsyslog works. libUSB works

So...
What I suspect...
- To do the application update on the target system I used the gui over VNC as I donot feel very comfortable with the command line commands for apps-maintenance.
- the gui over vnc however already takes 40% cpu load of this "not so powerful 500MHz single core cpu"
- running the apps update took about 20mins during which the gui was completely frozen
- Afterwards I did a recheck on update and "it showed no lost items"
But....
- I expect the upgrade process had issues
- and although the gui for apps-maintenance is handy, with this slow processor it does not allow proper checking of logging on "whether all went ok"
- issues I wold consider "not super strange". I basically ran the system on close to 100% cpu use for 20mins. Disk being a compact flash card.

Anyways... result...

I can confirm TC17 runs on Via Eden 500MHz 32bit single core cpu
- including samba
- including ssh
- including vnc
- including libUSB
- including gcc (yes I tested a recompile of some part)
- including rsyslog

[edit]
still problem with usb-serial-6.18.2-tinycore.tcz kernel extension
I used to get text-data but when I run
cat /dev/ttyUSB0
I now get binary characters.
I guess it has something to do with settings
It's a bit strange as this 6.18.2 version DID work on TC17 core.gz and vmlinuz before I did upgrade all other apps.
I will figure this out I think

---------
Thanks for the replies & reading,.. I was a bit stuck yesterday.
Upgrading is always kind of a headache as its is a quite "big" thing
Very beneficial about upgrading of tinycore is that a simple rename of core.gz, vmlinuz, tce/optional and tce/onboot.lst allows you to switch between upgrade and original. VERY convenient!!!!

going forward..
- On a next update I will use the command line tools.
- That will give me better logging
- and NOT consume 40% cpu for vnc

[Stefann]

I can no longer edit the previous post

but.. usb-serial-6.18.2-tinycore.tcz kernel extension now also works

for some reason the baudrate setting got lost
Its a bit strange because that is set on every start of my application and I did restart it multiple times
But anyways... It now runs
Thats probably an "application weakness"

ALL WORKS

and thanks for the reading & helping

[Rich]

Hi Stefann

... - On a next update I will use the command line tools. ...

Tinycore has a single command that handles that. It's called:

Code: [Select]

update-everything

I can no longer edit the previous post ...

That's right. Years ago we added a 30 minute time limit after
which you can no longer edit a post.

The issue was twofold:
1. When you post, it shows up as a new post. When you edit a post
   someone already read, there is no obvious indication it was changed.
2. We had individuals changing information in posts that made replies
   to that post look irrelevant.

If you need to update information, add a new post.

If you have an important correction, for example, a command line typo that will
cause the command to fail or do something bad, add a new post requesting it
be corrected and it will be handled.

Either way, readers will remain informed.

[Stefann]

A...
I can now use some serious help....

As said in above posts, I succeeded to upgrade my Via eden 500MHz single core cpu 32bit system to TC17.
Everything works. But...
- After about 2 days it crashed. Application crashed, ssh crashed, VNC crashed. It's headless so the only thing I could do is a power cycle.

That "could have been a glitch"... but 2 days later it crashed again. I did just reboot as well.

Until last weekend the system did run on TC15. It could run multiple month without problem. In fact, I have not seen a crash since I started using TC 2 years ago.
So.... It very very very likely has something to do with the upgrade
(apart form TC17 this could also be human error or flash-card issues because I suddenly created big backup files).
The system runs my home automation.
- C program
- some PHP scripts
- some python scripts
- apache server
- vnc server (but currently no connected clients)
- samba
- ssh
- serial port over USB interface (with kernel extension)
Let's refer this as the "live system"

Apart from this system, I have a 2nd system that I use as a test setup. This system is a Via Eden 1GHz 2core cpu.
It's also upgrade to TC17. It already runs TC17 for some weeks without any glitch.
But..... it does not run the application.
Lets refer this to be the "test system"

I need to think of a clever plan to trouble shoot this:
- run the application on the "test system". It will not be able to connect to all peripherals but it will at least run code.
- step by step revert back the "live system". probably 1st run with TC17 core and vmlinuz but still non-upgraded applications.

any tips on how I can put logging in place?
- problem is that everything runs in ram. So I have zero diagnostics in case of a crash.

I probably have to connect a monitor to give me some hint in case of crash?

Any other trouble shoot ideas?

(I'm not expecting anyone to have an idea on rootcause at this stage, just tips on how to approach this would be very helpful)

[Rich]

Hi Stefann

... any tips on how I can put logging in place?
- problem is that everything runs in ram. So I have zero diagnostics in case of a crash. ...

Yes, but your extensions are on persistent storage, right?
So send any log messages to  /etc/sysconfig/tcedir/SomeFileName.log

[Leee]

With everything running in RAM and you backups suddenly getting bigger, maybe you're running out of memory?  Does the test system have more memory than the live system?

[Stefann]

Hi Stefann

... any tips on how I can put logging in place?
- problem is that everything runs in ram. So I have zero diagnostics in case of a crash. ...

Yes, but your extensions are on persistent storage, right?
So send any log messages to  /etc/sysconfig/tcedir/SomeFileName.log

True… but “what should I log”?
My application does log but just progress things

With everything running in RAM and you backups suddenly getting bigger, maybe you're running out of memory?  Does the test system have more memory than the live system?

No, backups are not suddenly bigger.
When going to tc17 I made copy of tce/optional folder. That occupies about 280M diskspace. But still 3G available.

[Stefann]

Oh by the way @rich…
I’m normally logging to ramdisk.
I normally do 1 single status diskwrite per day
For reason I have flash disk. I want to keep that healthy.
A bit extra temporal disk logging is ok.
But I don’t want mass logging on flash

[Stefann]

For those interested, output of “top”.
I don’t see anything suspicious.
My home automation application is: /krubo/work/krubo /krubo/work/1wire.def

Anyway…
It’s late now, I restarted the application and have a sleep.
Tomorrow I will undo the apps update but keep the tc17 core.gz and vmlinuz.
I will also start the application on my test system without pheripherals
Than run that for few days to see whether it holds.

If it fails I will go back to tc15 to see whether it still runs on tc15.
It has been running for multiple month on tc15 without a glitch but all the activity in upgrading may have degraded some old hardware (maybe the flash card isn’t healthy anymore?)
If tc15 works and tc17 does not, I will test tc16.

Code: [Select]


Mem: 410040K used, 551480K free, 34800K shrd, 4412K buff, 204960K cached
CPU:  5.4% usr  6.4% sys  0.0% nic 88.1% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.31 0.23 0.16 2/197 4900
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 4130     1 tc       S    11572  1.2   0 10.6 /krubo/work/krubo /krubo/work/1wire.def
 3481     1 root     S    54508  5.6   0  0.3 /usr/local/sbin/rsyslogd
 4880  4849 tc       R     3600  0.3   0  0.3 top
 4014     1 root     S     8192  0.8   0  0.2 x0vncserver -PasswordFile=/home/tc/.vnc/passwd
 4848  4844 tc       S     7968  0.8   0  0.2 sshd-session: tc@pts/0
 4017  3993 tc       S     442m 47.0   0  0.0 /usr/local/sbin/httpd -k start
 4019  3993 tc       S     395m 41.9   0  0.0 /usr/local/sbin/httpd -k start
 4050  3993 tc       S     391m 41.6   0  0.0 /usr/local/sbin/httpd -k start
 4314  3993 tc       S     382m 40.6   0  0.0 /usr/local/sbin/httpd -k start
 3993     1 root     S     162m 17.2   0  0.0 /usr/local/sbin/httpd -k start
 3995     1 root     S    44968  4.6   0  0.0 Xvesa -br -screen 1024x768x32 -shadow -2button -mouse /dev/i
 3821  3798 root     S    18868  1.9   0  0.0 /usr/local/sbin/smbd -D
 3798     1 root     S    18812  1.9   0  0.0 /usr/local/sbin/smbd -D
 4120     1 tc       S     9852  1.0   0  0.0 wbar
 4003     1 tc       S     9628  1.0   0  0.0 flwm
 4844  3855 root     S     7564  0.7   0  0.0 sshd-session: tc [priv]
 3855     1 root     S     7312  0.7   0  0.0 sshd: /usr/local/sbin/sshd [listener] 0 of 10-100 startups
 3641     1 root     S     6056  0.6   0  0.0 /usr/local/sbin/nmbd -D


[Rich]

Hi Stefann

... True… but “what should I log”? ...

Well, I would start with syslog to capture any system errors. If you included
the syslog boot code in your boot loaders config file, remove it.
Add this to /opt/bootlocal.sh:

Code: [Select]

/sbin/syslogd -O /etc/sysconfig/tcedir/messages -s 1000 -b 5 -l 5-O The log files will be messages, messages.0, ..., messages.4.
-s Each file will be limited to 1000K (1 Mbyte) before rotating to the next log file.
-b There will be 5 rotated logs kept for a total of 6Mbytes.
-l Log only messages of Warning or above.
These are the syslog message levels:

Code: [Select]

Level Severity Description Example Use Case
0 Emergency System is unusable Hardware failure causing a full system crash
1 Alert Immediate action required Database corruption detected
2 Critical Critical conditions Firewall dropping all traffic unexpectedly
3 Error Error conditions Authentication failures
4 Warning Potential issues High CPU or memory usage
5 Notice Normal but significant events Configuration changes applied
6 Informational General system activity User logins, completed backups
7 Debug Debugging information Function traces, variable dumps
You could also run free once every 5 minutes to keep a log of memory and swap usage.
Create a file called FreeMem.sh under .local/bin in your home directory and make it
executable (chmod 755 .local/bin/FreeMem.sh).
Place the following in that file:

Code: [Select]

#!/bin/sh

Dest="/etc/sysconfig/tcedir/FreeMem.log"

while true
do
echo -ne "$(date)\n$(free -m)\n\n" >> "$Dest"
# Limit log file to last 2400 lines (400 entries) if it is greater or equal to 2460 lines (410 entries).
[ $(wc -l "$Dest" | cut -d " " -f1) -ge 2460 ] && echo "$(tail -n 2400 $Dest)" > "$Dest"
sleep 300
doneThe log file will hold about 33 hours worth of entries and be about 100 Kbytes in size.

To launch it, enter this into /opt/bootlocal.sh:

Code: [Select]

/home/tc/.local/bin/FreeMem.sh &This assumes you are running as user tc. Change tc to your user name if required.

    [Edit]: Changed  messages.1, ..., messages.5  to  messages.0, ..., messages.4.  Rich

[Stefann]

Thanks for the suggestions. It helps.
It’s now 7am here in the Netherlands.
When I got up the system had crashed again. After about 8 hours.
Everything halted. Apache server not reachable, ssh not reachable.
So..
- Either the system is completely crashed
- Or the network has crashed
Some “non network peripherals” are also dead which indicates that the application is frozen. As far as I have tested earlier the application keeps running if network drops so I guess most changes are that system has completely crashed.

So.. it’s a bit chilly here now because the heatpump has not been operational this night.

Although the whole thing  could be hardware related as the TC upgrade process required me to do a lot more disk actions that this system normally does, my big suspecting is that it is TC17 related (or TC16 as it was running TC15).
That I expect makes it a valid entry for this thread (if I’m polluting something feel free to correct me on that).

What I will do:
- run the thing on TC17 core.gz and vmlinuz but with all TC15 applications (.tcz files).
- include syslog logging as suggested by @rich. Thanks for that! I’m normally using rsyslog and use the logging for application logging. I will however revert to the original syslog and use the commands as you suggested. I will plug-in an external usb and write to that. By doing so I will avoid write-stress on my compact flashdrive. I will pause all application logging for a while. That’s no big deal, I only need that for application development and that’s on halt now anyway.
- in addition start running the application without peripherals on my “test system” on TC17. I will also run the python and php “helper programs” on that. Those helper programs grab data from internet api’s (get weather info, get energy prices, read my car status). As this interaction is in a “less controlled environment” it could be that the issue is in those. It’s still very very very strange that any application failure can completely crash the OS.
- find and connect my old monitor and keyboard somewhere in the mess at the attic. I kept these for especially this reason (Hope I can find them). If the OS is not crashed but only the network that should allow me to operate the system after crash.

Guess that should bring me somewhere.
Again especially thanks @rich.
I had to think on “how to log without wearing out my compact flashdrive” but your suggestion is very key to “not being blind”.

Any additional suggestions are welcome.