Re: Tiny Core v17.0 upgrade issues

[Stefann]

For those interested:

This is the system that is constantly crashing.
I have the version without serial port.
https://ftp.emacinc.com/LegacyProducts/EmbeddedServers/ebox-4300/Manual/ebox_4300_manual.pdf

This is my 2nd system:
https://www.parkytowers.me.uk/thin/hp/t510/

[Stefann]

Ok….
It’s still running..
Way to early for a real conclusion but last few days I had a crash every 15-ish hours while now it’s running already for 24 hours.
- vanilla TC17.1 from download location
- no usb-serial, getting that data from an alternative source by modbus-over-ip
- usb with serial connection is physically unplugged
- usb-serial kext is not loaded
- extra 3x100ms delay added in application by “nanosleep()” command. As a result about 6.6% cpu load compared to about 10% earlier.

The theory I’m exploring a bit:
- it’s caused by overload. The extra delay brings 1/3 extra idle time so reduces the load.

The reason this at least “could fly”:
- system has a power requirement of 5V 3A. I’m however using a 5V 2A “mini ups”.
- long long ago I measured the power consumption and it was 6W, so,…. 5V 1.2A. For that reason I expected the 2A supply to be adequate (also I had difficulty to find a 3A mini ups for reasonable price).
- however… maybe the system draws current spikes that exhaust the power supply and cause power dips.

This would also explain why the crash is completely without logging. Even when having kernel logging over syslog enabled at the lowest log-level.

It’s still “not nice” if this is the root cause. I had chosen the VIA EDEN processor because it is extremely low power. It has a tpd of only 1 watt. It’s a bit stupid if a full system than still needs a 15watt supply.
But… that’s for later…
Priority is to get a solid fix on the root cause. In my experience (where I did a very big amount of trouble shooting in my work life) “finding the root cause “ is 80% of the work. Solutions are often relatively straightforward once root cause is solidly known.

@rich: yes, all tests use the same storage. The system uses a CF card as harddrive. I do only 1 write per day to that drive to avoid flash-damage, All other writes are on ram disk (that’s a major reason I like tiny core, it runs in ram).

We’ll see whether I get to an other crash……

[gadget42]

power supplies can become faulty over time. degraded board-level components can cause low/poor performance regarding hertz/voltage/wattage/etc. nothing lasts forever(naturally).

kudos and keep us posted!

[Stefann]

power supplies can become faulty over time. degraded board-level components can cause low/poor performance regarding hertz/voltage/wattage/etc. nothing lasts forever(naturally).

kudos and keep us posted!

Status: still running after 48 hrs… hope is growing.

To answer…. Well… yes…
The computer is from 2008 and had a 3A plug power adapter included. That supply died after few years. I replaced it with a 2A plug power adapter. That kept functioning for many years. It was powering my TC15 setup with zero crashes. Than crashes started to happen after upgrade to TC17.0.
So.. as TC15 was running “really crash free” I did not suspect the power supply.

Quite recently, I don’t know the exact date but I think after things were running smooth at self compiled kernel, I replaced the plug-supply with a mini-ups.
I got a crash after 5 weeks (although the mini-ups upgrade may have been during that timeframe).
But anyways… the mini ups is brand new. But… quite cheap… bought on Amazon but I expect directly from china:
https://www.amazon.nl/dp/B0F9VWNXQD?ref=ppx_yo2ov_dt_b_fed_asin_title&th=1

Still… long time ago I measured 6watt consumption at 5V, that’s only 1.2A.
But of course….. there may be current spikes and power dips.
Also... Application has massively grown since than. At that moment I don't think I had any network connected device that I was polling.

Anyway…. At this moment it’s theory.
I do not have an oscilloscope to check the supply on spikes.
I decided to try keep it running without touching for 5 days. That will be until Sunday afternoon EU time. If that goes well it’s definitely a sign.

In the mean time…
I installed powertop.
Output:

Code: [Select]

Power est.              Usage       Events/s    Category       Description
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. CX700/VX700 [S3 UniChrome Pro]
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. VX900/VT8xxx High Definition Audio Controller
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. VT82xx/62xx/VX700/8x0/900 UHCI USB 1.1 Controller
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. CX700M2/VX700/VX800/820-Series Serial ATA & EIDE-Controller
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. CX700/VX700-Series Host Interface Control
    0 mW    100.0%                      Device         USB device: UHCI Host Controller
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. CX700/VX700 Host Bridge
    0 mW    100.0%                      Device         USB device: UHCI Host Controller
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. CX700/VX700-Series North-South Module Interface Control
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. VT82xx/62xx/VX700/8x0/900 UHCI USB 1.1 Controller
    0 mW     59.4 pkts/s                Device         Network interface: eth0 (8139too)
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. USB 2.0 EHCI-Compliant Host-Controller
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. CX700/VX700-Series Error Reporting
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. CX700/VX700 Internal Module Bus
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. CX700/VX700-Series DRAM Bus Control
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. CX700/VX700-Series Power Management and Testing Control
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. CX700/VX700 Host Bridge
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. CX700/VX700-Series Bus Control and Power Management
    0 mW    100.0%                      Device         PCI Device: VIA Technologies, Inc. VT8237/CX700/VX700-Series PCI to PCI Bridge
    0 mW      0.0%                      Device         runtime-alarmtimer.0.auto
    0 mW      0.0%                      Device         runtime-pcspkr
    0 mW      0.0%                      Device         runtime-vga-framebuffer.0
    0 mW      0.0%                      Device         runtime-i8042
    0 mW      0.0%                      Device         runtime-PNP0C0C:00
    0 mW      0.0%                      Device         runtime-serial8250
    0 mW      0.0%                      Device         runtime-PNP0C04:00
    0 mW      0.0%                      Device         runtime-PNP0C0E:00
    0 mW      0.0%                      Device         runtime-PNP0103:00
    0 mW      0.0%                      Device         runtime-PNP0800:00
    0 mW      0.0 pkts/s                Device         nic:dummy0
    0 mW      0.0 pkts/s                Device         nic:tunl0
    0 mW      0.0%                      Device         USB device: EHCI Host Controller
Why do I not get power data? (all at 0mW)
Anybody has a suggestion for a fix?

Also, observing top:

Code: [Select]

Mem: 699732K used, 261736K free, 48616K shrd, 6768K buff, 329676K cached
CPU:  3.8% usr  3.8% sys  0.0% nic 92.2% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.05 0.20 0.17 1/216 11868
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 5399     1 tc       S    11584  1.2   0  6.4 /krubo/work/krubo /krubo/work/1wire.def
11868 11358 tc       R     3600  0.3   0  0.5 top
 3379     1 root     S    54620  5.6   0  0.1 /usr/local/sbin/rsyslogd
11357 11353 tc       S     7972  0.8   0  0.1 sshd-session: tc@pts/0
   15     2 root     IW       0  0.0   0  0.1 [rcu_sched]
   25     2 root     SW       0  0.0   0  0.1 [kcompactd0]
 3882  3873 tc       S     443m 47.1   0  0.0 /usr/local/sbin/httpd -k start
 3879  3873 tc       S     443m 47.1   0  0.0 /usr/local/sbin/httpd -k start
 3894  3873 tc       S     443m 47.1   0  0.0 /usr/local/sbin/httpd -k start
 4056  3873 tc       S     443m 47.1   0  0.0 /usr/local/sbin/httpd -k start
 3873     1 root     S     162m 17.2   0  0.0 /usr/local/sbin/httpd -k start
 3871     1 root     S    51160  5.3   0  0.0 Xvesa -br -screen 1024x768x32 -shadow -2button -mouse /dev/input/mice,5 -nolisten tcp -I
 3721  3707 root     S    18868  1.9   0  0.0 /usr/local/sbin/smbd -D
 3707     1 root     S    18812  1.9   0  0.0 /usr/local/sbin/smbd -D
 3969     1 root     S    11864  1.2   0  0.0 x0vncserver -PasswordFile=/home/tc/.vnc/passwd
 3991     1 tc       S     9852  1.0   0  0.0 wbar
 3876     1 tc       S     9844  1.0   0  0.0 flwm
 4154  3991 tc       S     8768  0.9   0  0.0 apps
 4158  3991 tc       S     8768  0.9   0  0.0 apps
 4156  3991 tc       S     8768  0.9   0  0.0 apps
 4157  3991 tc       S     8768  0.9   0  0.0 apps
 4160  3991 tc       S     8768  0.9   0  0.0 apps
 4161  3991 tc       S     8768  0.9   0  0.0 apps
 4159  3991 tc       S     8768  0.9   0  0.0 apps
11410  3991 tc       S     8768  0.9   0  0.0 apps
11411  3991 tc       S     8768  0.9   0  0.0 apps
11353  3747 root     S     7564  0.7   0  0.0 sshd-session: tc [priv]
 3747     1 root     S     7312  0.7   0  0.0 sshd: /usr/local/sbin/sshd [listener] 0 of 10-100 startups
 3546     1 root     S     6056  0.6   0  0.0 /usr/local/sbin/nmbd -D
 3269     1 root     S     3604  0.3   0  0.0 crond
    1     0 root     S     3604  0.3   0  0.0 /sbin/init
 3875     1 root     S     3604  0.3   0  0.0 ntpd -p pool.ntp.org
11864  3269 tc       S     3604  0.3   0  0.0 {exe} ash /krubo/work/krubo_watchdog
 3326     1 root     S     3604  0.3   0  0.0 /sbin/udhcpc -b -i eth0 -x hostname:huis -p /var/run/udhcpc.eth0.pid
11358 11357 tc       S     3600  0.3   0  0.0 -sh
 3310     1 tc       S     3600  0.3   0  0.0 -sh
11867 11864 tc       S     3468  0.3   0  0.0 sleep 3m
   90     1 root     S     2352  0.2   0  0.0 /sbin/udevd --daemon
  388    90 root     S     2352  0.2   0  0.0 /sbin/udevd --daemon
  392    90 root     S     2352  0.2   0  0.0 /sbin/udevd --daemon
   27     2 root     SWN      0  0.0   0  0.0 [khugepaged]
   56     2 root     IW       0  0.0   0  0.0 [kworker/0:2-eve]
   14     2 root     SW       0  0.0   0  0.0 [ksoftirqd/0]
   12     2 root     IW       0  0.0   0  0.0 [kworker/u4:0-ev]
   52     2 root     IW       0  0.0   0  0.0 [kworker/u4:3-kv]
 3356     2 root     SW       0  0.0   0  0.0 [cifsd]Only “slightly remarkable thing” is the 92.2% nic use. I constantly poll network connected devices so this is not strange.
is that alarming??

Also….
CPU has a tpd of 1watt.
If this goes down because of 10watt spikes it must be in the other components.

Any suggestions to “bring that consumption down”?
I don’t need a mouse, also not audio: any suggestion how I can disable that?

My current angle of attack:
- grow evidence that power weakness is the root cause. Try to have that “beyond doubt”.
- reduce power by disabling non-used components
- likely buy a plug-power meter, that will help to measure impact of disabling components
- also see to what extend tweaking the application can reduce power (although 10% cpu use is not alarming)
- experiment with making the powersupply more robust. Add 10.000uF condensator on supply line.

Yes.. I will keep posting.
lol… I think I have “most viewed thread over past 3 years" :-)

[Rich]

Hi Stefann

... Also, observing top:

Code: [Select]

Mem: 699732K used, 261736K free, 48616K shrd, 6768K buff, 329676K cached
CPU:  3.8% usr  3.8% sys  0.0% nic 92.2% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.05 0.20 0.17 1/216 11868

 ----- Snip ----- Only “slightly remarkable thing” is the 92.2% nic use. I constantly poll network connected devices so this is not strange.
is that alarming??

Actually, you misread the  nic  use, so no, it's not alarming.

What it really says is:
CPU usage (or cycles if you wish)
 3.8% usr - time running un-niced user processes
 3.8% sys - time running kernel processes
 0.0% nic - time running niced user processes
92.2% idle - time spent in the kernel idle handler
 0.0% io - time waiting for I/O completion
 0.0% irq - time spent servicing hardware interrupts
 0.0% sirq - time spent servicing software interrupts

Found here:
https://unix.stackexchange.com/a/499297

If you want to monitor the NIC, maybe iftop.tcz would be of some help.

[Stefann]

Hi Stefann

... Also, observing top:

Code: [Select]

Mem: 699732K used, 261736K free, 48616K shrd, 6768K buff, 329676K cached
CPU:  3.8% usr  3.8% sys  0.0% nic 92.2% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.05 0.20 0.17 1/216 11868

 ----- Snip ----- Only “slightly remarkable thing” is the 92.2% nic use. I constantly poll network connected devices so this is not strange.
is that alarming??

Actually, you misread the  nic  use, so no, it's not alarming.

What it really says is:
CPU usage (or cycles if you wish)
 3.8% usr - time running un-niced user processes
 3.8% sys - time running kernel processes
 0.0% nic - time running niced user processes
92.2% idle - time spent in the kernel idle handler
 0.0% io - time waiting for I/O completion
 0.0% irq - time spent servicing hardware interrupts
 0.0% sirq - time spent servicing software interrupts

Found here:
https://unix.stackexchange.com/a/499297

If you want to monitor the NIC, maybe iftop.tcz would be of some help.

Ah! Thanks. At least “reducing ip traffic” is not a high priority.

Note, I don’t intend to be non polite with the bold. Just draw attention to the questions I have.

Still unclear…
- any clue why everything is 0 mW in powertop?
- and hints on disabling hw to save power? I have seen examples of people who reduce power by 80% by  disabling all hw components they donot need. I did search a bit on internet but could not find a good lead to get me started on that,

[Rich]

Hi Stefann
I suspect your motherboard does not  measure/expose power readings.
I think powertop is intended for battery powered devices like laptops.

[Stefann]

Hi Stefann
I suspect your motherboard does not  measure/expose power readings.
I think powertop is intended for battery powered devices like laptops.

Thanks... so using power top has only limited value. Well... at least I have a view on "what devices are in use".

Short update: still running after 2.5 days (before it crashed 2x within about 15 hrs)

Small help request:
- My system has VGA by "integrated VIA UniChrome pro 2". Can I disable, unpowered that? Would I need to install org for that? or can I do by xvga? or is using the "text" boot option enough? I'm normally working on this system by ssh but I would need text-console "for emergencies", mainly to select "last functioning boot label" during startup.
- any way I can unpowered the audio and mouse (and does that even make sense?)
- any other suggestions to reduce power?

Status in more detail:...
- I bought a "mains plug energy meter" online, it will arrive tuesday
- I bought 4x 4700uF condensator to try de-spiking the 5V supply
- Last 2 days I was away and operated things remote from ssh, I'm back now and found the power adapter I had been using until I swapped to a mini-ups few weeks ago: It is 3.5A, not 2A. So.... Although the mini-ups I'm currently using is 2A and "formally underrated", The TC17.0 crashes I had until 4 weeks ago where with properly rated power supply. This still does not rule out the power supply. We will see how this evolves.....
- Last post I said I did not have an oscilloscope. Well... I have a very old one dated 1966, https://www.radiomuseum.org/r/philips_hf_zweistrahl_oszillogra.html
I have not used it for 15 years and had lost the probe-cables. But... I found the probe cables... So I can use it. I did not test it yet.

So... plenty of options for further testing...
However... Its now running for 2.5 day, I keep it running for 5 days, until Sunday afternoon EU-time. This is a very tricky problem. I do not want to draw wrong conclusions because of "too short testing" (in fact all conclusions between march & august now seem wrong).
For info: the application is basically an endless loop that continuously calls about 15 services. I had already included a configurable nanosleep() in that loop in case the loop would run "too fast". So.... The extra 100ms I inserted last Tuesday was done without any recompilation. Just by a start with a different configuration file. That makes the suspicion towards power dip quite strong. "zero software change".


Note: I certainly want a robust system that does not crash if the application is using a bit more cpu. But.... This is a 24/7 home control application. I also want this to run as low power as possible. I'm actually quite annoyed that it consumes 6W while the processor has a tad of only 1 watt. So my hunt to reduce power is not fully related to the crashes.
 

[Rich]

Hi Stefann
The same chip that is used for graphics is used for text mode, so
you wouldn't want to disable that. Using the "text" boot option is
the best you can do. Oh, and disconnect the video cable.

You probably didn't set up audio, so there's probably nothing to do there.

Unplug the mouse, though I doubt it will save much power.

Since your CPU load is less than 10% it's not working very hard. CPU power
scales linearly with frequency. So check the current governor setting:

Code: [Select]

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
See what other governors are currently available:

Code: [Select]

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
See how those other governors affect power consumption (example):

Code: [Select]

echo conservative | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governorYou'll definitely want your energy for this test.

You may also need to search for which Linux drivers are available for your CPU.

[Stefann]

Thanks!
Again.. thanks a lot, you’re helping me a lot!

Looks like I donot have a scaling_governor.
Anything I could do/try with what I do have?

Code: [Select]

tc@huis:/sys/devices/system/cpu$ ls
cpu0/            hotplug/         offline          present
cpufreq/         isolated         online           smt/
cpuidle/         kernel_max       possible         uevent
enabled          modalias         power/           vulnerabilities/
tc@huis:/sys/devices/system/cpu$ cd cpu0
tc@huis:/sys/devices/system/cpu/cpu0$ ls
cpu_capacity   firmware_node  power/         topology/
driver         hotplug/       subsystem      uevent
tc@huis:/sys/devices/system/cpu/cpu0$ cd power
tc@huis:/sys/devices/system/cpu/cpu0/power$ ls
autosuspend_delay_ms      pm_qos_resume_latency_us  runtime_status
control                   runtime_active_time       runtime_suspended_time
tc@huis:/sys/devices/system/cpu/cpu0/power$ cd ..
tc@huis:/sys/devices/system/cpu/cpu0$ cat cpu_capacity
1024
tc@huis:/sys/devices/system/cpu/cpu0$ cd ..
tc@huis:/sys/devices/system/cpu$ ls
cpu0/            hotplug/         offline          present
cpufreq/         isolated         online           smt/
cpuidle/         kernel_max       possible         uevent
enabled          modalias         power/           vulnerabilities/
tc@huis:/sys/devices/system/cpu$ cd cpufreq
tc@huis:/sys/devices/system/cpu/cpufreq$ ls
tc@huis:/sys/devices/system/cpu/cpufreq$
On “Linux drivers for my cpu”. I’m ok to search for that. But do you mean “drivers that load during boot”? That would be ok. I wil NOT flash any firmware or bios. If I make a mistake there I have high change to brick my system. I would probably need a very old windows system with a rom drive to flash the original firmware which I donot have. So… no flashing of firmware.

With all that said
- energy testing only when my energy meter arrives hopefully Tuesday.
- no system interruptions until Sunday afternoon to try for a 5 day crashfree run.

[Stefann]

I checked/searched a bit on drivers.
I don’t think there is an alternative for my cpu.
I did find drivers for the graphics. Via eden itself offers drivers on their download page.
It’s a bit of searching to see “what is what” and “whether I donot already have latest drivers”
Alternatively: they clearly support xorg, so I could simply install xorg as that is a tinycore package.

However… I kind of feel simply booting in text mode will maybe bring the best power saving I can think of. Is that strange?
As said… testing as of Tuesday when I have my energy meter.

Yes,.. since Tuesday  I’m running with unplugged monitor and unplugged keyboard.
I donot have a mouse with this system.

[Rich]

Hi Stefann
Try loading these 2 drivers:

Code: [Select]

sudo modprobe acpi-cpufreq.ko.gz
sudo modprobe p4-clockmod.ko.gzThen check for a governor again.

Also:

Code: [Select]

tce-load -wi util-linuxand provide the output of:

Code: [Select]

lscpu

[Stefann]

Hi Stefann
Try loading these 2 drivers:

Code: [Select]

sudo modprobe acpi-cpufreq.ko.gz
sudo modprobe p4-clockmod.ko.gzThen check for a governor again.

Also:

Code: [Select]

tce-load -wi util-linuxand provide the output of:

Code: [Select]

lscpu

I will do,
But after Sunday afternoon.
I don’t want to tinker with the system during a “5 day no-crash test”.
It may be safe to do, but I don’t want to risk.
And whatever it shows… I’m not going to apply changes until Sunday anyways.
Also don’t have the energy meter until Tuesday.

Thanks!

Note: although all this is annoying, it’s also kind of a fun expedition. And I’m learning a lot during this journey.

[Rich]

Hi Stefann
Agreed.

Making these kind of changes during a test could invalidate the results.