Tiny Core Linux
Tiny Core Base => TCB Talk => Topic started by: GNUser on June 15, 2022, 08:59:02 AM
-
My home wireless router is a repurposed 64-bit laptop running TCL13. The other day my wife lost wifi connection in the middle of an online class. When the issue came to my attention, everything seemed fine but when I SSH'd into the router I saw that, based on output of the uptime command, the router had rebooted in the middle of her class.
An unexpected reboot happened at least one more time since then: I rebooted the router approximately 72 hours ago, but when I SSH into the router and run uptime I get only 19 hours of uptime.
I have very few extensions loaded on the router (forum error, see attached), so I doubt an extension is rebooting the system without my knowledge.
SSH password authentication is disabled (keypair authentication is required) and only my user (user "bruno") is allowed to SSH into the router, so someone/something from the internet rebooting my router without my permission seems highly unlikely.
Is there anything in TCL base that automatically reboots the system? Can you think of any other explanation for unexpected reboots?
-
My guess would be hardware issue, but just in case
look at the TC logs for any info gleaned just before shutdown.
Other than TC,
I would make sure the fans and heat sinks are clear of lint and
fans spinning. look in the bios for any timers or power/thermal
state action setting or any NIC controls that may be present.
If the laptop is old and the internal battery is shot and you are
running it solely off its power supply make sure all the connections
to the power supply are secure and/or is not failing to where it may
be glitching off for a blink with bios set to auto reboot on power failure.
-
Hi, tacpilot. Thanks for the input.
Hardware failure is a good idea. The laptop is always connected to power adapter but it does still have its original (old and low capacity) battery. I would expect a hardware or power issue to lead to a poweroff rather than a reboot, but I will replace the battery just in case.
Checking TC log entries might be more difficult because only my /opt directory is persistent, so I would expect any log entries from a previous reboot command to have been lost.
This morning I remastered the base system (corepure64.gz) and deleted /sbin/reboot and /sbin/halt (which are just links to busybox). Now the only way to reboot is to run "busybox reboot" as root (or with sudo). If an intruder or misbehaving daemon just runs "reboot" or "halt", then they will actually be running shell scripts that I wrote and that will log a bunch of things to my persistent /opt directory. Hehehe.
-
You may also look at bios power settings to see if you can set it to stay off on power failure.
This way if intermittent power is the issue .., then the system will just stay off ..
This may help narrowing it to hardware and not some reboot option.
-
A kernel panic would reboot in one minute by default (configurable), but power loss sounds more likely.
-
Hi GNUser
Before replacing the battery, yank the power cord. If the battery keeps the laptop running for 15-20 minutes, you can save the expense
of replacing it for the time being.
I'm not aware of anything in Tinycore that would reboot the system, unless maybe a cron job was setup to do that.
You could also look into redirecting/forwarding syslog and/or dmesg messages to a persistent file or another machine through your LAN.
-
Thank you all for the valuable insights!
Rich, I will test the battery as you suggested before investing in a replacement.
Curaga, I didn't realize a kernel panic would trigger a reboot. This is very good to know. Are kernel panics logged? If so, where to? Perhaps I could create a persistent symlink (via mydata.tgz) from log's normal location to a file in my persistent /opt.
-
A kernel panic would reboot in one minute by default (configurable), but power loss sounds more likely.
Thermal issues due to failing fans and/or clogged heat sinks and/or failing thermal pad/compound
can possibly cause kernel panic when components start reaching their thermal limits.
-
Hi GNUser
I'm certain kernel panics are logged in dmesg. I'm pretty sure they would also show up in syslog.
tc@E310:~/Scripting/RecordScan$ syslogd --help
BusyBox v1.29.3 (2018-12-19 15:29:37 UTC) multi-call binary.
Usage: syslogd [OPTIONS]
System logging utility
(this version of syslogd ignores /etc/syslog.conf)
-n Run in foreground
-R HOST[:PORT] Log to HOST:PORT (default PORT:514)
-L Log locally and via network (default is network only if -R)
-C[size_kb] Log to shared mem buffer (use logread to read it)
-O FILE Log to FILE
-s SIZE Max size (KB) before rotation (default 200KB, 0=off)
-b N N rotated logs to keep (default 1, max 99, 0=purge)
-l N Log only messages more urgent than prio N (1-8)
-S Smaller output
-D Drop duplicates
tc@E310:~/Scripting/RecordScan$
Maybe you could start syslog something like this to catch what happens:
syslogd -D -s 0 -O /opt/`date +"%Y%m%d%H%M%S"`.logThat will create a log file whose name is a timestamps of when syslogd started in your persistent directory.
It will also create a new log file on every restart.
Note:
Those are backticks found on the ~ key around the date command, not single quotes.
-
That's a great idea, Rich. I've removed syslog from my boot commands. Going forward I'll start syslog using the command you suggested (as a line in /opt/bootlocal.sh).
This problem is all but solved. If it's a battery issue, I'll know later today when I do some tests. If it's an unauthorized reboot from a rogue daemon or an SSH intruder, it will get logged. If it's a kernel panic it will get logged.
If I discover anything interesting I'll update the thread. Thank you all.
-
A quick update: The laptop's battery is so old that it holds no charge whatsoever. However, it is not responsible for the unexpected reboots.
When I yank the power cord, laptop dies immediately. Quickly plugging it back in does not cause laptop to turn back on. So it seems that a brief power failure would cause a poweroff, not a reboot.
So, concerning the unexpected rebbots, it seems I've narrowed the list of suspects to occasional kernel panics, a background process, or (less likely) an SSH intruder.
-
...
So, concerning the unexpected reboots, it seems I've narrowed the list of suspects to occasional kernel panics, a background process, or (less likely) an SSH intruder.
don't rule out hardware issues. have had both laptops and desktops that developed "sudden shutdown and/or reboot syndrome" at varying equipment ages. sometimes the condition was solved by a thorough cleaning(down to a stripped motherboard-level wash) and reassembling(naturally this involves what some describe as "reseating" the board-level cards/modules/sub-assemblies/etc) but occasionally the condition remained a mystery as it becomes impractical to troubleshoot to the smc level.
thermal issues have been covered ad nauseam since bbs days so that should be a no-brainer.
hey, great repurposing nevertheless! kudos!
-
If it's caused by kernel panics and not associated with any recent software changes, I'd expect that to be caused by a hardware fault.
I'd run Memtest86 to see whether the RAM might be getting dodgy. It could equally be the CPU, or in my experience it's most often the chipset in which case what's failed will be some soldered-on chip that you can't replace. Though you'll have to swap the CPU to be sure of that last one.
-
If you look at the bios in the computer, some bios support the power on feature when power comes back in the cord.
-
Hi patrikg
That has already been ruled out:
... When I yank the power cord, laptop dies immediately. Quickly plugging it back in does not cause laptop to turn back on. ...
-
Ok, you guys convinced me to keep hardware issue on the list of possibilities. At this point I'm really hoping that the problem recurs soon so that we can have a log to look at ;D
If it turns out to be a hardware issue, I have a different laptop sitting on a shelf somewhere that could take over. That would be easier than swapping out this laptop's CPU or giving it the kind of cleanup you are suggesting. Let's wait and see what the log shows.
-
You may want to install setup something like lm-sensors so as to include
temperature monitoring in your logs. Correlating temps to some random
crash could help narrow the cause.
I'd run Memtest86 to see whether the RAM might be getting dodgy.
Should always be in the top of your toolbox .. Wouldnt be the first time I've seen
mem causing issues.Then pulling out, cleaning, and reinstalling the same
sticks fixed the issue.
Thermal issues can cause mem issues .. especially being a laptop, deff need watch temps..
-
Thanks, tacpilot. I installed lm-sensors and will keep a persistent temperature log. I will also check the RAM.
-
Just a reminder to check the sensor logs to make sure they are generating as expected.
-
It happened again in the middle of the night last night. syslog shows nothing unusual (in fact, last entries in the syslog were 10 minutes before the reboot). I have a script that logs temperatures every 10 seconds and the last entry shows normal temperatures. Whatever caused the reboot didn't run "reboot" or "/sbin/reboot" because I deleted /sbin/reboot and have a script in my PATH called "reboot" that does nothing except create a log entry.
Could it be that even though this laptop can operate with a broken battery (or even no battery), sometimes some low-level piece of software (e.g., BIOS or kernel) causes a reboot when it sees a broken battery or missing battery?
This morning I removed the harddrive and put it in a different laptop (with new battery, different power adapter, and different RAM). If it happens again I think it would point to something in the OS or one of the background processes.
P.S. The only hardware that remains the same in my setup is the modem, powerstrip, and harddrive (plus everything on it, namely TCL13 and my extensions). I will replace the powerstrip and harddrive soon so that I can confidently exclude a hardware issue.
-
Neither syslog nor my temperature log gave me any leads the last time there was an unexpected reboot. Can you guys think of anything else I should be logging?
-
Hi GNUser
... Can you guys think of anything else I should be logging?
Power supply voltages (and currents) if possible.
... Whatever caused the reboot didn't run "reboot" or "/sbin/reboot" because I deleted /sbin/reboot and have a script in my PATH called "reboot" that does nothing except create a log entry. ...
Doesn't prevent someone from running this:
sudo busybox reboot :o ::)
There are some watchdog options that are set in the kernel configuration. You could try adding this boot code:
nowatchdog
Post the results of this:
cat /proc/interrupts
These suggestions are for the original laptop since I suspect that and not the hard drive.
-
Doesn't prevent someone from running this:
sudo busybox reboot :o ::)
Of course not ;) Not having /sbin/reboot is just to stop processes that may try to reboot but are too unsophisticated to do "sudo busybox reboot". Until I sort this out, "sudo busybox reboot" and "sudo busybox poweroff" is how I myself and rebooting and powering off my "router".
Logging power supply voltages and currents is a good idea. I will arrange that.
Thanks for the "nowatchdog" boot code. I will try it on the new machine if the problem recurs.
I don't have access to the old machine at the moment. I will post output of "cat /proc/interrupts" on the new machine if the problem recurs.
-
Hi, Rich. I'm at work but my wife just let me know that WiFi went down again. I can confirm via SSH that the new laptop-router (i.e., different RAM, new battery, different power adapter, different BIOS) rebooted at 11:51 a.m. my local time.
I will attach syslog, temperature/voltage log (from within 10 seconds of the reboot), and output of "cat /proc/interrupts"
Things are pointing to a software issue but I have no idea how to clinch it. I'm so frustrated by this :'( At this point I'm completely out of ideas.
-
I'm getting a sinking feeling about this. I'm going to try downgrading the laptop-router to TCL12 x86_64. I don't remember ever having this problem in the past.
Rich, after you've downloaded the three files above to inspect them, would you kindly delete the attachments from my post? There are some private things in there (MAC addresses, etc) that I'd prefer not to be a permanent part of this thread.
-
Hi GNUser
Attachments removed. :)
There's a lot going on in that log file and I don't understand all of it. A few things did catch my
attention, though I don't know if they are problems.
Two copies of dnsmasq being launched:
Jun 17 09:40:12 x200 daemon.info dnsmasq[3531]: started, version 2.79 cachesize 150
Jun 17 09:40:12 x200 daemon.info dnsmasq[3531]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
Jun 17 09:40:12 x200 daemon.info dnsmasq-dhcp[3531]: DHCP, IP range 192.168.x.x -- 192.168.x.x, lease time 1d
Jun 17 09:40:12 x200 daemon.info dnsmasq-dhcp[3531]: DHCP, sockets bound exclusively to interface wlan1
Jun 17 09:40:12 x200 daemon.info dnsmasq[3531]: reading /etc/resolv.conf
Jun 17 09:40:12 x200 daemon.info dnsmasq[3531]: using nameserver 75.75.75.75#53
Jun 17 09:40:12 x200 daemon.info dnsmasq[3531]: using nameserver 75.75.76.76#53
----- Snip -----
Jun 17 09:40:13 x200 daemon.info dnsmasq[3531]: using nameserver 193.138.218.74#53
----- Snip -----
Jun 17 09:40:13 x200 daemon.info dnsmasq[3761]: started, version 2.79 cachesize 150
Jun 17 09:40:13 x200 daemon.info dnsmasq[3761]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
Jun 17 09:40:13 x200 daemon.info dnsmasq-dhcp[3761]: DHCP, IP range 192.168.x.x -- 192.168.x.x, lease time 1d
Jun 17 09:40:13 x200 daemon.info dnsmasq-dhcp[3761]: DHCP, sockets bound exclusively to interface wlan2
Jun 17 09:40:13 x200 daemon.info dnsmasq[3761]: reading /etc/resolv.conf
Jun 17 09:40:13 x200 daemon.info dnsmasq[3761]: using nameserver 193.138.218.74#53
----- Snip -----By the way, that's the only reference to wlan2 I see in there.
This series of errors:
Jun 17 11:07:26 x200 auth.info sshd[5301]: Accepted publickey for bruno from 162.x.x.x port 47x ssh2: RSA SHA256:Secret
Jun 17 11:07:29 x200 auth.err sshd[5303]: error: connect_to 192.168.x.161 port 22: failed.
Jun 17 11:07:29 x200 auth.info sshd[5306]: Accepted publickey for bruno from 162.x.x.x port 47x ssh2: RSA SHA256:Secret
Jun 17 11:07:32 x200 auth.err sshd[5308]: error: connect_to 192.168.x.62 port 22: failed.
Jun 17 11:07:35 x200 auth.info sshd[5309]: Accepted publickey for bruno from 162.x.x.x port 47x ssh2: RSA SHA256:Secret
Jun 17 11:07:38 x200 auth.err sshd[5311]: error: connect_to 192.168.x.161 port 22: failed.
Jun 17 11:07:39 x200 auth.info sshd[5314]: Accepted publickey for bruno from 162.x.x.x port 47x ssh2: RSA SHA256:Secret
Jun 17 11:07:42 x200 auth.err sshd[5316]: error: connect_to 192.168.x.62 port 22: failed.For some reason your (LAN) subnet was alternating between 62 and 161, or 2 devices were trying to connect at about the same time.
Then I saw this message amongst the traffic 2 seconds before the end:
Jun 17 11:48:27 x200 daemon.notice hostapd: wlan1: STA xx.xx.xx.xx.xx.xx IEEE 802.11: did not acknowledge authentication response
-
Hi, Rich. Thank you very much for looking it over. I have two wireless USB adapters connected to the laptop (wlan1 and wlan2). There are two instances of hostapd and two instances of dnsmasq, one for each wireless USB adapter. So that's all fine. wlan2 is there for when we have guests staying over. We don't have any guests visiting us right now, so wlan2 isn't doing much.
Some devices on the network get an IP via DHCP (192.168.x.100-192.168.x.200) and others get static addresses (192.168.x.2-192.168.x.99), so that's also fine.
Sometimes devices in my home network get powered off or go into suspend, so that might explain the lack of authentication response, although I'm not 100% sure about that final excerpt that you quoted.
Thanks for deleting the attachments. I have downgraded the laptop-router's OS to TCL12 x86_64. Let's see how that goes.
-
at work cant rem password and reset option not seem to be working.
Still leaning towards hardware.
My next guess was for failing RAM or HDD..
since you have removed all the other options from the
equation, then it would appear to be a failing HDD.
You should be able to verify by looking at the logs stored
in the HDD it self and/or running drive health checking tools.
Till then, an easier method to see if some weird software may
be at fault, you can remove the HDD from the equation.
- Set the (copy2fs.flg) so that everything gets copied to RAM.
- use bootcode noswap to ensure its not trying to use the drives swap space
- then unmount anything mounted to that drive
at that point, even if the drive goes offline, it wont matter to TC
-
TCL uses harddrive at boot but then runs from RAM, so a harddrive failure shouldn't cause a crash and reboot. Nevertheless, I will be replacing the harddrive soon just to exclude all hardware.
-
looking at the corebook.pdf .. page 8
1.9. Copy mode...
seems to indicate unless the copy2fs.flg is set ..
any externally loaded extensions are mounted on drive not in RAM.
if swap space exists on drive it is mounted by default as well
betting new HDD will solve the issue
-
HDD health can be checked with "smartctl -a /dev/sda" (from smartmontools.tcz), but that's not likely to cause a reboot.
-
I have two wireless USB adapters connected to the laptop (wlan1 and wlan2).
If you used them with both laptops then it's also (slightly) possible that one is failing and that's exposing a bug in the wireless driver which causes a kernel crash.
-
A kernel panic would reboot in one minute by default (configurable)
Since rebooting is not configured for any power related settings, and other hardware
have been removed from the equation, the only thing seems able to force a reboot
is the kernel panic setting.
Going on the assumption that a (failing drive / data corruption) is causing kernel panic.
If you used them with both laptops then it's also (slightly) possible that one is failing and that's exposing a bug in the wireless driver which causes a kernel crash.
Following previous suggestion to ensure are running solely from RAM, would
help expose such issues if reboots continue.
Could go as far as to set the reboot time for kernel panic to extreme high number
so as to visually verify that a panic is what's happening before it gets a chance to reboot.
-
smartmontools shows nothing concerning and total bytes written = 0.172373 TiB (well below the drive's expected useful life).
No reboots for over 12 hours, which is a record since this issue came to my attention :) The only difference between current setup and the previous iteration (which did have an unexpected reboot) is the TCL downgrade to version 12.
I'm chalking this up to a linux kernel regression between 5.10.3 and 5.15.10 that affects this specific use case (wireless router with multiple wireless interfaces). But with the other small changes between TCL versions it's hard to definitively exclude some other possibility.
Topic may be marked as solved. If I have bad news (another unexpected reboot) or good news (a week or more without unexpected reboots), I'll let you all know.
You are a sharp and generous bunch. Thank you very much for all your help.
P.S. CNK, in one of the several iterations of my setup I did change the wireless USB adapters to different ones with identical chipset (Ralink RT5370) and the issue recurred. So it was not a hardware problem with the adapters.
-
Downgrading to TCL12 x86_64 seems to have done the trick: No more unexpected reboots for 72 hours and counting! Maybe the new kernel version in TCL13 x86_64 had a regression that affected this particular use case.
Topic may be marked as solved :D
P.S. The laptop has been running without a battery at all. So my hypothesis that a missing or broken battery may have been responsible for the reboots seems to have been disproven.
-
Hi GNUser
... Maybe the new kernel version in TCL13 x86_64 had a regression that affected this particular use case. ...
Here are 2 other possible causes:
1. One or more extensions were updated.
2. One or more extensions were not updated but should have been.
Topic marked as solved. :)
-
Hi, Rich. If I were to do any more testing now and cause additional router reboots at bad times, wife would kill me.
To exclude your hypothesis #2, when TCL14 comes out I'll submit new versions of the two workhorse extensions (hostapd.tcz and dnsmasq.tcz) and will test it out.
-
Hi GNUser
It was merely an observation, not a suggestion to do more testing.
-
Hi, Rich. I know--that was my bad conscience putting me on the defensive because I am lazy and didn't want to do what I knew needed to be done.
I went ahead and compiled latest hostapd and dnsmasq on TCL13 x86_64 and am testing everything on my router. Worst that can happen is that I have one more unexpected reboot, I have to downgrade to TCL12 on the router (again), and wife makes me sleep on the sofa for one night ;)
Juanito, I will submit refreshed hostapd and dnsmasq extensions for TCL13 x86_64 now.
-
Hi GNUser
... and didn't want to do what I knew needed to be done. ...
Once again, It didn't need to get done. You solved the issue in a way that met your needs, and
there's nothing wrong with that.
Allow me to play the role of PITA:
... Maybe the new kernel version in TCL13 x86_64 had a regression that affected this particular use case. ...
Here are 2 other possible causes:
1. One or more extensions were updated.
2. One or more extensions were not updated but should have been.
3. An existing bug was made more visible by the new kernel version. The issue will recur in the older
kernel, but not for 53 days, You will not make the connection because you will have forgotten
about this thread by then. ;D
-
Good news: I've had no unexpected reboots while running the router with TCL13.1 Pure64 and the updated hostapd.tcz and dnsmasq.tcz extensions. It's been 48 hours and counting.
I'm glad I pressed on. Refreshing the two extensions seems like a more proper fix (and more likely to help the community) than just downgrading to TCL12.
Thank you all, especially Rich, for your help in coming up with the list of possible root causes. This was a tough one!
-
Hi GNUser
Congratulations on your success and avoiding nights on the couch. ::)