Tiny Core Linux
Tiny Core Base => Raspberry Pi => Topic started by: xuraax on August 13, 2022, 02:40:01 AM
-
I would like to run a Python script, which runs well from command line, directly after booting on a headless Pi.
I have read previous help on this topic but nothing seems to work for me including running the script with &, inserting a sleep 60 delay before the " python3 /home/tc/trial.py & " in the bootlocal.sh file. Even a simple python file consisting of an infinite loop which prints just a string to screen. Inserting " echo 'Hello There' " directly into bootlocal.sh also produces nothing. Clearly I am missing something quite basic.
Any help appreciated.
xuraax
-
Hi xuraax
Try adding the command to the end of .profile in your home directory.
-
Hi Rich.
Thanks for your suggestion.
I can confirm that the script does run when invoked from the .profile file. However for some reason it does not operate in the same way as it does when run from the command line. I still need to analyse this properly.
I first thing I noticed was the following:
My script turns on an LED to signify that it is functioning and then goes into a loop to establish communication via the UART. Each attempt is confirmed by a print statement which signals the attempt number.
The strange thing is that even if a wait for several minutes before I SSH into the system when I am certain that the link has been established, the message that appears on the screen is as if the attempt in communication has just started.
My other question is: why did this script not work at all in bootlocal.sh?
Regards
-
It would be good to see your Python script and how is it called. Without information it is just guessing.
-
It would be good to see your Python script and how is it called. Without information it is just guessing.
I have no problem with sending the script but I must warn you it is about 200 lines long and, coupled with the fact that I am not an expert programmer, it can get confusing. Just tell me how to proceed and i will comply.
The script is built around an old RPI to which I added a small 2 transistor circuit to the UART to interface over the K-line to my bike's ECU. Once logged on it collects all data and sends it to a csv file which I further process off line using Excel. The idea is to collect live data whilst on the move.
Regards
-
After writing the above mail I realised I could demonstrate what is happening by writing a very simple script which I called trial.py and invoke that in the .profile file. The script simply consists of the following:
import time
i=0
while True:
print("Sample no.: ",i)
i+=1
time.sleep (1)
In the .profile file I inserted
sudo python3 trial.py
Note that I need to use sudo in my own program since this is required to access the UART.
After saving and rebooting and SSHing into RPI one notes that the first thing that comes on the screen is "Sample no.: 0"
Keeping the script running and exiting SSH and then reentering again with SSH, the screen still comes up with "Sample no: 0" even if I allow some time to pass before reconnecting.
Also with the script is running and you do ^C the script appears to terminate with the correct error message. However in fact the script is still running in the background and doing ps aux
shows that python3 trial.py is still running with its own job number.
-
If i don't misunderstand your request, you could do it like this.
.profile is running when entering the shell.
Too keep the script running after exiting the shell you could add nohup to the script so the shell don't exit its child processes.
https://linuxhint.com/how_to_use_nohup_linux/
-
Too keep the script running after exiting the shell you could add nohup to the script so the shell don't exit its child processes.
nohup is an interesting command but it does not achieve what I want. What I am looking for is the following:
1. when I power up my RPI, my script initiates and goes about its work collecting data loop after loop.
2. at some later point when I want to "look in" I would like to SSH into the RPI to see what is happening during that particular loop. if say at teh point of entry the variable in my trial.py happens to be i = 1000, I would like to see teh following on screen:Sample no.: 1000
3. while here I could decide to interrupt my loop and stop the script completely or continue monitoring on screen the data being collected.
In the past, I have done something similar (but not with autostart) by using TMUX which allows you to look into a process by attaching or deattaching from that process.
-
Use 'screen'
-
Screen appears to be another version of tmux.
what I tried was to insert in the .profile file:
screen -S mysession
sudo python3 trial.py
in the hope that, when I later ssh into the box, I can do: screen -r mysession
to reconnect but this did not work because I had not deattached from "mysession" using ^A^D
after initiating the script.
Doing: screen -ls
listed 2 still attached processes named "mysession" (why 2?)
I guess this method could work if I am able to somehow invoke ^A^D from the .profile file to deattach "mysession".
any ideas?
-
Hi xuraax,
Just my 2 cents here, as I know almost nothing about Python.
I think you can execute Python code from bootlocal.sh (I don't see any reason why not), but I don't think you can write any output to a terminal or console. When bootlocal.sh runs, there is no tty connected to that process, afaik.
The easiest way I can see is to log your output to a file and then reading that file when you ssh into the system. If you want that fully automated, you could add
tail -f <FILE> to your .profile script.
To avoid the file growing until eternity, maybe you should also write some kind of cleanup mechanism that clears the whole file, or certain entries in the file.
Cheers.
-
Hi xuraax
... The script is built around an old RPI to which I added a small 2 transistor circuit to the UART to interface over the K-line to my bike's ECU. Once logged on it collects all data and sends it to a csv file which I further process off line using Excel. The idea is to collect live data whilst on the move.
So it sounds like you probably have the data collection working and now want to be able to display snapshots periodically.
My recommendation is you run your program in the background and do not print anything to the screen.
Print the data to a file (Data.txt) and overwrite the file whenever fresh data comes in.
Use a second file (Status.txt) initialized to 0 to synchronize accessing the file:
When you want to read the contents of Data.txt, set the contents of Status.txt to 1.
When your python program sees the read request and is done updating Data.txt, it sets the contents of Status.txt to 2.
As long as the contents of Status.txt remains equal to 2, your python program skips updating Data.txt.
When you see Status.txt has been set to 2, it is safe to read the file.
After you have read Data.txt, you set Status.txt back to 0 which tells your python program it is safe to resume updating Data.txt.
-
Hi xuraax
The following can be used to monitor the contents of Status.txt and display the contents of
Data.txt when your python program signals that it is safe to do so.
#!/bin/sh
DataFile="Data.txt"
StatusFile="Status.txt"
# Create Status file if it doesn't exist.
[ -f "$StatusFile" ] || echo 0 > "$StatusFile"
while true
do
sleep 2
status="`cat $StatusFile`"
case "$status" in
0) # Signal that we want to read DataFile.
echo 1 > "$StatusFile"
;;
1) # Wait for status to change to 2.
continue
;;
2) # Clear the screen, display the data, and reset StatusFile to 0.
clear
cat "$DataFile"
echo 0 > "$StatusFile"
;;
esac
done
-
I think you can execute Python code from bootlocal.sh (I don't see any reason why not), but I don't think you can write any output to a terminal or console. When bootlocal.sh runs, there is no tty connected to that process, afaik.
Thank you for your suggestions. In fact my full script includes also an LED output to signal what is happening and one input to test the position of a push button. As you proposed I have removed all tty output and rely on the LED to tell me if the script has started, if connection has been established and then that data is being collected and stored to a file. The push button interrupts the loop and closes the data collection file and then the script itself.
This still did not work for me in bootlocal.sh but does work using the .profile file. Maybe the reason why the former method is not working for me could be because my script needs to use UART 0 which is probably being used by the system as console output during the booting process.
Anyway...I still would like to explore further the use of screen
invoked from the .profile file and it seems that if I can manage to invoke ^A ^D
from here I may be able to achieve my aim. Indeed looking further into the commands available in screen
I noted a command called detach
which I had hoped would do the trick for me but alas no joy!
-
The following can be used to monitor the contents of Status.txt and display the contents of
Data.txt when your python program signals that it is safe to do so.
Thank you Rich for your suggestions. I will consider it. Another possibility would be to save to a different .csv file every, say, 1000 samples from the script running in background and just open up a particular .csv when I log in with ssh.
I still would like to see if can "peek in" into my running script using "screen" if it is at all possible.
-
You can remove the tty from command line argument to the kernel, the console.
So you free the UART.
And remove the tty in inittab.
-
You can remove the tty(serial) from command line argument to the kernel, the console.
(/boot/cmdline.txt). So you free the UART/serial port.
And remove the getty tty(serial) in /etc/inittab.
And remove the startserial.sh if you running that script, in boot script.
When i referring to tty(serial) you have to consider that the serial port have lots of different names.
console=serial0,115200
console=ttyAMA0,115200
Just have the console=tty1 in kernel command line argument(tty1 is the first text mode virtual console.), if you using raspberry pi with hdmi.
You can remove all console and using headless raspberry pi with out any console.
Just access the RPI with network ssh.
You can also do the following when using headless rpi:
Add the line to config.txt to gain little more memory, because you don't use the hdmi/gpu
gpu_mem=16
And the user tc have to have rights to the port, lots of linux distributions uses some type of group to the serial port, so if you add the default tc user the the correct group you don't need to use sudo for your user to use the serial port.
To easy see what group you should add the tc user to, you just list the serial device in dev.
Lets see what my arch linux pc com1 serial port.
[root@arch-new ~]# ls -la /dev/ttyS0
crw-rw---- 1 root uucp 4, 64 18 aug 09.18 /dev/ttyS0
[root@arch-new ~]#
Ahhh it's uucp.
So if my regular user need com1 I just need to add that user to group uucp.
Happy hacking.
-
And the user tc have to have rights to the port, lots of linux distributions uses some type of group to the serial port, so if you add the default tc user the the correct group you don't need to use sudo for your user to use the serial port.
To easy see what group you should add the tc user to, you just list the serial device in dev.
Lets see what my arch linux pc com1 serial port.
[root@arch-new ~]# ls -la /dev/ttyS0
crw-rw---- 1 root uucp 4, 64 18 aug 09.18 /dev/ttyS0
[root@arch-new ~]#
Ahhh it's uucp.
So if my regular user need com1 I just need to add that user to group uucp.
Happy hacking.
Thank you, i will certainly try out several of your suggestions.
With regards to the one above, I have tried in the past, admittedly on other versions of linux, to add my user to the correct group but this still was not enough. I needed also to give read/write options to the users besides root. This worked until the next boot, which reinstated the restrictions. The second line from your code quote would always reset to: crw--w---- etc. after reboot.
-
Hi xuraax,
To find out why your script won't run from bootlocal.sh, you could add debug code into your python script: at important points within your script, log variable values to a log file and make sure to redirect error messages to a log file. Then after boot, you can verify why the script does not behave as expected.
Does the system log (/var/log/messages) tell you anything useful?
Alternatively, you could connect a screen to your Pi while starting up. Per default, TinyCore starts a shell for the tc user during boot and shows boot output on the terminal for that user. Although often it is a lot of output within a couple of seconds. In the past, I even had to resort to recording the output on my smartphone and then playback in slow motion to capture the error messages that I needed :-)
-
I guess your script requires user privilege
bootlocal.sh is run by root
su tc -c 'python3 yourscript'
-
I guess your script requires user privilege
bootlocal.sh is run by root
su tc -c 'python3 yourscript'
Why is this better than?:
sudo python3 myscript
-
Every command in bootlocal.sh is run by root
Every command in .profile is run by tc
To run a command as tc in bootlocal.sh, you need to switch user.
sudo python3 myscript
This line will be run as root, or may not execute at all.
-
Hi,
I have just discovered why my Python script does not run at all when run from bootlocal.sh. It appears that in this mode, the script starts. It then stops when it attempts to import the "serial" module issuing an error which is not reported when you connect headless but can be seen when connecting a monitor to the HDMI port.
What can I do to get the "serial" module visible ALSO at startup (from root)?
-
Hi xuraax
... It then stops when it attempts to import the "serial" module issuing an error ...
What is the error message?
Where does the imported module come from?
Does it get download over the network? If so, it's possible the network is not up yet.
This will block (up to 60 seconds) until the network is up:
# Wait for network to come up
SEC="60"
while [ $SEC -gt 0 ]
do
SEC=$(($SEC - 1))
ifconfig | grep -q "Bcast:" && break
sleep 1
done
-
The problem is not network related.
When I boot up with the USB dongle removed I get the same message. When I then run "sudo python3 test.py" there is no issue which implies the python module "serial" is inside somewhere.
The message I get at startup is:
ModuleNotFoundError: No Module named 'serial'.
Python reports the script name and the line line number where my offending "import serial" resides.
Also when I remove the offending line from my script, the script runs properly at boot up.
it seems running python3 from root is different than running python3 from sudo.
-
My suggestions is to try following.
Add the path to the working directory to sudo like this in bootlocal.sh:
sudo -D /home/tc /home/tc/test.py
So sudo change the working directory before executing the test.py script.
Add the shebang line at the first line in the test.py python file like this:
#!/usr/bin/env python3
Add the execution bit to the python file:
sudo chmod u+x /home/tc/test.py
Happy hacking :)
-
I'm pretty sure there is no serial module included with python3. You must have installed pyserial into a local user account which is not available when using bootlocal. Look in /home/tc/.local/lib/...... do you see python modules?
What extensions are loaded
-
Will try but in the meantime, can you please answer the below questions?
Your suggestion 1: understood.
Your suggestion 2: What does this do? I always understood that a line started with # is ignore by the interpreter.
Your suggestion 3: Why does the execution bit need to be set given that this programs runs in the python3 interpreter?
Thanks
My suggestions is to try following.
Add the path to the working directory to sudo like this in bootlocal.sh:
sudo -D /home/tc /home/tc/test.py
So sudo change the working directory before executing the test.py script.
Add the shebang line at the first line in the test.py python file like this:
#!/usr/bin/env python3
Add the execution bit to the python file:
sudo chmod u+x /home/tc/test.py
Happy hacking :)
-
I'm pretty sure there is no serial module included with python3. You must have installed pyserial into a local user account which is not available when using bootlocal. Look in /home/tc/.local/lib/...... do you see python modules?
What extensions are loaded
It has been some time since I installed pyserial so I don't recall how I did it. Probably through pip.
Here are the contents of the directories you mentioned:
~/.local/lib/python3.8/site-packages/
pyserial-3.5.dist-info/
serial/
Both are directories.
The first looks like it contains text files.
The second contains several python scripts but none of these are called "serial.py"
For me the simplest solution would be to have all modules available even from root. i hope this is possible?
regards
-
So you installed pyserial using pip with the --user flag. That will only work for user sessions. Delete all of the python libraries from .local
Then install the extension py3.8serial.tcz
-
When the execution bit is set, you telling the system that the file is executable.
It can be some elf(bin asm) file or it can be a script file. If it's a script file the system don't know what interpreter it should run.
So you need to tell the system what interpreter it should run.
You telling that in the first line, and that is also know as the shebang line.
You can see that line in lots of script languages, like in bash script.
Lets make an example.
First make this file.
nano test
#!/usr/bin/bash
echo "Hello World!"
Then you have to tell the system that the file is executable, you do that with the chmod command.
Like this:
chmod u+x test
Then you can test if it works with just the line
./test
The system loads the script and execute the first line program, and feeds the rest of the script to the interpreter in this case it's bash.
Why you have to add "./", that is just for telling the command interpreter that you trying to execute some file from your current directory.
You don't use the extension of the file to tell the system what interpreter to use. (like in Windows)
You just add the shebang line. And add the executing bit to the file.
You don't need to tell what interpreter to use when calling the script in the command line. You have provided that in the script.
And the filename can be what ever you like.
Unix don't use the filename extension of the file to determine what to do.
-
@Patrickg
Thank you for the extensive explanation.
@Paul_123
In the end your suggestion to install py3.8serial.py worked for me.
My only, small, regret is that I have been unable when attaching headless, to "peek" into the still running script to monitor the progress. I have experimented with suggestions to use screen in the following 2 ways:
screen -S dummy -d -m python3test.py exec $SHELL
OR
screen -dmS dummy
screen -S dummy -p 0 -X stuff "python3 test.py\n"
These had some limited effect when included in the .profile method but none at all when using the bootlocal.sh method
-
Your best bet is to write to a log file. Then just use tail to monitor.
tail -f logfile
If you don't need to keep the data, just write to the /tmp directory, but make sure you track the file size to make sure you don't fill up memory. You can use python's logging functions to rotate the logs to keep the size under control.
-
Your best bet is to write to a log file. Then just use tail to monitor.
tail -f logfile
If you don't need to keep the data, just write to the /tmp directory, but make sure you track the file size to make sure you don't fill up memory. You can use python's logging functions to rotate the logs to keep the size under control.
Does that log file need to have a special format?
The reason I ask is because my script already sends collected data in text format to a .txt file.
my test script opens up a text file and writes some text followed by CR every second until the loop ends after 100.
When I triedtail -f test.txt
nothing happened until my script reached the end, at which point "tail" returned all the 100 lines in the file and waited there.
-
Logging is just a text file. It’s likely because you are not flushing buffers writing to disk. It’s easier to just use the built in logging
-
You can run linux sync command to flush.
-
Hi xuraax
... my test script opens up a text file and writes some text followed by CR every second until the loop ends after 100. ...
After each write, do something like the following:
open status.tmp
write the same text to status.tmp
close status.tmp
os.system('sync')
os.system('mv -f status.tmp status.txt')Now anytime you want to see the current status, just:
cat status.txtYou won't have to deal with a file that keeps growing. No risk of reading a partially updated file.
-
After each write, do something like the following:
......
......
You won't have to deal with a file that keeps growing. No risk of reading a partially updated file.
Thank you very much for this very interesting proposal. At this stage I think I will just forgo this requirement and concentrate on the main job of the script.
I take this opportunity to thank all who have contributed with suggestions.