my adventures with installing python3.9 on TC15

[Stefann]

Hi,
This weekend I succeeded in getting python3.9 on TC15 TinyCore.
I would like to share my experiences because it was not a completely smooth ride.
I will skip all detours I made and just get to how I finally succeeded.
Note that I run my system frugal: with tc home directory in ram.

Code: [Select]

got python3.9 on by getting python3.9-setuptools.tcz downloaded and "on demand"
check:
tc@hp510:~$ python3 --version
Python 3.6.15

tc@hp510:~$ python3.9 --version
Python 3.9.18note: not sure I would need the setup tools. I installed them just to be sure.

strangly enough the 3.9 not only installs 3.9 but also some 3.6 residuals:

Code: [Select]

tc@hp510:/usr/local/bin$ ls -al python3*
lrwxrwxrwx    1 root     root             9 Mar 22 12:47 python3 -> python3.6
lrwxrwxrwx    1 root     root            16 Mar 22 12:47 python3-config -> python3.6-config
-rwxr-xr-x    2 root     root         13760 Dec  7  2023 python3.6
lrwxrwxrwx    1 root     root            17 Mar 22 12:47 python3.6-config -> python3.6m-config
-rwxr-xr-x    2 root     root         13760 Dec  7  2023 python3.6m
-rwxr-xr-x    1 root     root          3061 Dec  7  2023 python3.6m-config
-rwxr-xr-x    1 root     root         13728 Jan 27  2024 python3.9
-rwxr-xr-x    1 root     root          3034 Jan 27  2024 python3.9-config
after that I did need some modules that are not offered as tcz.
I tried "pip" but that did not work
I succeeded getting pip by:

Code: [Select]

python3.9 -m ensurepip --default-pip
python3.9 -m pip install --upgrade pip

This did NOT install pip in /usr/local but in my tc home directory:
tc@hp510:~/.local/lib/python3.9/site-packages
After that I successfully installed some modules:

Code: [Select]

python3.9 -m pip install python_dateutil==2.8.2
python3.9 -m pip install beautifulsoup4==4.11.1
python3.9 -m pip install pytz==2022.2.1
python3.9 -m pip install requests==2.28.1
python3.9 -m pip install paho_mqtt==1.6.1

all got installed in:
~/.local/lib/python3.9/site-packages
As I run with home in RAM (so, not persistent) this would unnecessarily increase the backup for home
I solved that by following "Chapter 14. Creating a personal (data) extension" from the core book to create a data extension for the content of ~/.local/lib/python3.9/site-packages and adding home/tc/.local/lib/python3.9/site-packages to .xfiletool.lst.

So...
This all works...
But it's not super clean

What I would hope (actually not for me as I'm fine now, but probably for others)
- cleanup the python3.9 tcz package such that no 3.6 stuff is created
- update pip in python3.9 tcp package such that it works out of the box, and not installs on ~/.local

Next to that..
- creating a data extension for ~/.local/lib/python3.9/site-packages is a bit ugly.
- it would be better to create tcz program extensions for the modules

I'm afraid all of these are at this moment beyond my capabilities.

Actually... I feel that "if I would have installed the modules under root", they would have landed in /usr/local/lib/python3.9/site-packages from which either a data-extension or program extension would have been a much cleaner approach.
However.... running pip under root gives a fat warning that it may corrupt python. So... I did not dare to do.

All in all:
I'm totally happy now.
Fixing all of the above is really beyond my capabilities so I'm afraid I cannot assist to fix.
So with that said: Either consider this an instruction guide towards pyhton3.9 install OR a suggestion for fix.

[Juanito]

The tree file for python3.9-setuptools in the tc-15.x x86 repo does not contain any references to python3.6.

This means that some other extension you are loading is pulling in python3.6.

[Stefann]

Mhh strange…
I thought I checked that.
But as I said… it was a full Saturday of hobbying until I had it working.
I tried to make it a “clean story” but I may have done something wrong.

I will check again tomorrow and report back.

[Stefann]

OK, you are right.
Don't understand it because I thought I had checked this !!!!!!
But anyways..

Code: [Select]

run visual apps browser
click "dependancies"
click "python3.6.tcz"
click "required by"
- libxml2-dev.tcz.      >> required by php-8.3-dev.tcz
- libxml2-python.tcz. >> required by php-8.3-dev.tcz
- php-8.3-dev.tcz.     >> yes I have installed that
- python3.6-pip.tcz
- python3.6-setuptools.tcz

so, yes I have php-8.3-dev.tcz installed.
I use php, but not sure I also need the -dev module. I merely loaded it "just to be sure".

Next step

Code: [Select]

removed php-8.3-dev.tcz from on demand list.
reboot with backup

tc@hp510:/usr/local/bin$ ls py* -al
lrwxrwxrwx    1 root     root             8 Mar 24 17:55 pydoc3 -> pydoc3.9
-rwxr-xr-x    1 root     root            84 Jan 27  2024 pydoc3.9
lrwxrwxrwx    1 root     root             9 Mar 24 17:55 python3 -> python3.9
lrwxrwxrwx    1 root     root            16 Mar 24 17:55 python3-config -> python3.9-config
-rwxr-xr-x    1 root     root         13728 Jan 27  2024 python3.9
-rwxr-xr-x    1 root     root          3034 Jan 27  2024 python3.9-config
so.. yesss... python3.6 disappeared!

I only use php inside apache webserver.
I checked my application
webpage still seems to function.
So... keep it off for now.

That's solves 1 thing!
Happy with it. don't like to have 2 versions of python on my system, so thanks Juanito

now... python3.9 apparently is also installed even though I did not install it.

Code: [Select]

Checked "required by"
- indeed many "required by's"
- especially "libusb-compat-dev.tcz" >>> yes... and I sure need that one
So now it remains a bit unclear why I did need to upgrade pip >> I'm not going to look into that, I'm happy it works
And a bit ugly to have pip & extra modules under ~/.local >> I will live with that.

[Juanito]

There are 15+ python3.9 extensions in the tc-15.x x86 repo.

If you don't see the one you need, it's not difficult to make a python extension.

Taking as an example python certifi:

Code: [Select]

tce-load -i compiletc python3.9-dev squashfs-tools

wget https://files.pythonhosted.org/packages/71/da/e94e26401b62acd6d91df2b52954aceb7f561743aa5ccc32152886c76c96/certifi-2024.2.2.tar.gz
tar xf certifi-2024.2.2.tar.gz

cd certifi-2024.2.2

python3.9 setup.py build
sudo python3.9 setup.py install --root=/tmp/pkg
Then delete all of the __pycache__ subdirectories under /tmp/pkg/usr/local/lib/python3.9/site-packages

Finally make an extension:

Code: [Select]

cd /tmp
sudo mksquashfs pkg/ python3.9-certifi.tcz

[Stefann]

Thanks,
Yes! There are extensions in TC15 But unfortunately not the ones I needed.

Thanks for the example, seems doable, and I really needed that as a guidance otherwise I would never have gotten there.

For your information:
- I’m not familiar with python at all, the only reason I want python is because someone wrote a python coded interface to read cloud-data from my EV-car and I wanted to use that interface
- which got me on a very steep learning curve towards python.
- installing python via tcz mechanism was relatively straightforward, but extending with non tcz modules was an enormous trial & error excersize
- after lot of trying and googling I got where I currently am as described in 1st post. With some spots.. but working. Happy!
- but… as said… I’d don’t like the spots.

So…
Thanks for the example… really helps…
- so far I used pip for extending. That works relatively smooth. But… you cannot run pip as root as it may messup versions. And as a result of that you get a rather ugly install in ~/.local.
- as by your example, proper way to install in /usr/local is by downloading the .gz and installing by setup.py. Thanks for that guidance… it’s totally new for me. It will definitely cost me trial and error to do that but seems doable (And that’s part of the fun).
- creating the .tcz extends after that I should be able to do. I basically already did the same already by creating a data .tcz extension.

I will give it a try!
This will take some days. There is more on my plate.
But I’ll report back how it went.

Thanks!

[Stefann]

Hi,
I got time to try.
Some success and some not:
1/ there are more modules in the apps browser than I was aware of! originally I only looked for "python3.9*", I however realised that modules donot necessarily need to be python3.9 specific. Still however... none of the found modules had the version I was looking for.
No problem... try to install using the guidance of Juanito....

2/ I used the method describe by Juanito, YES! worked for many:
I was able to install:
beautifulsoup4, v4.11.1
pytz, v2022.2.1
requests, v2.28.1
paho_mqtt, v1.6.1
==> these went OK!

However got some problems:
A/ beautifulsoup4 wants "soupsieve"
- I downloaded and tried to build but "it does NOT have a setup.py file" --> I'm stuck

B/ python_dateutil, v2.8.2 wants to have "wheel" > ok installed that, but still not working:

Code: [Select]

tc@hp510:/krubo/work/python/python-dateutil-2.8.2$ python3.9 setup.py build
/usr/local/bin/python3.9: No module named pip.__main__; 'pip' is a package and cannot be directly executed
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/setuptools/installer.py", line 75, in fetch_build_egg
    subprocess.check_call(cmd)
  File "/usr/local/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/python3.9', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpa5_rstqz', '--quiet', 'setuptools_scm']' returned non-zero exit status 1.
It wants to use pip... and actually I doNOT want to use pip because that installs things under ~/.local and that is not what I want.
So... stuck....

any guidance forwards???

[Stefann]

OK,...
soupsieve problem I was able to fix >>> I installed an earlier version that DID have the setup.py file

so... now only python_dateutil, v2.8.2 problem.

It wants to be installed by pip
Which I initially did (1st post of this thread)
But than all got in ~/.local which I did not like
And according to what I found on internet I should not use pip as root to avoid that.

not sure how to move form here

[Juanito]

As a test I installed/updated various missing packages using "sudo pip3 --blah-blah" and then "python3.9 setup.py build" worked for me with python-dateutil (python-dateutil-2.9.0.post0.tar.gz)

[Stefann]

Ok...
I got a little bit further...

Code: [Select]

tc@hp510:/krubo/work/python/python-dateutil-2.8.2$ sudo python3.9 -m ensurepip --default-pip
tc@hp510:/krubo/work/python/python-dateutil-2.8.2$ sudo python3.9 -m pip install --upgrade pip
tc@hp510:/krubo/work/python/python-dateutil-2.8.2$ sudo python3.9 -m pip install --upgrade setuptools
tc@hp510:/krubo/work/python/python-dateutil-2.8.2$ python3.9 setup.py build
tc@hp510:/krubo/work/python/python-dateutil-2.8.2$ sudo python3.9 setup.py install --root=/tmp/pkg

tc@hp510:/krubo/work/python/python-dateutil-2.8.2$ cd /tmp
removed __pycache__
tc@hp510:/tmp$sudo mksquashfs pkg/ python3.9-dateutil.tcz
cp python3.9-dateutil.tcz /mnt/sda1/tce/optional
yess worked

After that I did reboot WITHOUT backup >> to be sure that I did not got any damage form the "sudo python3.9 -m pip" commands

Than I tried to run my application:
> complaining that it did need "six module"
> also made tcz for that

Again tried to run my application:
> complaining that it did need "urllib3 module"
> downloaded and tried to built
> damn... no setup.py script
stuck again.

Anyway... I think I call it a day for today.
t.b.h..... I have the feeling "this is not the way to do it"....
> I'm running into dependancy after dependancy
> the newer models seem to be only properly upgradable by pip.
> In fact "upgrade by pip" actually works and is actually quite simple >>> see 1st post
> but.... "upgrade by pip" ends in ~/.local/lib/python3.9/site_packages folder which is in user space and not OS space which I don't like
> running pip by sudo would solve this BUT has danger that pip upgrades underlying packages that cause broken dependancies elsewhere

So...
I guess the proper way out is to figure out:
- how use pip under sudo
- without danger of breaking things
- and with method to created .tcz from the result

I call it a day for now.
Will pick later if I have some ideas by myself OR from a suggestion here

And very much thank you to Juanito! Little by little you bring me further.

(it of course does not help that I'm an absolute beginner in python and also no big linux hero)

[Stefann]

Well.... I got it working!
And thanks Juanito for pointing me in the right direction.

here is how I fixed things......
1/ closing remark last post: (it of course does not help that I'm an absolute beginner in python and also no big linux hero)
I improved on that by reading a lot about pip.

2/ As I more or less concluded yesterday...
The "python3.9 setup.py build" method is vey cumbersome.
It runs from dependancy into dependancy.
And newer packages donot even have a "setup.py" script anymore.

3/ So I started from scratch again.
Removed all python stuff, started with a clean sheet.
>>> and this is why I really like tiny core... If your system is cluttered after few days of fiddling around with installs it's very easy to start from scratch again.
I decided to go for the pip route....

note: I do the package handling from the GUI as I donot feel super secure with command line with those. That's why you do not see command line commands for the package handling

First... get a recent pip........

Code: [Select]

python3.9 -m ensurepip
cd /tmp
mkdir pkg

python3.9 -m pip install --upgrade pip --root /tmp/pkg

delete all of the __pycache__ subdirectories under /tmp/pkg/usr/local/lib/python3.9/site-packages/pip

sudo mksquashfs pkg/ python3.9-pip.tcz
cp *tcz /mnt/sda1/tce/optional

add python3.9-pip.tcz to onboot.lst

reboot WITHOUT backup >> to avoid that the files in ~/.local are backed up

---
tc@hp510:/tmp$ python3.9 -m pip --version
pip 25.0.1 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)
Huray!!! i got an upgraded pip in the normal location!

Explanation:
The 1st "ensure pip" got me a basic pip.
It got installed in my ~/.local but that does not hurt because by rebooting without backup all that got deleted

The 2nd "--upgrade pip" got installed to /tmp/pkg
From there its a quiet straightforward tcz generation
Note: the upgraded pip "cannot run" form /tmp/pkg because /tmp/pkg is not in the search path
So.. I did need to create the tcz, add to onboot.lst and reboot to get a functional pip
By rebooting without backup "no trash" was pulled in, the 1st "ensure pip" got flushed.

4/ with rebooted system got the modules I need:

Code: [Select]

python3.9 -m pip install python_dateutil==2.8.2 --root /tmp/pkg
python3.9 -m pip install beautifulsoup4==4.11.1 --root /tmp/pkg
python3.9 -m pip install pytz==2022.2.1 --root /tmp/pkg
python3.9 -m pip install requests==2.28.1 --root /tmp/pkg
python3.9 -m pip install paho_mqtt==1.6.1 --root /tmp/pkg

delete all of the __pycache__ subdirectories under /tmp/pkg/usr/local/lib/python3.9/site-packages/*

sudo mksquashfs pkg/ python3.9-goodies2.tcz
cp *tcz /mnt/sda1/tce/optional

add python3.9-goodies2.tcz to onboot.lst

reboot WITHOUT backup >> to avoid that the files in ~/.local are backed up
tested python application >> yess works!!!

Conclusion:
- all are now in standard /usr/local/lib/python3.9/site-packages/* folder >> good!
- I decided to get pip in a dedicated python3.9-pip.tcz because I expect to need pip more often and unrelated to current application
- I decided to do all modules I need in 1 goodies2.tcz for reason that pip made sure version conflicts got resolved. I could have done a tcz per modules. Its ok for me like this

As said, thanks Juanito for pointing me in the right direction