Tiny Core Linux

Tiny Core Base => Raspberry Pi => Topic started by: aug on July 16, 2017, 08:31:31 AM

Title: piCore 7 alex iot python
Post by: aug on July 16, 2017, 08:31:31 AM
I try to play with alex and some python scripts but it seems that not the full python package in core7 is installed. how can i install it? Which package?
Title: Re: piCore 7 alex iot python
Post by: Juanito on July 17, 2017, 09:14:14 AM
Code: [Select]
$ tce-load -wil python
Title: Re: piCore 7 alex iot python
Post by: aug on July 17, 2017, 11:10:28 AM
Code: [Select]
$ tce-load -wil python
And this loads all???
Title: Re: piCore 7 alex iot python
Post by: Juanito on July 17, 2017, 11:15:00 AM
Try it and see  :)
Title: Re: piCore 7 alex iot python
Post by: syang on July 23, 2017, 03:12:18 PM
What I did is install pip and then all the required packages then build a customized tcz from python site-packages folder, then load the tcz at boot up.
Title: Re: piCore 7 alex iot python
Post by: njs on September 28, 2017, 06:50:11 AM
How do I install pip?  I don't see a pip package in http://tinycorelinux.net/9.x/armv6/tcz/  Is it part of python.tcz? 

And once I have pip installed, will I then be able to use pip to install the necessary python modules (e.g. picamera, PIL, smbus, astral) without needing a special tcz for each python module?

thanks.
Title: Re: piCore 7 alex iot python
Post by: Juanito on September 28, 2017, 09:20:42 AM
It looks like (I didn't test) pip is included in the python3.x extensions, but not the python(2.x) extensions.

If you need to use python-2.x, you could download and compile the pip source.

Once pip (and maybe python-setuptools) is installed, you can download and install any python module you like, but it will not persist after re-boot unless you add a lot of files to your backup.
Title: Re: piCore 7 alex iot python
Post by: njs on September 28, 2017, 03:34:40 PM
Great. I would actually prefer python3.  I will need the modules to persist, but as far as adding the files to my backup, is that assuming Cloud or Copy mode?  Can't I avoid this with Mount mode?  The piZero doesn't have a lot of RAM and I don't think I ant all this stuff loaded into RAM disk...

Thanks. 
Title: Re: piCore 7 alex iot python
Post by: Juanito on September 29, 2017, 01:11:05 AM
If you want to avoid using ram you'll have to make extensions out of the modules
Title: Re: piCore 7 alex iot python
Post by: njs on September 29, 2017, 06:00:30 AM
okay, but that's not sufficient, right.  modules are still loading to ramdisk by default, correct?  How do I change that? Mount mode?

thanks.
Title: Re: piCore 7 alex iot python
Post by: Misalf on September 29, 2017, 07:07:23 AM
The backup file is copied to RAM but extensions are mounted by default.
In cloud mode, extensions are downloaded to RAM and then mounted (they are gone after reboot).
If you have persistent storage for extensions, mounted extensions don't consume RAM except for several KB for the mount point.
Title: Re: piCore 7 alex iot python
Post by: Greg Erskine on September 29, 2017, 03:03:15 PM
Can't I avoid this with Mount mode?  The piZero doesn't have a lot of RAM and I don't think I ant all this stuff loaded into RAM disk...

Why use piCore if you don't want an small embedded Linux in RAM?
Title: Re: piCore 7 alex iot python
Post by: Greg Erskine on September 29, 2017, 03:06:19 PM
The backup file is copied to RAM but extensions are mounted by default.
In cloud mode, extensions are downloaded to RAM and then mounted (they are gone after reboot).
If you have persistent storage for extensions, mounted extensions don't consume RAM except for several KB for the mount point.

Aren't extensions copied from persistent storage to /tmp (RAM) then mounted?
Title: Re: piCore 7 alex iot python
Post by: Rich on September 29, 2017, 05:16:02 PM
Hi Greg Erskine
Extensions are loop mounted in  /tmp/tceloop/  and then linked in to the file system, usually  /usr/local/  which is located in RAM.
Title: Re: piCore 7 alex iot python
Post by: njs on September 29, 2017, 07:10:26 PM
The backup file is copied to RAM but extensions are mounted by default.
In cloud mode, extensions are downloaded to RAM and then mounted (they are gone after reboot).

I thought I read that cloud mode was the default.  How do I know what mode I am in?

If you have persistent storage for extensions, mounted extensions don't consume RAM except for several KB for the mount point.

That sounds like what I want.

Why use piCore if you don't want an small embedded Linux in RAM?

I DO want small, tightly-wound, highly-reliable, embedded Linux.  And in particular, I want it to boot as fast as possible.  I don't really care if the entire OS is in RAM except that: 1) copying the entire OS to RAM would undoubtably slow down the boot process and 2) it will waste RAM, which is scare on the PiZero to begin with.

BTW, I'll also need to store my app data (photos) directly to SD.  Any reason I should create a 3rd partition vs. just creating a data directory on /mnt/mmc0blk0p2?
Title: Re: piCore 7 alex iot python
Post by: Rich on September 29, 2017, 07:21:35 PM
Hi njs
If you have your extensions in  /mnt/mmc0blk0p2/tce/optional  then you have persistent storage.

Quote
Any reason I should create a 3rd partition vs. just creating a data directory on /mnt/mmc0blk0p2?
No need to create a third partition.
Title: Re: piCore 7 alex iot python
Post by: njs on September 29, 2017, 08:49:04 PM
Okay, thanks.  So now its a "minor detail" of creating TC extensions for each python package I need. 

Unfortunately after reading Creating an Extension on the Wiki this seems like a daunting process!  The main problem is when I install a python package using pip I'm not sure what it is all actually being installed or where or what the dependencies may be.  Some packages are probably just pure python stored in a single directory under /usr/local/lib/python3.6/site-packages but others seem to be a lot more complex (e.g. Pillow which involved a lengthy compiling process that presumably generated various unknown binaries stored in an various unknown locations).  With a little Googling I found these, which seem applicable but still leave me confused about what i should actually do.   :-\

https://stackoverflow.com/questions/2915471/install-a-python-package-into-a-different-directory-using-pip
https://docs.python.org/3/install/index.html#custom-installation

Any advice would be appreciated!!!

     

Title: Re: piCore 7 alex iot python
Post by: Misalf on September 30, 2017, 08:00:23 AM
Cloud mode is the default in so far as extensions are downloaded to, and mounted from temporary memory in RAM if there was neither a  tce  directory found during boot nor specified via  tce=  boot code.
I personally don't use that mode and I could see a use case only for pxe booting devices that don't have any storage at all.

You can use any data directory on any partition no problem. I use to create symlinks in $HOME for directories that contain a lot of data so only the symlink is in the backup but not the data itself.
Title: Re: piCore 7 alex iot python
Post by: njs on September 30, 2017, 10:00:39 AM
You can use any data directory on any partition no problem. I use to create symlinks in $HOME for directories that contain a lot of data so only the symlink is in the backup but not the data itself.

Can I just use the following in config.txt?
Code: [Select]
home=  /mnt/mmc0blk0p2

If anyone is willing to give some step by step instructions for making a private extension from a python package (Pillow would probably be the best non-trivial example) I'd greatly appreciate it!!!
Title: Re: piCore 7 alex iot python
Post by: Juanito on September 30, 2017, 11:22:03 AM
After downloading the source code from:

https://pypi.python.org

..you can see the general idea, for example, from here:

http://www.tinycorelinux.net/7.x/x86_64/tcz/src/python-cryptography/compile_python-cryptography

..together with the extension making section of the wiki, this should be enough
Title: Re: piCore 7 alex iot python
Post by: njs on September 30, 2017, 03:16:57 PM
Thanks.  So in the example:
Code: [Select]
tce-load -i python-dev python-setuptools python-idna pyasn1 python-cffi python-cparser openssl-dev

cd cryptography-1.4

python setup.py build
sudo python setup.py install --old-and-unmanageable

is the "--old-and-unmanageable" switch the key?  What does this do, force all build output (including scripts, binaries, etc.) into a known location from which I can make the extension?
Title: Re: piCore 7 alex iot python
Post by: Juanito on October 01, 2017, 10:12:17 AM
If you'd like a list of files to package, try this:
Code: [Select]
$ python setup.py build
$ touch mymarker
$ sudo python setup.py install --old-and-unmanageable
$ sudo find / -not -type 'd' -cnewer mymarker | grep -v "\/proc\/" | grep -v "^\/sys\/" | tee files
$ vi files [remove files obviously not required]
Title: Re: piCore 7 alex iot python
Post by: Juanito on October 06, 2017, 08:21:21 AM
I compiled pillow and its dep olefile from source on corepure64 to test, you can find the build instructions here:

http://tinycorelinux.net/8.x/x86_64/tcz/src/python-olefile
http://tinycorelinux.net/8.x/x86_64/tcz/src/python-pillow
Title: Re: piCore 7 alex iot python
Post by: njs on October 09, 2017, 08:51:08 PM
Thanks for the tips / examples.  Just getting back to this and I hope to try it soon...
Title: Re: piCore 7 alex iot python
Post by: njs on October 12, 2017, 10:13:11 PM
I was going to try building Pillow per your instructions but I cannot seem to load the python-setuptools.tcz extension via "tce-load -iw":
Code: [Select]
root@box:/tmp/Pillow/Pillow-4.3.0#  tce-load -iw compiletc python-dev python-setuptools python-olefile tiff-dev freetype-dev
compiletc is already installed!
python-dev is already installed!
Downloading: python-setuptools.tcz
freetype-dev.tcz.dep OK
libpng-dev.tcz.dep OK
Connecting to repo.tinycorelinux.net (89.22.99.37:80)
wget: server returned error: HTTP/1.1 404 Not Found
md5sum: python-setuptools.tcz.md5.txt: No such file or directory
Error on python-setuptools.tcz

Can I manually download it from somewhere?

Thanks
Title: Re: piCore 7 alex iot python
Post by: Juanito on October 12, 2017, 10:59:34 PM
Note that the instructions referenced were for CorePure64 and thus use CorePure64 extensions.

I seem to recall that you wanted to use python-3.x, whereas the instructions use python-2.x. You will also need to compile (python-)olefile before (python-)pillow.

I'm not certain, but it looks like setuptools is included in the armv7 python3.6 extension and thus would not need to be loaded as a separate extension.
Title: Re: piCore 7 alex iot python
Post by: njs on October 13, 2017, 05:47:08 AM
I'm using pi Zero (Arm 6) so per http://tinycorelinux.net/9.x/armv6/tcz/ the extension is apparently named setuptools.tcz not  python-setuptools.tcz

I'm not sure what olefile is or where to find it for Arm 6 but at least I can start the build but it fails due to missing headers for jpeg lib...

Code: [Select]
root@box:/tmp/Pillow/Pillow-4.3.0# python setup.py build_ext --enable-zlib --enable-jpeg --enable-tiff --enable-freetype
Single threaded build, not installing mp_compile:1 processes
running build_ext

The headers or library files could not be found for jpeg,
a required dependency when compiling Pillow from source.

Please see the install instructions at:
   https://pillow.readthedocs.io/en/latest/installation.html

Traceback (most recent call last):
  File "setup.py", line 799, in <module>
    raise RequiredDependencyException(msg)
__main__.RequiredDependencyException:

The headers or library files could not be found for jpeg,
a required dependency when compiling Pillow from source.

Please see the install instructions at:
   https://pillow.readthedocs.io/en/latest/installation.html
Title: Re: piCore 7 alex iot python
Post by: Juanito on October 13, 2017, 06:07:12 AM
You can find olefile here:

https://github.com/decalage2/olefile/releases/download/v0.44/olefile-0.44.tar.gz

..pillow will not compile without it.

When you get a message analgous to "The headers or library files could not be found for blah", it means you need to install the extension blah-dev.

Note that if you install libtiff-dev, it will also install the necessary jpeg extensions.
Title: Re: piCore 7 alex iot python
Post by: njs on October 13, 2017, 06:39:59 AM
Thanks for the continued advice.  Loading the libtiff-dev extension as you suggested fixed the JPEG dependency but then it griped about missing freetype but since I don't think I need or want freetype (whatever that is) for my app, I just built as follows:

Code: [Select]
root@box:/tmp/Pillow/Pillow-4.3.0# python setup.py build_ext  --enable-jpeg
...
--------------------------------------------------------------------
PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 4.3.0
platform     linux2 2.7.13 (default, Dec 23 2016, 18:28:09)
             [GCC 6.2.0]
--------------------------------------------------------------------
--- JPEG support available
*** OPENJPEG (JPEG2000) support not available
--- ZLIB (PNG/ZIP) support available
*** LIBIMAGEQUANT support not available
--- LIBTIFF support available
*** FREETYPE2 support not available
*** RAQM support not available
*** LITTLECMS2 support not available
*** WEBP support not available
*** WEBPMUX support not available
--------------------------------------------------------------------

And I actually did not need olefile.  According to https://pypi.python.org/pypi/olefile/0.40:  "Since 2014 olefile/OleFileIO_PL has been integrated into Pillow, the friendly fork of PIL. olefile will continue to be improved as a separate project, and new versions will be merged into Pillow regularly."

Now to see if I build it for python3... AND make a TC extension... AND then repeat the process for a bunch of other modules (e.g. picamera, pydrive, yagmail, wifi, astral, ipify).  It would be REALLY nice to be able to use pip installs and make make extension from those rather than trying to build each of these from source...
Title: Re: piCore 7 alex iot python
Post by: Juanito on October 13, 2017, 06:52:35 AM
What you may have missed is that during compiling pillow, setuptools downloaded and installed olefile so that you did not need to compile it manually.

The problem with this approach is that olefile will probably not be optimised for piCore.

If you use the python3.6 extension you can use pip/setuptools without needing any other extensions because they are built-in to the extension.

..However if you use pip/setuptools to install a bunch of python packages, you will soon run into problems making them persistant within the RPi ram and speed constraints.
Title: Re: piCore 7 alex iot python
Post by: njs on October 13, 2017, 04:17:02 PM
However if you use pip/setuptools to install a bunch of python packages, you will soon run into problems making them persistant within the RPi ram and speed constraints.

Can you elaborate on what you mean by that and how building from source is different?
Title: Re: piCore 7 alex iot python
Post by: njs on October 13, 2017, 07:43:53 PM
okay here's what I tried based on your examples...

Code: [Select]
tce-load -i compiletc python-dev setuptools python-olefile tiff-dev freetype-dev libtiff-dev
cd /tmp
tar -xf Pillow-4.3.0.tar.gz
cd Pillow-4.3.0/
python setup.py build_ext --enable-jpeg
touch mymarker
sudo python setup.py build_ext --enable-jpeg install
sudo find / -not -type 'd' -cnewer mymarker | grep -v "\/proc\/" | grep -v "^\/sys\/" | tee python2.7-Pillow-files.lst

The resulting file list is attached.

However when I tested the install there seems to be a problem  :(...

Code: [Select]
python
Python 2.7.13 (default, Dec 23 2016, 18:28:09)
[GCC 6.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from PIL import Image
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "PIL/Image.py", line 56, in <module>
    from . import _imaging as core
ImportError: cannot import name _imaging

Any suggestions?
Title: Re: piCore 7 alex iot python
Post by: Juanito on October 14, 2017, 12:59:59 AM
Here are the files from your list that you need:
Code: [Select]
/usr/local/bin/pilprint.py
/usr/local/bin/painter.py
/usr/local/bin/thresholder.py
/usr/local/bin/player.py
/usr/local/bin/enhancer.py
/usr/local/bin/explode.py
/usr/local/bin/pilconvert.py
/usr/local/bin/pilfont.py
/usr/local/bin/pilfile.py
/usr/local/bin/gifmaker.py
/usr/local/bin/viewer.py
/usr/local/bin/pildriver.py
/usr/local/lib/python2.7/site-packages/olefile-0.44-py2.7.egg
/usr/local/lib/python2.7/site-packages/Pillow-4.3.0-py2.7-linux-armv6l.egg

The *egg files are zip archives, which makes things difficult to figure out and tends to waste space as they often contain unneeded *.pyc files, unstripped libs, text files, etc.

With reference to:

http://tinycorelinux.net/8.x/x86_64/tcz/python-pillow.tcz.list

..Pillow-4.3.0-py2.7-linux-armv6l.egg should contain _imaging.so, the lack of which causes the "cannot import name _imaging" error. You can check for this using the unzip command.
Title: Re: piCore 7 alex iot python
Post by: Juanito on October 14, 2017, 01:12:06 AM
Can you elaborate on what you mean by that and how building from source is different?

I presumed (perhaps wrongly) that you were planning to add the pip/setuptools installed packages to your backup, which would use a lot of ram and make booting slower. You could however make personal extensions out of the packages (as long as you could figure out which files were needed).

Another problem with pip/setuptools installed packages is that they always seem to use *egg files, which usually contain unneeded bloat. If you compile from source, it often does not result in *egg files (as is the case with olefile) or, can be prevented with "--old-and-unmanageable". This being said, with pillow, I had to manually unzip the *egg file to strip the libs and remove bloat.
Title: Re: piCore 7 alex iot python
Post by: njs on October 14, 2017, 07:24:55 AM
I presumed (perhaps wrongly) that you were planning to add the pip/setuptools installed packages to your backup, which would use a lot of ram and make booting slower. You could however make personal extensions out of the packages (as long as you could figure out which files were needed).

Another problem with pip/setuptools installed packages is that they always seem to use *egg files, which usually contain unneeded bloat. If you compile from source, it often does not result in *egg files (as is the case with olefile) or, can be prevented with "--old-and-unmanageable". This being said, with pillow, I had to manually unzip the *egg file to strip the libs and remove bloat.

Yes, I'm trying to make extensions for each package to be loop mounted as I am trying to minimize boot time.

I guess I don't understand why "sudo python setup.py build_ext --enable-jpeg
Code: [Select]
install" does a half job leaving stuff in egg files not available for import by my application.  The whole purpose of this command would seem to be to to create a functioning installation, although I've always used pip (or a few times apt get blah)...
Title: Re: piCore 7 alex iot python
Post by: Juanito on October 14, 2017, 11:35:47 PM
Normally python packages installed with pip/setuptools that contain *egg files do work.

Did you make your test directly after installing pillow or after having made an extension of it?

Edit: Using an RPi3 and piCore 9.x armv7, I got the following:
Code: [Select]
$ tce-load -i compiletc python3.6-dev tk-dev libtiff-dev freetype-dev
$ sudo pip3.6 install pillow
...
  Running setup.py install for pillow ... done
Successfully installed pillow-4.3.0

$ ls  /usr/local/lib/python3.6/site-packages
OleFileIO_PL.py              README.txt                   olefile-0.44-py3.6.egg-info/ setuptools/
PIL/                         __pycache__/                 pip/                         setuptools-28.8.0.dist-info/
Pillow-4.3.0-py3.6.egg-info/ easy_install.py              pip-9.0.1.dist-info/
README                       olefile/                     pkg_resources/

$ viewer.py
Syntax: python viewer.py imagefile

..so pip did not install the *egg files, but installed the packages in the old way instead.

With reference to:

http://tinycorelinux.net/8.x/x86_64/tcz/python-pillow.tcz.list

..what I don't see are files analagous to /usr/local/lib/python2.7/site-packages/Pillow-4.2.1-py2.7-linux-x86_64.egg-info/scripts/*, but maybe that is due to using python-2.7.x or an earlier version of pillow.