piCore 7 alex iot python

[Rich]

Hi njs
If you have your extensions in  /mnt/mmc0blk0p2/tce/optional  then you have persistent storage.

Any reason I should create a 3rd partition vs. just creating a data directory on /mnt/mmc0blk0p2?

No need to create a third partition.

[njs]

Okay, thanks.  So now its a "minor detail" of creating TC extensions for each python package I need. 

Unfortunately after reading Creating an Extension on the Wiki this seems like a daunting process!  The main problem is when I install a python package using pip I'm not sure what it is all actually being installed or where or what the dependencies may be.  Some packages are probably just pure python stored in a single directory under /usr/local/lib/python3.6/site-packages but others seem to be a lot more complex (e.g. Pillow which involved a lengthy compiling process that presumably generated various unknown binaries stored in an various unknown locations).  With a little Googling I found these, which seem applicable but still leave me confused about what i should actually do.   

https://stackoverflow.com/questions/2915471/install-a-python-package-into-a-different-directory-using-pip
https://docs.python.org/3/install/index.html#custom-installation

Any advice would be appreciated!!!

     

[Misalf]

Cloud mode is the default in so far as extensions are downloaded to, and mounted from temporary memory in RAM if there was neither a  tce  directory found during boot nor specified via  tce=  boot code.
I personally don't use that mode and I could see a use case only for pxe booting devices that don't have any storage at all.

You can use any data directory on any partition no problem. I use to create symlinks in $HOME for directories that contain a lot of data so only the symlink is in the backup but not the data itself.

[njs]

You can use any data directory on any partition no problem. I use to create symlinks in $HOME for directories that contain a lot of data so only the symlink is in the backup but not the data itself.

Can I just use the following in config.txt?

Code: [Select]

home=  /mnt/mmc0blk0p2

If anyone is willing to give some step by step instructions for making a private extension from a python package (Pillow would probably be the best non-trivial example) I'd greatly appreciate it!!!

[Juanito]

After downloading the source code from:

https://pypi.python.org

..you can see the general idea, for example, from here:

http://www.tinycorelinux.net/7.x/x86_64/tcz/src/python-cryptography/compile_python-cryptography

..together with the extension making section of the wiki, this should be enough

[njs]

Thanks.  So in the example:

Code: [Select]

tce-load -i python-dev python-setuptools python-idna pyasn1 python-cffi python-cparser openssl-dev

cd cryptography-1.4

python setup.py build
sudo python setup.py install --old-and-unmanageable
is the "--old-and-unmanageable" switch the key?  What does this do, force all build output (including scripts, binaries, etc.) into a known location from which I can make the extension?

[Juanito]

If you'd like a list of files to package, try this:

Code: [Select]

$ python setup.py build
$ touch mymarker
$ sudo python setup.py install --old-and-unmanageable
$ sudo find / -not -type 'd' -cnewer mymarker | grep -v "\/proc\/" | grep -v "^\/sys\/" | tee files
$ vi files [remove files obviously not required]


[Juanito]

I compiled pillow and its dep olefile from source on corepure64 to test, you can find the build instructions here:

http://tinycorelinux.net/8.x/x86_64/tcz/src/python-olefile
http://tinycorelinux.net/8.x/x86_64/tcz/src/python-pillow

[njs]

Thanks for the tips / examples.  Just getting back to this and I hope to try it soon...

[njs]

I was going to try building Pillow per your instructions but I cannot seem to load the python-setuptools.tcz extension via "tce-load -iw":

Code: [Select]

root@box:/tmp/Pillow/Pillow-4.3.0#  tce-load -iw compiletc python-dev python-setuptools python-olefile tiff-dev freetype-dev
compiletc is already installed!
python-dev is already installed!
Downloading: python-setuptools.tcz
freetype-dev.tcz.dep OK
libpng-dev.tcz.dep OK
Connecting to repo.tinycorelinux.net (89.22.99.37:80)
wget: server returned error: HTTP/1.1 404 Not Found
md5sum: python-setuptools.tcz.md5.txt: No such file or directory
Error on python-setuptools.tcz

Can I manually download it from somewhere?

Thanks

[Juanito]

Note that the instructions referenced were for CorePure64 and thus use CorePure64 extensions.

I seem to recall that you wanted to use python-3.x, whereas the instructions use python-2.x. You will also need to compile (python-)olefile before (python-)pillow.

I'm not certain, but it looks like setuptools is included in the armv7 python3.6 extension and thus would not need to be loaded as a separate extension.

[njs]

I'm using pi Zero (Arm 6) so per http://tinycorelinux.net/9.x/armv6/tcz/ the extension is apparently named setuptools.tcz not  python-setuptools.tcz

I'm not sure what olefile is or where to find it for Arm 6 but at least I can start the build but it fails due to missing headers for jpeg lib...

Code: [Select]

root@box:/tmp/Pillow/Pillow-4.3.0# python setup.py build_ext --enable-zlib --enable-jpeg --enable-tiff --enable-freetype
Single threaded build, not installing mp_compile:1 processes
running build_ext

The headers or library files could not be found for jpeg,
a required dependency when compiling Pillow from source.

Please see the install instructions at:
   https://pillow.readthedocs.io/en/latest/installation.html

Traceback (most recent call last):
  File "setup.py", line 799, in <module>
    raise RequiredDependencyException(msg)
__main__.RequiredDependencyException:

The headers or library files could not be found for jpeg,
a required dependency when compiling Pillow from source.

Please see the install instructions at:
   https://pillow.readthedocs.io/en/latest/installation.html


[Juanito]

You can find olefile here:

https://github.com/decalage2/olefile/releases/download/v0.44/olefile-0.44.tar.gz

..pillow will not compile without it.

When you get a message analgous to "The headers or library files could not be found for blah", it means you need to install the extension blah-dev.

Note that if you install libtiff-dev, it will also install the necessary jpeg extensions.

[njs]

Thanks for the continued advice.  Loading the libtiff-dev extension as you suggested fixed the JPEG dependency but then it griped about missing freetype but since I don't think I need or want freetype (whatever that is) for my app, I just built as follows:

Code: [Select]

root@box:/tmp/Pillow/Pillow-4.3.0# python setup.py build_ext  --enable-jpeg
...
--------------------------------------------------------------------
PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 4.3.0
platform     linux2 2.7.13 (default, Dec 23 2016, 18:28:09)
             [GCC 6.2.0]
--------------------------------------------------------------------
--- JPEG support available
*** OPENJPEG (JPEG2000) support not available
--- ZLIB (PNG/ZIP) support available
*** LIBIMAGEQUANT support not available
--- LIBTIFF support available
*** FREETYPE2 support not available
*** RAQM support not available
*** LITTLECMS2 support not available
*** WEBP support not available
*** WEBPMUX support not available
--------------------------------------------------------------------

And I actually did not need olefile.  According to https://pypi.python.org/pypi/olefile/0.40:  "Since 2014 olefile/OleFileIO_PL has been integrated into Pillow, the friendly fork of PIL. olefile will continue to be improved as a separate project, and new versions will be merged into Pillow regularly."

Now to see if I build it for python3... AND make a TC extension... AND then repeat the process for a bunch of other modules (e.g. picamera, pydrive, yagmail, wifi, astral, ipify).  It would be REALLY nice to be able to use pip installs and make make extension from those rather than trying to build each of these from source...

[Juanito]

What you may have missed is that during compiling pillow, setuptools downloaded and installed olefile so that you did not need to compile it manually.

The problem with this approach is that olefile will probably not be optimised for piCore.

If you use the python3.6 extension you can use pip/setuptools without needing any other extensions because they are built-in to the extension.

..However if you use pip/setuptools to install a bunch of python packages, you will soon run into problems making them persistant within the RPi ram and speed constraints.