Tiny Core Linux
Tiny Core Extensions => TCE News => TCE 2.x => Topic started by: Jason W on November 27, 2009, 12:04:12 AM
-
Updated:
Title: curl-dev.tcz
Description: Client and library for network transfers dev files.
Version: 7.19.7
Author: Daniel Stenberg
Original-site: http://curl.haxx.se/
Copying-policy: MIT
Size: 217KB
Extension_by: Jason W
Comments: Use this extension along with curl.tczl to compile
apps dependent on curl.
Change-log: 2008/09/10 First version
2008/12/12 Rebuilt for i486, size, cramfs.
Current: 2009/11/26 Updated and built against openssl-0.9.8k.
Title: curl.tczl
Description: Client and library for network transfers.
Version: 7.19.7
Author: Daniel Stenberg
Original-site: http://curl.haxx.se/
Copying-policy: MIT
Size: 200K
Extension_by: Jason W
Comments: ---------
Change-log: 2008/09/10 First version
2008/12/12 Rebuilt for i486, size, cramfs.
Current: 2009/11/26 Updated and rebuilt against openssl-0.9.8k.
-
the dep file still says openssl-0.9.8h.tcz - should it be openssl-0.9.8k?
-
Fixed.
-
Hmm, I don't think that this was a successful change:
tc@box:~$ curl http://example.com
curl: error while loading shared libraries: libssl.so.0.9.8: cannot open shared object file: No such file or directory
Tested on a "plain" TC 2.9rc4 system.
-
Oops, I did build it against openssl-0.9.8k, but I forgot to make a wrapper for it as it does not automatically find the libs. Some openssl dependent apps do and some don't. I will make one when I get home tonight.
-
Better yet, I see how to rebuild it with a hard coded openssl path that wont need a wrapper, by adding:
env LDFLAGS=-R/usr/local/openssl-0.9.8k/lib ./configure --with-ssl
-
Updated and fixed:
Title: curl-dev.tcz
Description: Client and library for network transfers dev files.
Version: 7.20.0
Author: Daniel Stenberg
Original-site: http://curl.haxx.se/
Copying-policy: MIT
Size: 237KB
Extension_by: Jason W
Comments: Use this extension along with curl.tcz to compile
apps dependent on curl.
Change-log: 2008/09/10 First version
2008/12/12 Rebuilt for i486, size, cramfs.
2009/11/26 Updated and built against openssl-0.9.8k
Current: 2010/02/18 Updated to 7.20.0 and hardlinked against openssl-0.9.8k.
Title: curl.tcz
Description: Client and library for network transfers.
Version: 7.20.0
Author: Daniel Stenberg
Original-site: http://curl.haxx.se/
Copying-policy: MIT
Size: 200K
Extension_by: Jason W
Comments: ---------
Change-log: 2008/09/10 First version
2008/12/12 Rebuilt for i486, size, cramfs.
2009/11/26 Updated and rebuilt against openssl-0.9.8k.
Current: 2010/02/18 Updated to 7.20.0 and hardlinked against openssl-0.9.8k.
-
Since the last reported fix, I have set up various virgin test rigs using TC 2.8.1. Whenever curl is installed and a check for missing dependencies is made, openssl-0.9.8k.tcz is always reported as missing and must be installed seperately.
For example: http://forum.tinycorelinux.net/index.php?topic=5131.0
-
openssl-0.9.8k.tcz is in the curl dep file. Is it installed at the time the check for dependencies is made?
-
openssl-0.9.8k.tcz is in the curl dep file. Is it installed at the time the check for dependencies is made?
No. It is consistently identified as missing.
The latest instance of this occurred today - see the link given in my earlier post. The only apps installed to this virgin system were OSS, Midori, getFlash10 and Flash10. At this point the check for missing dependencies was run which again identified openssl-0.9.8k.tcz.
-
I will check when I get home today and see if booting "base norestore" and using the Appbrowser to install curl.tcz also fetches openssl-0.9.8k.tcz. I will also see if checking for missing deps exhibits the behavior you are seeing when those two extensions are installed.
-
As a reference point these are the results I obtained from running the tests you propose.
I will check when I get home today and see if booting "base norestore" and using the Appbrowser to install curl.tcz also fetches openssl-0.9.8k.tcz.
Booting "base norestore"
Via Appbrowser installing curl.tcz
Both curl.tcz and openssl-0.9.8k.tcz are installed.
I will also see if checking for missing deps exhibits the behavior you are seeing when those two extensions are installed.
Booting "base norestore"
Via Appbrowser installing getFlash10.tcz
curl.tcz is installed, openssl-0.9.8k.tcz is not installed. openssl-0.9.8h.tcz is listed as a dependency.
-
Thanks for the info, it is an issue with the Getflash10 dep file. I will correct it now.
-
These dep files contain curl and therefore need to have openssl-0.9.8k.tcz included. I added openssl-0.9.8k.tcz to those that do not have it. I did not remove openssl-0.9.8h.tcz from them as some of the apps themselves are built against openssl-0.9.8h and some only contain openssl in its dep file because of it is a curl dependency. I would rather prune a harmless redundant entry as they are tested rather than break the extensions and then fix them as they are reported broken.
bibletime.tcz.dep
bitstormlite.tcz.dep
centerim-encrypt.tcz.dep
centerim.tcz.dep
fbreader.tcz.dep
firefox.tcz.dep
flash.tcz.dep
getFlash10.tcz.dep
gnash-dev.tcz.dep
gnash.tcz.dep
gpredict.tcz.dep
icecat.tcz.dep
logjam.tcz.dep
moc.tcz.dep
namoroka.tcz.dep
openlierox.tcz.dep
php5-pear.tcz.dep
php5.tcz.dep
qmmp.tcz.dep
shiretoko.tcz.dep
sword-dev.tcz.dep
sword-locale.tcz.dep
sword.tcz.dep
transmission-locale.tcz.dep
transmission.tcz.dep
wormux-locale.tcz.dep
wormux.tcz.dep
xine-xvesa.tcz.dep
-
Jason: I'm sure you've got a lot on your plate, but maybe you could consider at some stage to rebuild curl with 'openssl-0.9.8m.tcz'.
That might then allow in a second step to change from 'openssl-0.9.8k.tcz' to 'openssl-0.9.8m.tcz' for all those extensions listen in reply #13 (and maybe some more). I'm happy to undertake some analysis which of the currently listed 37 extensions really require 'openssl-0.9.8k.tcz' (considering that 30 of them are also listing 'curl.tcz' as dependency).
-
I will aim for rebuilding curl tomorrow. And it should greatly get rid of the ssl-k dependence. I will then do some ldd of those extensions and those that are not linked to ssl but require curl would probably be good to then change the dep files to ssl-m.
-
Thanks Jason!! What I was offering with regards to analysis was obviously what you are planning to do anyway.
I'm looking forward to the new extension, but I was not expecting for this to happen so soon.
-
Actually, it would be a great help if it could be determined which apps that depend on curl also are linked to openssl. That would make a quick and safe transition with adjusting the dep files so we know which ones they are.
Since if the apps themselves are built agains openssl-0.9.8k, then that would have to remain a dep. But I think that is the case with very few.
-
Jason: Just read your reply #17 and will now start to undertake some analysis. I'll report back with my approach and my findings here later.
-
Thanks, that would be a great help!
-
In a way it is a bit embarrassing that it took me that long to come back with the result of my investigation, but here is now what I found out:
Executive summary:
As far as I can tell there is no need to keep the 'openssl-0.9.8k.tcz' extension. Even the 'curl.tcz' extension (which was explicitly build against '0.9.8k'), appears to work quite happily with '0.9.8m'. Hence all "inherited dependencies" should also not require '0.9.8k' any more. Please note that these tests were done using TC 2.10, so only the latest version of recursive dependency resolution was used here. But I don't think that it makes any difference to the findings as such.
Description of testing approach:
For the testing I created a script ('chk_openssl.sh', see attachments), which tests any extension specified as argument at the script execution. If no extension names are specified, all current .dep files of the tcz repository are examined, and those containing 'openssl-0.9.8k.tcz' will be tested. At the time of test execution this applied to 33 extensions.
The check consists of setting up a chroot "jail". Therefore the extension under investigation (together with all it's dependencies) will be downloaded and those files are moved into the "jail". An option exists to remove either openssl extension from the "jail" (and adjust the .dep files accordingly), so that the interesting "What if 0.9.8k is gone?" question can be investigated. After the "jail" has been set up, the "real" analysis commences in which the extension under investigation is installed (inside the "jail"). All freshly installed executables and shared objects are checked, whether they point to the '0.9.8k' versions of 'libcrypto.so' and 'libssl.so' (which are the shared libraries at the centre of the question).
Interpretation of the test results:
The main findings come from two script executions (see attachments):
(1) Using option '-vv' (for maximum verbosity), the log was captured in file 'chk_openssl-vv.log' (which can be found in 'chk_openssl-verbose.tgz'). Stripping away the verbose messages (resulting in file: 'chk_openssl.log') the result can be summarised in the following way:
- There are 182 warning messages where an openssl shared library could not be found. The following extensions contain such cases: links, tshark, vsftpd-ssl, wireshark, x11vnc, and xmms2.
Furthermore there are quite a few cases where a couple of files in 'openssl-0.9.8k.tcz' itself fail to "find" 'libcrypto.so.0.9.8' (e.g. in the '/usr/local/openssl-0.9.8k/lib/engines' directory).
A further check shows that none of these extensions contain 'openssl-0.9.8m.tcz' in their dependencies. IIRC there were in the past a few more .dep files that contained '0.9.8k' as well as '0.9.8m'. But this might be changing now due to the introduction of the recursive dependency resolution.
- There are 138 messages that indicate that a file is explicitly referring to the '0.9.8k' version of a shared library. The following extensions contain such cases: audacious-plugins, bibletime, bitstormlite, curl, fbreader, logjam, mpd, openlierox, openssl-0.9.8k, qmmp, transmission, wormux, xine-xvesa, and xmms2.
For some of these extensions 'openssl-0.9.8m.tcz' is also installed, but the "preference" for the '0.9.8k' version indicates to me that those extensions were build with '0.9.8k'.
- Finally there are 3 messages that state that there is "no version information available". All 3 files are to be found in 'dropbox.tcz'. As far as I can "see" those files were not compiled under TC, but rather extracted as binary files from a dropbox archive file.
(2) Adding the '-k' option for another test execution resulted in the "complete elimination" of 'openssl-0.9.8k.tcz', and only the 'openssl-0.9.8m.tcz' extension got installed. This produced the (verbose) result captured in 'chk_openssl-kvv.log' (found in 'chk_openssl-verbose.tgz') and the non-verbose summary in 'chk_openssl-k.log'. Comparing the results with the first test run leads to the following statements:
- All executables and shared objects of all extensions under test "found" the respective openssl shared libraries when only the 'openssl-0.9.8m.tcz' extensions was installed.
- The findings regarding the 'dropbox.tcz' extension were the same as before, that means that this extension might need to be checked individually (or rebuild from source).
Final observation:
Here is a (pretty crude) way to demonstrate that 'curl' appears to work fine with '0.9.8m':
(1) Install 'curl' and show: the version information, the openssl shared libraries used, and a simple test of a HTTPS request.
tc@box:~$ tce-load -wi curl
Downloading: openssl-0.9.8k.tcz
Connecting to 10.0.2.2 (10.0.2.2:80)
openssl-0.9.8k.tcz 100% |*******************************| 1072k --:--:-- ETA
openssl-0.9.8k.tcz: OK
Downloading: curl.tcz
Connecting to 10.0.2.2 (10.0.2.2:80)
curl.tcz 100% |*******************************| 204k --:--:-- ETA
curl.tcz: OK
tc@box:~$
tc@box:~$ curl -V
curl 7.20.0 (i686-pc-linux-gnu) libcurl/7.20.0 OpenSSL/0.9.8k zlib/1.2.3
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp
Features: IPv6 Largefile NTLM SSL libz
tc@box:~$
tc@box:~$ ldd `which curl` | grep -E 'lib(crypto|ssl)\.so' | sed 's# (.*$##'
libssl.so.0.9.8 => /usr/local/openssl-0.9.8k/lib/libssl.so.0.9.8
libcrypto.so.0.9.8 => /usr/local/openssl-0.9.8k/lib/libcrypto.so.0.9.8
tc@box:~$
tc@box:~$ curl -k https://examples.com
<html>Apache is functioning normally</html>
(2) Make 'openssl-0.9.8k' "un-available", and show that 'curl' is "broken".
tc@box:~$ sudo mv /usr/local/openssl-0.9.8k /usr/local/openssl-0.9.8k-
tc@box:~$
tc@box:~$ curl -V
curl: error while loading shared libraries: libssl.so.0.9.8: cannot open shared object file: No such file or directory
(3) Install 'openssl-0.9.8m', and repeat the steps done previously.
tc@box:~$ tce-load -wi openssl-0.9.8m
Downloading: openssl-0.9.8m.tcz
Connecting to 10.0.2.2 (10.0.2.2:80)
openssl-0.9.8m.tcz 100% |*******************************| 1088k --:--:-- ETA
openssl-0.9.8m.tcz: OK
tc@box:~$
tc@box:~$ curl -V
curl 7.20.0 (i686-pc-linux-gnu) libcurl/7.20.0 OpenSSL/0.9.8m zlib/1.2.3
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp
Features: IPv6 Largefile NTLM SSL libz
tc@box:~$
tc@box:~$ ldd `which curl` | grep -E 'lib(crypto|ssl)\.so' | sed 's# (.*$##'
libssl.so.0.9.8 => /usr/local/lib/libssl.so.0.9.8
libcrypto.so.0.9.8 => /usr/local/lib/libcrypto.so.0.9.8
tc@box:~$
tc@box:~$ curl -k https://examples.com
<html>Apache is functioning normally</html>
-
Thanks for taking the time to test. It seems like most extensions can simply have their dep files adjusted. I will download and test them one at a time and change the dep files as they prove to run ok.
-
maro: yes, thanks for the thought and effort you put into your testing.
I agree that functional testing of each extension will need to be done although if the extensions pass the ldd test that maro has provided I dont think it would be unreasonable to update the dep files and let the extension maintainers take care of or users report back any issues found. Xchat for example did not complain or stop connecting via ssl when when h was replaced by k or m, but the boxes to select ssl connection were grayed out in the network list, so I rebuilt it against m. I also think it would be nice to have a list of the extensions that will require updating when openssl is updated.
</$.02>
Kingdomcome
-
If it is agreeable then I will rebuild curl against openssl-0.9.8m, upload it, and then adjust the existing dep files. Chances are there will be little or no issue.
-
Rebuilt against openssl M version:
Title: curl-dev.tcz
Description: Client and library for network transfers dev files.
Version: 7.20.0
Author: Daniel Stenberg
Original-site: http://curl.haxx.se/
Copying-policy: MIT
Size: 236K
Extension_by: Jason W
Comments: Use this extension along with curl.tcz to compile
apps dependent on curl.
Change-log: 2008/09/10 First version
2008/12/12 Rebuilt for i486, size, cramfs.
2009/11/26 Updated and built against openssl-0.9.8k
2010/02/18 Updated to 7.20.0 and hardlinked against openssl-0.9.8k.
Current: 2010/03/28 Rebuilt against openssl-0.9.8m
Title: curl.tcz
Description: Client and library for network transfers.
Version: 7.20.0
Author: Daniel Stenberg
Original-site: http://curl.haxx.se/
Copying-policy: MIT
Size: 376K
Extension_by: Jason W
Comments: ---------
Change-log: 2008/09/10 First version
2008/12/12 Rebuilt for i486, size, cramfs.
2009/11/26 Updated and rebuilt against openssl-0.9.8k.
2010/02/18 Updated to 7.20.0 and hardlinked against openssl-0.9.8k.
Current: 2010/03/28 Rebuilt against openssl-0.9.8m
-
Dep file conversion to openssl-0.9.8m is done. The most common case was where openssl-0.9.8k was tacked at the end of the dep file because of the curl dependency. So I don't forsee any issues, but if there are please report them.
-
Jason: I just noticed that the 'curl.tcz.dep' file content changed from 'openssl-0.9.8m.tcz' to 'libldap.tcz'.
Would you please help me to understand why all those additional extensions (cyrus-sasl.tcz, gnutls.tcz, libgpg-error.tcz, libgcrypt.tcz, and libldap.tcz) are now required, since I fail to see (using 'ldd') how the 'curl.tcz' extension would require anything apart from 'openssl-0.9.8m.tcz'.
-
Jason: I just noticed that the 'curl.tcz.dep' file content changed from 'openssl-0.9.8m.tcz' to 'libldap.tcz'.
Would you please help me to understand why all those additional extensions (cyrus-sasl.tcz, gnutls.tcz, libgpg-error.tcz, libgcrypt.tcz, and libldap.tcz) are now required, since I fail to see (using 'ldd') how the 'curl.tcz' extension would require anything apart from 'openssl-0.9.8m.tcz'.
This is just the result of the dep file conversion script, see the other topic:
http://forum.tinycorelinux.net/index.php?topic=5629.0
I converted my mirror repo and got the same new dep file.
-
Perhaps it was compiled against gnutls instead of openssl this update?
-
My build notes and all are at home, but I remembered adding --disable-ldap for the build that was submitted as ldap was on my system at the time, and was not in previous builds. The libldap dependency was not removed before I uploaded the new extension. I will upload the correct dep file with openssl M version.
An error on my part making the dep file, has nothing at all to do with converting it to recursive.
-
An error on my part making the dep file, has nothing at all to do with converting it to recursive.
Of course it is just a side effect found during the test :)
-
Thanks for finding and looking into the dep file issue. With ldap installed, the curl build coughs up an error of missing ldap, so I first thought I had to have ldap installed. Later realized to use the --disable-ldap flag.
I had to take care of some other stuff today, I will fix curl.tcz.dep now.