WelcomeWelcome | FAQFAQ | DownloadsDownloads | WikiWiki

Author Topic: curl  (Read 12324 times)

Offline Jason W

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 9730
Re: curl
« Reply #15 on: March 14, 2010, 12:04:26 AM »
I will aim for rebuilding curl tomorrow.  And it should greatly get rid of the ssl-k dependence.  I will then do some ldd of those extensions and those that are not linked to ssl but require curl would probably be good to then change the dep files to ssl-m.

Offline maro

  • Hero Member
  • *****
  • Posts: 1228
Re: curl
« Reply #16 on: March 14, 2010, 12:37:20 AM »
Thanks Jason!! What I was offering with regards to analysis was obviously what you are planning to do anyway.

I'm looking forward to the new extension, but I was not expecting for this to happen so soon.

Offline Jason W

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 9730
Re: curl
« Reply #17 on: March 14, 2010, 07:58:22 PM »
Actually, it would be a great help if it could be determined which apps that depend on curl also are linked to openssl.  That would make a quick and safe transition with adjusting the dep files so we know which ones they are. 

Since if the apps themselves are built agains openssl-0.9.8k, then that would have to remain a dep.  But I think that is the case with very few.

Offline maro

  • Hero Member
  • *****
  • Posts: 1228
Re: curl
« Reply #18 on: March 14, 2010, 08:39:46 PM »
Jason: Just read your reply #17 and will now start to undertake some analysis. I'll report back with my approach and my findings here later.

Offline Jason W

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 9730
Re: curl
« Reply #19 on: March 14, 2010, 08:44:02 PM »
Thanks, that would be a great help!

Offline maro

  • Hero Member
  • *****
  • Posts: 1228
Re: curl
« Reply #20 on: March 28, 2010, 12:28:43 AM »
In a way it is a bit embarrassing that it took me that long to come back with the result of my investigation, but here is now what I found out:

Executive summary:
As far as I can tell there is no need to keep the 'openssl-0.9.8k.tcz' extension. Even the 'curl.tcz' extension (which was explicitly build against '0.9.8k'), appears to work quite happily with '0.9.8m'. Hence all "inherited dependencies" should also not require '0.9.8k' any more. Please note that these tests were done using TC 2.10, so only the latest version of recursive dependency resolution was used here. But I don't think that it makes any difference to the findings as such.

Description of testing approach:
For the testing I created a script ('chk_openssl.sh', see attachments), which tests any extension specified as argument at the script execution. If no extension names are specified, all current .dep files of the tcz repository are examined, and those containing 'openssl-0.9.8k.tcz' will be tested. At the time of test execution this applied to 33 extensions.

The check consists of setting up a chroot "jail". Therefore the extension under investigation (together with all it's dependencies) will be downloaded and those files are moved into the "jail". An option exists to remove either openssl extension from the "jail" (and adjust the .dep files accordingly), so that the interesting "What if 0.9.8k is gone?" question can be investigated. After the "jail" has been set up, the "real" analysis commences in which the extension under investigation is installed (inside the "jail"). All freshly installed executables and shared objects are checked, whether they point to the '0.9.8k' versions of 'libcrypto.so' and 'libssl.so' (which are the shared libraries at the centre of the question).

Interpretation of the test results:
The main findings come from two script executions (see attachments):

(1) Using option '-vv' (for maximum verbosity), the log was captured in file 'chk_openssl-vv.log' (which can be found in 'chk_openssl-verbose.tgz'). Stripping away the verbose messages (resulting in file: 'chk_openssl.log') the result can be summarised in the following way:
  • There are 182 warning messages where an openssl shared library could not be found. The following extensions contain such cases: links, tshark, vsftpd-ssl, wireshark, x11vnc, and xmms2.
    Furthermore there are quite a few cases where a couple of files in 'openssl-0.9.8k.tcz' itself fail to "find" 'libcrypto.so.0.9.8' (e.g. in the '/usr/local/openssl-0.9.8k/lib/engines' directory).
    A further check shows that none of these extensions contain 'openssl-0.9.8m.tcz' in their dependencies. IIRC there were in the past a few more .dep files that contained '0.9.8k' as well as '0.9.8m'. But this might be changing now due to the introduction of the recursive dependency resolution.
  • There are 138 messages that indicate that a file is explicitly referring to the '0.9.8k' version of a shared library. The following extensions contain such cases: audacious-plugins, bibletime, bitstormlite, curl, fbreader, logjam, mpd, openlierox, openssl-0.9.8k, qmmp, transmission, wormux, xine-xvesa, and xmms2.
    For some of these extensions 'openssl-0.9.8m.tcz' is also installed, but the "preference" for the '0.9.8k' version indicates to me that those extensions were build with '0.9.8k'.
  • Finally there are 3 messages that state that there is "no version information available". All 3 files are to be found in 'dropbox.tcz'. As far as I can "see" those files were not compiled under TC, but rather extracted as binary files from a dropbox archive file.

(2) Adding the '-k' option for another test execution resulted in the "complete elimination" of 'openssl-0.9.8k.tcz', and only the 'openssl-0.9.8m.tcz' extension got installed. This produced the (verbose) result captured in 'chk_openssl-kvv.log' (found in 'chk_openssl-verbose.tgz') and the non-verbose summary in 'chk_openssl-k.log'. Comparing the results with the first test run leads to the following statements:
  • All executables and shared objects of all extensions under test "found" the respective openssl shared libraries when only the 'openssl-0.9.8m.tcz' extensions was installed.
  • The findings regarding the 'dropbox.tcz' extension were the same as before, that means that this extension might need to be checked individually (or rebuild from source).

Final observation:
Here is a (pretty crude) way to demonstrate that 'curl' appears to work fine with '0.9.8m':

(1) Install 'curl' and show: the version information, the openssl shared libraries used, and a simple test of a HTTPS request.
  tc@box:~$ tce-load -wi curl
  Downloading: openssl-0.9.8k.tcz
  Connecting to 10.0.2.2 (10.0.2.2:80)
  openssl-0.9.8k.tcz   100% |*******************************|  1072k --:--:-- ETA
  openssl-0.9.8k.tcz: OK
  Downloading: curl.tcz
  Connecting to 10.0.2.2 (10.0.2.2:80)
  curl.tcz             100% |*******************************|   204k --:--:-- ETA
  curl.tcz: OK
  tc@box:~$
  tc@box:~$ curl -V
  curl 7.20.0 (i686-pc-linux-gnu) libcurl/7.20.0 OpenSSL/0.9.8k zlib/1.2.3
  Protocols: dict file ftp ftps http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp
  Features: IPv6 Largefile NTLM SSL libz
  tc@box:~$
  tc@box:~$ ldd `which curl` | grep -E 'lib(crypto|ssl)\.so' | sed 's# (.*$##'
          libssl.so.0.9.8 => /usr/local/openssl-0.9.8k/lib/libssl.so.0.9.8
          libcrypto.so.0.9.8 => /usr/local/openssl-0.9.8k/lib/libcrypto.so.0.9.8
  tc@box:~$
  tc@box:~$ curl -k https://examples.com
  <html>Apache is functioning normally</html>

(2) Make 'openssl-0.9.8k' "un-available", and show that 'curl' is "broken".
  tc@box:~$ sudo mv /usr/local/openssl-0.9.8k /usr/local/openssl-0.9.8k-
  tc@box:~$
  tc@box:~$ curl -V
  curl: error while loading shared libraries: libssl.so.0.9.8: cannot open shared object file: No such file or directory

(3) Install 'openssl-0.9.8m', and repeat the steps done previously.
  tc@box:~$ tce-load -wi openssl-0.9.8m
  Downloading: openssl-0.9.8m.tcz
  Connecting to 10.0.2.2 (10.0.2.2:80)
  openssl-0.9.8m.tcz   100% |*******************************|  1088k --:--:-- ETA
  openssl-0.9.8m.tcz: OK
  tc@box:~$
  tc@box:~$ curl -V
  curl 7.20.0 (i686-pc-linux-gnu) libcurl/7.20.0 OpenSSL/0.9.8m zlib/1.2.3
  Protocols: dict file ftp ftps http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp
  Features: IPv6 Largefile NTLM SSL libz
  tc@box:~$
  tc@box:~$ ldd `which curl` | grep -E 'lib(crypto|ssl)\.so' | sed 's# (.*$##'
          libssl.so.0.9.8 => /usr/local/lib/libssl.so.0.9.8
          libcrypto.so.0.9.8 => /usr/local/lib/libcrypto.so.0.9.8
  tc@box:~$
  tc@box:~$ curl -k https://examples.com
  <html>Apache is functioning normally</html>

Offline Jason W

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 9730
Re: curl
« Reply #21 on: March 28, 2010, 08:11:55 AM »
Thanks for taking the time to test.  It seems like most extensions can simply have their dep files adjusted.  I will download and test them one at a time and change the dep files as they prove to run ok. 


Offline Kingdomcome

  • Sr. Member
  • ****
  • Posts: 286
Re: curl
« Reply #22 on: March 28, 2010, 10:45:23 AM »
maro: yes, thanks for the thought and effort you put into your testing.

I agree that functional testing of each extension will need to be done although if the extensions pass the ldd test that maro has provided I dont think it would be unreasonable to update the dep files and let the extension maintainers take care of or users report back any issues found.  Xchat for example did not complain or stop connecting via ssl when when h was replaced by k or m, but the boxes to select ssl connection were grayed out in the network list, so I rebuilt it against m.  I also think it would be nice to have a list of the extensions that will require updating when openssl is updated.

</$.02>

Kingdomcome

Offline Jason W

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 9730
Re: curl
« Reply #23 on: March 28, 2010, 04:05:14 PM »
If it is agreeable then I will rebuild curl against openssl-0.9.8m, upload it, and then adjust the existing dep files.  Chances are there will be little or no issue.

Offline Jason W

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 9730
Re: curl
« Reply #24 on: March 28, 2010, 05:10:21 PM »
Rebuilt against openssl M version:

Code: [Select]
Title:          curl-dev.tcz
Description:    Client and library for network transfers dev files.
Version:        7.20.0
Author:         Daniel Stenberg
Original-site:  http://curl.haxx.se/
Copying-policy: MIT
Size: 236K
Extension_by:   Jason W
Comments:       Use this extension along with curl.tcz to compile
apps dependent on curl.
Change-log:     2008/09/10 First version
        2008/12/12 Rebuilt for i486, size, cramfs.
2009/11/26 Updated and built against openssl-0.9.8k
2010/02/18 Updated to 7.20.0 and hardlinked against openssl-0.9.8k.
Current: 2010/03/28 Rebuilt against openssl-0.9.8m


Code: [Select]
Title:          curl.tcz
Description:    Client and library for network transfers.
Version:        7.20.0
Author:         Daniel Stenberg
Original-site:  http://curl.haxx.se/
Copying-policy: MIT
Size: 376K
Extension_by:   Jason W
Comments:       ---------
Change-log:     2008/09/10 First version
        2008/12/12 Rebuilt for i486, size, cramfs.
2009/11/26 Updated and rebuilt against openssl-0.9.8k.
2010/02/18 Updated to 7.20.0 and hardlinked against openssl-0.9.8k.
Current: 2010/03/28 Rebuilt against openssl-0.9.8m


Offline Jason W

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 9730
Re: curl
« Reply #25 on: March 28, 2010, 07:35:24 PM »
Dep file conversion to openssl-0.9.8m  is done.  The most common case was where openssl-0.9.8k was tacked at the end of the dep file because of the curl dependency.  So I don't forsee any issues, but if there are please report them. 

Offline maro

  • Hero Member
  • *****
  • Posts: 1228
Re: curl
« Reply #26 on: April 01, 2010, 09:36:56 PM »
Jason: I just noticed that the 'curl.tcz.dep' file content changed from 'openssl-0.9.8m.tcz' to 'libldap.tcz'.

Would you please help me to understand why all those additional extensions (cyrus-sasl.tcz, gnutls.tcz, libgpg-error.tcz, libgcrypt.tcz, and libldap.tcz) are now required, since I fail to see (using 'ldd') how the 'curl.tcz' extension would require anything apart from 'openssl-0.9.8m.tcz'.

Offline bmarkus

  • Administrator
  • Hero Member
  • *****
  • Posts: 7183
    • My Community Forum
Re: curl
« Reply #27 on: April 02, 2010, 01:58:37 AM »
Jason: I just noticed that the 'curl.tcz.dep' file content changed from 'openssl-0.9.8m.tcz' to 'libldap.tcz'.

Would you please help me to understand why all those additional extensions (cyrus-sasl.tcz, gnutls.tcz, libgpg-error.tcz, libgcrypt.tcz, and libldap.tcz) are now required, since I fail to see (using 'ldd') how the 'curl.tcz' extension would require anything apart from 'openssl-0.9.8m.tcz'.

This is just the result of the dep file conversion script, see the other topic:

http://forum.tinycorelinux.net/index.php?topic=5629.0

I converted my mirror repo and got the same new dep file.
« Last Edit: April 02, 2010, 02:00:14 AM by bmarkus »
Béla
Ham Radio callsign: HA5DI

"Amateur Radio: The First Technology-Based Social Network."

Offline curaga

  • Administrator
  • Hero Member
  • *****
  • Posts: 11026
Re: curl
« Reply #28 on: April 02, 2010, 12:33:09 PM »
Perhaps it was compiled against gnutls instead of openssl this update?
The only barriers that can stop you are the ones you create yourself.

Offline Jason W

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 9730
Re: curl
« Reply #29 on: April 02, 2010, 01:06:31 PM »
My build notes and all are at home, but I remembered adding --disable-ldap for the build that was submitted as ldap was on my system at the time, and was not in previous builds.  The libldap dependency was not removed before I uploaded the new extension.    I will upload the correct dep file with openssl M version.

An error on my part making the dep file, has nothing at all to do with converting it to recursive.