howto convert a nonrecursive dep file into recursive?

[jls]

 

[Guy]

Some extensions are dependencies of dependencies. Or even dependencies of dependencies of dependencies.

In the old system, dependencies of dependencies were all listed in the dep file application which needed them.

With the new system, they are listed in the dep file of the dependency which needs them, so they do not need to be listed in the dep file of the application.


To convert them, dep files need to be made for all dependencies with dependencies. Then dependencies of dependencies can be left out of the dep file of the application.

[curaga]

ldd is not dependable here, as it shows the full dep chain. Use readelf to see the direct dependencies of binaries and libs:

readelf -d sylpheed | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libgtk-x11-2.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgdk-x11-2.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgdk_pixbuf-2.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libpangocairo-1.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libcairo.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libpango-1.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgobject-2.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgmodule-2.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libglib-2.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libsylpheed-plugin-0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libsylph-0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libcrypto.so.0.9.8]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

vs.

ldd sylpheed
        linux-vdso.so.1 =>  (0x00007fff64fff000)
        libgtk-x11-2.0.so.0 => /usr/lib/libgtk-x11-2.0.so.0 (0x00007f5733d52000)
        libgdk-x11-2.0.so.0 => /usr/lib/libgdk-x11-2.0.so.0 (0x00007f5733ab3000)
        libgdk_pixbuf-2.0.so.0 => /usr/lib/libgdk_pixbuf-2.0.so.0 (0x00007f573389a000)
        libpangocairo-1.0.so.0 => /usr/lib/libpangocairo-1.0.so.0 (0x00007f573368e000)
        libcairo.so.2 => /usr/lib/libcairo.so.2 (0x00007f5733413000)
        libpango-1.0.so.0 => /usr/lib/libpango-1.0.so.0 (0x00007f57331c9000)
        libgobject-2.0.so.0 => /usr/lib/libgobject-2.0.so.0 (0x00007f5732f86000)
        libgmodule-2.0.so.0 => /usr/lib/libgmodule-2.0.so.0 (0x00007f5732d83000)
        libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x00007f5732a99000)
        libsylpheed-plugin-0.so.0 => /usr/lib/libsylpheed-plugin-0.so.0 (0x00007f5732894000)
        libsylph-0.so.0 => /usr/lib/libsylph-0.so.0 (0x00007f5732625000)
        libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x00007f57322af000)
        libc.so.6 => /lib/libc.so.6 (0x00007f5731f6a000)
        libXinerama.so.1 => /usr/X11R7/lib/libXinerama.so.1 (0x00007f5731d68000)
        libXrandr.so.2 => /usr/X11R7/lib/libXrandr.so.2 (0x00007f5731b60000)
        libXcursor.so.1 => /usr/X11R7/lib/libXcursor.so.1 (0x00007f5731956000)
        libstdc++.so.6 => /usr/lib/../lib/libstdc++.so.6 (0x00007f5731653000)
        libgcc_s.so.1 => /usr/lib/../lib/libgcc_s.so.1 (0x00007f5731446000)
        libXcomposite.so.1 => /usr/X11R7/lib/libXcomposite.so.1 (0x00007f5731244000)
        libXext.so.6 => /usr/X11R7/lib/libXext.so.6 (0x00007f5731031000)
        libXdamage.so.1 => /usr/X11R7/lib/libXdamage.so.1 (0x00007f5730e2f000)
        libXfixes.so.3 => /usr/X11R7/lib/libXfixes.so.3 (0x00007f5730c2a000)
        libatk-1.0.so.0 => /usr/lib/../lib/libatk-1.0.so.0 (0x00007f5730a0c000)
        libpixman-1.so.0 => /usr/X11R7/lib/libpixman-1.so.0 (0x00007f57307b5000)
        libpng12.so.0 => /usr/lib/../lib/libpng12.so.0 (0x00007f5730592000)
        libXrender.so.1 => /usr/X11R7/lib/libXrender.so.1 (0x00007f5730389000)
        libX11.so.6 => /usr/X11R7/lib/libX11.so.6 (0x00007f573004e000)
        libxcb.so.1 => /usr/X11R7/lib/libxcb.so.1 (0x00007f572fe34000)
        libXau.so.6 => /usr/X11R7/lib/libXau.so.6 (0x00007f572fc32000)
        libXdmcp.so.6 => /usr/X11R7/lib/libXdmcp.so.6 (0x00007f572fa2d000)
        libgio-2.0.so.0 => /usr/lib/../lib/libgio-2.0.so.0 (0x00007f572f77a000)
        libresolv.so.2 => /lib/libresolv.so.2 (0x00007f572f566000)
        libpangoft2-1.0.so.0 => /usr/lib/../lib/libpangoft2-1.0.so.0 (0x00007f572f338000)
        libfontconfig.so.1 => /usr/lib/../lib/libfontconfig.so.1 (0x00007f572f0fa000)
        libfreetype.so.6 => /usr/lib/../lib/libfreetype.so.6 (0x00007f572ee6b000)
        libz.so.1 => /lib/libz.so.1 (0x00007f572ec55000)
        libexpat.so.1 => /usr/lib/../lib/libexpat.so.1 (0x00007f572ea28000)
        libdl.so.2 => /lib/libdl.so.2 (0x00007f572e824000)
        libm.so.6 => /lib/libm.so.6 (0x00007f572e5a5000)
        libnsl.so.1 => /lib/libnsl.so.1 (0x00007f572e38e000)
        libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0x00007f572e146000)
        /lib/ld-linux-x86-64.so.2 (0x00007f573430d000)

[jls]

I'm asking 4 a script 2 convert an old .dep into an new .dep which is recursive

[Juanito]

we were using a script during the testing of recursive deps, but I think it was modified - maybe it's better Jason/Robert suggest the best way to handle this

[bmarkus]

While script published by Jason does a great job, it is using a strict matemathical algorithm without knowing anything about the real dependencies. Result is a set of minimalist dep files where it is guaranteed that all necessary will be loaded but not guranteed that all real or direct dependencies kept.

Best way is to adjust dep files manually by the package creator knowing real dependencies. There are either only few and easy to handle dependencies or typical cases easy to recognize. For example many GUI based on GTK+ If you see gtk2.tcz you can remove its dependencies

atk.tcz
cairo.tcz
pango.tcz
glib2.tcz
fontconfig.tcz
expat2.tcz
libxml2.tcz
graphics-libs-1.tcz
pixman.tcz
Xlibs_supp


python.tcz , dbus-glib are similar.

These cases can be scripted to make life easier, but at the end you have to check and correct result yourself.

[cjgau]

Hi bmarkus,

not guaranteed that all real or direct dependencies kept.

if every package creator knowing real dependencies, then would it be guaranteed to have minimal real dependency so that recursive works?

[bmarkus]

Hi bmarkus,

not guaranteed that all real or direct dependencies kept.

if every package creator knowing real dependencies, then would it be guaranteed to have minimal real dependency so that recursive works?

Of course nothing is guaranteed. There are missing and unneeded dependencies for sure in current 'old' deps due to human mistakes including myself. There will be bugs in the future as well. I'm saying only that package creator knows dependencies and can make better dep lists than a script processing dep files only.

[Jason W]

If all needed dependencies are already listed in a dep file, none of them will get erased if they are not already satisfied at a lower level in the dep chain.  At least I have not seen a case using the script I use to convert my dep files once I debugged it.  Of course if you are missing dependencies in your dep file, then simply add them before converting it to recursion or add it after once checking it over.

I am not against folks converting their dep files by hand.  But I have seen so very few submissions of converted existing dep files.  I am going to post the tool I use here for those who want it, even if there is disagreement with it's use.  As with any tool, simply verify the results.  

The recursive routine used in the script was taken directly from tce-load.  Run in the same directory as your dep files are.

Code: [Select]

#!/bin/sh

rsync -av distro.ibiblio.org::distros/tinycorelinux/2.x/tcz/*.dep /tmp/repodeplist

# routine taken from tce-load
recursive_scan() {

  echo ${1} >> /tmp/deplist
  deps=""
  if [ -f /tmp/repodeplist/${1}.dep ]; then

    deps=`cat /tmp/repodeplist/${1}.dep`
    for d in $deps; do
      recursive_scan $d
    done
  fi
}

for F in `ls *.dep`; do
  for I in `cat "$F"`; do

recursive_scan "$I"

for E in `cat "$F"`; do

 if cat /tmp/deplist | grep -v "$I" | grep "$E" >/dev/null; then
   echo "$E"
  sed -i "/^"$E"/d" "$F"
 fi
 done

rm /tmp/deplist

  done

done



[bmarkus]

Original hal.tcz.dep is

expat2.tcz
glib2.tcz
dbus.tcz
dbus-glib.tcz
libpci.tcz
pci-utils.tcz
libusb.tcz
usb-utils.tcz
udevadm.tcz
acl.tcz

which is converted by the script to

dbus-glib.tcz
udevadm.tcz

I would say the correct is:

dbus-glib.tcz
pci-utils.tcz
usb-utils.tcz
udevadm.tcz

[Jason W]

pci-utils.tcz  and usb-utils.tcz are satisfied by udevadm.tcz so the result of the script is correct in terms of the definition of recursion.  Brutally efficient, but technically correct.  If there is a technical flaw in the script I definitely want to know about it as I don't want to post a flawed utility.  What you are talking about is a matter of preference as to the number of redundancies in the dep file.  Personally, I prefer none as I make no distinction whether a dependency is an app or lib.  .   

You are free to include redundancies in your dep files.  But as I understand it, not including redundancies is not incorrect. 

[curaga]

As the udevadm extension is unnecessary on >=2.9, and recursive deps require at least 2.10, I think it wouldn't be needed anymore.


edit: quick fix

[bmarkus]

pci-utils.tcz  and usb-utils.tcz are satisfied by udevadm.tcz so the result of the script is correct in terms of the definition of recursion.  Brutally efficient, but technically correct.  If there is a technical flaw in the script I definitely want to know about it as I don't want to post a flawed utility.  What you are talking about is a matter of preference as to the number of redundancies in the dep file.  Personally, I prefer none as I make no distinction whether a dependency is an app or lib.  .  

You are free to include redundancies in your dep files.  But as I understand it, not including redundancies is not incorrect.  

Jason,

script is really great and in most cases gives the proper result. What my 6th sense is saying that there will be or there are deps removed unnecesarily due to multiple dependencies which are not causing unresolved dep on the top.

Script is like machine online translation. Gives a funny result, but very useful if you do not speak target language or can be used as a basis for translation and enhance manually.

To be pragmatic, my proposal is to convert the whole repo and correct bugs on the fly. It is much faster and efficient than to wait for extension builders - including myself - to submit corrected lists. Of course, everybody is free to send dep updates later on. In such case no need to indicate it .info file as all extensions will be recursive. The only extra work is to exclude dep file already made recursive.

BTW I like the way how you are converting repo with scripts when big changes occure, like single type, squash, etc  

[bmarkus]

As the udevadm extension is unnecessary on >=2.9, and recursive deps require at least 2.10, I think it wouldn't be needed anymore.

Thanks for mentioning! Somehow I missed it. Tested, indeed HAL works fine on 2.10 with udevadm from the base. I will make necessary corrections in dep and info files.

[Jason W]

My main purpose with a script is to of course save time, and I admit it does produce some funny looking results.  Like epdfview, the gtk2 pdf viewer, has now only poppler.tcz as a dependency.  Gtk2 is often buried under gnome libs or even simpler ones like poppler.  But hopefully if it is technically correct then the extensions will load and run with no issue. 

How about this, I can script convert and upload folks extensions as they give the go ahead.  I will backup the existing dep files on the server, and then convert folks dep files as they say it is ok.  I can start with bmarkus' and we can verify to make sure it went ok.