Tiny Core Linux

Tiny Core Extensions => TCE Talk => Topic started by: polikuo on December 04, 2016, 10:49:14 AM

Title: A script to check dependencies
Post by: polikuo on December 04, 2016, 10:49:14 AM
I just make a script to check dependencies
It check all dependencies in a directory, that should come in handy when you're compiling  :)
At the same time, it shows which libraries are absent  :o
The result goes to standard output when the absent one goes to standard error

There're 4 different types of output
With option -f, you'll get a full list of sorted dependencies
with option -l, you'll get a long version of tcz list
with option -s, you'll get a short version of tcz list (similar to chkonboot.sh)
with option -n, you'll get a list of libraries that were not found (it goes to standard error)

The idea is quite simple

1. find all files and run ldd to check all dependencies
2. generate a full list
3. remove the files that are under the directory from the full list
4. generate a "NOTFOUND" list
5. generate a full tcz list
6. remove all redundant

Usage: get-dep-list [-option] your_top_directory
  Available Options (Default -ns)
  -f show full dependencies list
  -h show help
  -l show detected tczs (long)
  -n show dependencies that are not found
  -s show detected tczs (short)

(http://orig00.deviantart.net/630b/f/2016/339/1/a/tiny_core_get_dep_list_by_polikuo-daqn3rn.png)

Tested on 7.2 x86
Boot base norestore
tce-load only alsa
As you can see, the libsamplerate.so.0 is missing
After loading the libsamplerate.tcz extension fix the problem

(http://orig12.deviantart.net/80af/f/2016/339/0/b/tiny_core_get_dep_list_2_by_polikuo-daqn5pc.png)

By the way ~
Please add as a dependency, thanks.  ;)
However, the KERNEL module were not listed, extensions like these should be handled with care

Code here:
Code: [Select]
#!/bin/sh
# author polikuo
# version 1.0
# Dec 4 2016

. /etc/init.d/tc-functions
useBusybox

showhelp() {
  echo ${YELLOW}
  echo 'Usage: '"$(basename $0)" [-option] your_top_directory
  echo '  'Available Options \(Default -ns\)
  echo '  '\-f show full dependencies list
  echo '  '\-h show help
  echo '  '\-l show detected tczs \(long\)
  echo '  '\-n show dependencies that are not found
  echo '  '\-s show detected tczs \(short\)
  echo ${NORMAL}
  exit 1
}

unset TCEDIR LIBS NOTFOUND ELF TCZS file DEP DELETE not_needed LIST NN NF option B D F L O N S T DIR SELF REMOVE

DIR=false
for O in $@;
do
  echo $O | grep -Eqv "^-" && {
    cd $O 2> /dev/null && { DIR=true; break; } || { echo ${RED}can\'t cd to $O${NORMAL}; break; }
  }
done

(( $DIR )) || showhelp

D=0
F=false
L=false
N=false
S=false
while getopts fhlns option
do
case "$option" in
    f ) F=true; D=$((D+1));;
    h ) showhelp;;
    l ) L=true; D=$((D+1));;
    n ) N=true; D=$((D+1));;
    s ) S=true; D=$((D+1));;
    * ) showhelp;;
esac
done
[ $D -eq 0 ] && { N=true;S=true; }

# Preparing

TCEDIR=/etc/sysconfig/tcedir
ELF="$(find . -not -type d -exec ldd {} + 2> /dev/null | cut -d '(' -f 1 | tr -d '\t' | sort | uniq)"
SELF="$(for B in `find . -not -type d`; do basename $B; done | grep -Ev "\.$")"
LIBS="$(echo "$ELF" | tr -d ' ' | grep -v "notfound" | cut -d '>' -f 2)"
for REMOVE in $SELF
do
  LIBS="$(echo "$LIBS" | grep -v $REMOVE)"
done
NOTFOUND="$(echo "$ELF" | tr -d ' ' | grep "notfound" | cut -d '=' -f 1)"
TCZS="$(ls -l $LIBS 2> /dev/null | grep /tmp/tcloop/ | cut -d '>' -f 2 | cut -d '/' -f 4 | sort | uniq | sed 's/$/\.tcz/g')"

(( $F )) && echo "$ELF" && echo 1>&0
(( $L )) && echo "$TCZS" && echo 1>&0

# checking if TCZS existed

for file in $TCZS
do
  [ -s "$TCEDIR"/optional/"$file" ] || {
    echo ${MAGENTA}$file not found or size is zero${NORMAL}
    echo If you load it elsewhere, please move it and the dep file into your
    echo ${CYAN}$(ls -l /etc/sysconfig | grep tcedir | awk '{ print $11 "optional/" }')${NORMAL}
    exit 1
  }
done

# checking for repeated

chkrpt() {
  for DEP in $TCZS
  do
    if [ -s "$TCEDIR"/optional/"$DEP".dep ]
    then
      for DELETE in $(cat "$TCEDIR"/optional/"$DEP".dep)
      do
        if ( echo "$TCZS" | grep -q "$DELETE" )
        then
          not_needed="$not_needed"" $DELETE"
        fi
      done
    fi
  done

  LIST="$TCZS"

  for NN in `echo $not_needed | tr ' ' '\n' | sort | uniq`
  do
    LIST=`echo $LIST | sed "s/$NN//g"`
  done

  for DEP in $LIST
  do
    echo ${CYAN}$DEP${NORMAL}
  done
}

(( $S )) && chkrpt && echo 1>&0

# Show NOTFOUND

snf() {
  [ "$(echo "$NOTFOUND" | tr -d ' ')" = '' ] || {
  echo ${RED}"These files need to be checked"${NORMAL} 1>&2
  for NF in $NOTFOUND; do
    echo ${MAGENTA}$NF${NORMAL}
  done | sort | uniq 1>&2
  }
}

(( $N )) && snf

I've found many extensions either having missing dependencies or loading too much extensions
I'm afraid I can't list them all, but I hope this script would help

Feedbacks are welcomed  :D
Title: Re: A script to check dependencies
Post by: andyj on December 04, 2016, 07:54:42 PM
1. Not all dependencies are libraries, just as not all extensions are libraries. Be careful when you say extensions seem to have unneeded dependecies.
2. Watch out for circular (mutual) dependencies. Freetype <-> harfbuzz and cyrus-sasl <-> openldap are two that I know of off the top of my head.
3. Many dependencies are optional. This is particularly true for programs that use plugins / extensions / modules like Apache, PHP, LDAP, rsyslog, Postgresql, etc.
4. This requires that copy2fs.flg or copy2fs.lst not be used. Might be a problem in some cases.
Title: Re: A script to check dependencies
Post by: bmarkus on December 04, 2016, 09:47:17 PM
I'm using a similar script based on ldd and it is extremely useful to create a proper .dep file. andyj's comments are correct on the limitations, it helps a lot but you must fine tune result manually.

Few additional notes:

- With ldd you can't detect libraries which are detected by application at runtime and not linked
- Scripts inside of TCZ need their interpreter like BASH, PERL, PYTHON, etc.
- Required kernel modules need attention; most often forgotten is ipv6-KERNEL.tcz

Knowing these limitations it is really a useful tool! :)
Title: Re: A script to check dependencies
Post by: bmarkus on December 04, 2016, 10:06:15 PM
Ho can I save the result in a .dep file? Using > formatting color codes are saved too.
Title: Re: A script to check dependencies
Post by: polikuo on December 04, 2016, 10:10:10 PM
1. Not all dependencies are libraries, just as not all extensions are libraries. Be careful when you say extensions seem to have unneeded dependecies.
2. Watch out for circular (mutual) dependencies. Freetype <-> harfbuzz and cyrus-sasl <-> openldap are two that I know of off the top of my head.
3. Many dependencies are optional. This is particularly true for programs that use plugins / extensions / modules like Apache, PHP, LDAP, rsyslog, Postgresql, etc.
4. This requires that copy2fs.flg or copy2fs.lst not be used. Might be a problem in some cases.

Thanks for your reply~
I do understand that not all dependencies are libraries, I'm just trying to provide a quick solution for some general extensions that follows this rule

Scripting is never a perfect solution, but it serves as a reference
What I find it helpful is that it reminds you which files are missing

As for "copy2fs" part, my script test if your tcz existed in your TCEDIR
If I can't find the extension, the script exit with error messages

For the "optional" part, I believe the extension manager who made it would pick them wisely
Finally I have a question here
Does ldd link to the "optional" extensons ?
Title: Re: A script to check dependencies
Post by: polikuo on December 04, 2016, 10:14:38 PM
Ho can I save the result in a .dep file? Using > formatting color codes are saved too.
Oops !
Sorry I have no idea that happens.
I always run it under a terminal emulator
Then copy the deps into the editor
You can remove the "color call" from the script
line 10 ${YELLOW}
line 18 ${NORMAL}
line 106 ${CYAN} ${NORMAL}
line 116 ${RED} ${NORMAL}
line 118 ${MAGENTA} ${NORMAL}
Title: Re: A script to check dependencies
Post by: bmarkus on December 04, 2016, 10:27:38 PM
Don't ask users to edit the output list, just provide a plain text output. It is better to make it by default and use a -c option to create a colored list if someone need it.
Title: Re: A script to check dependencies
Post by: polikuo on December 04, 2016, 10:51:14 PM
Code: [Select]
#!/bin/sh
# author polikuo
# version 1.1
# Dec 4 2016

. /etc/init.d/tc-functions
useBusybox

showhelp() {
  (( $C )) && echo -n ${YELLOW}
  echo 'Usage: '"$(basename $0)" [-option] your_top_directory
  echo '  'Available Options \(Default -ns\)
  echo '  '\-c colorful output \(same as -cns\)
  echo '  '\-f show full dependencies list
  echo '  '\-h show help
  echo '  '\-l show detected tczs \(long\)
  echo '  '\-n show dependencies that are not found
  echo '  '\-s show detected tczs \(short\)
  (( $C )) && echo -n ${NORMAL}
  exit 1
}

unset TCEDIR LIBS NOTFOUND ELF TCZS file DEP DELETE not_needed LIST NN NF option B C D F L O N S T DIR SELF REMOVE

DIR=false
for O in $@;
do
  echo $O | grep -Eqv "^-" && {
    cd $O 2> /dev/null && { DIR=true; break; } || {
      (( $C )) && echo -n ${RED}
      echo can\'t cd to $O
      (( $C )) && echo -n ${NORMAL}
      break
    }
  }
done

(( $DIR )) || showhelp

D=0
C=false
F=false
L=false
N=false
S=false
while getopts cfhlns option
do
case "$option" in
    c ) C=true;;
    f ) F=true; D=$((D+1));;
    h ) showhelp;;
    l ) L=true; D=$((D+1));;
    n ) N=true; D=$((D+1));;
    s ) S=true; D=$((D+1));;
    * ) showhelp;;
esac
done
[ $D -eq 0 ] && { N=true;S=true; }

# Preparing

TCEDIR=/etc/sysconfig/tcedir
ELF="$(find . -not -type d -exec ldd {} + 2> /dev/null | cut -d '(' -f 1 | tr -d '\t' | sort | uniq)"
SELF="$(for B in `find . -not -type d`; do basename $B; done | grep -Ev "\.$")"
LIBS="$(echo "$ELF" | tr -d ' ' | grep -v "notfound" | cut -d '>' -f 2)"
for REMOVE in $SELF
do
  LIBS="$(echo "$LIBS" | grep -v $REMOVE)"
done
NOTFOUND="$(echo "$ELF" | tr -d ' ' | grep "notfound" | cut -d '=' -f 1)"
TCZS="$(ls -l $LIBS 2> /dev/null | grep /tmp/tcloop/ | cut -d '>' -f 2 | cut -d '/' -f 4 | sort | uniq | sed 's/$/\.tcz/g')"

(( $F )) && {
  (( $C )) && echo -n ${GREEN}
  echo "$ELF" && echo 1>&0
  (( $C )) && echo -n ${NORMAL}
}
(( $L )) && {
  (( $C )) && echo -n ${BLUE}
  echo "$TCZS" && echo 1>&0
  (( $C )) && echo -n ${NORMAL}
}

# checking if TCZS existed

for file in $TCZS
do
  [ -s "$TCEDIR"/optional/"$file" ] || {
    (( $C )) && echo -n ${MAGENTA}
    echo $file not found or size is zero
    (( $C )) && echo -n ${NORMAL}
    echo If you load it elsewhere, please move it and the dep file into your
    (( $C )) && echo -n ${CYAN}
    echo $(ls -l /etc/sysconfig | grep tcedir | awk '{ print $11 "optional/" }')
    (( $C )) && echo -n ${NORMAL}
    exit 1
  }
done

# checking for repeated

chkrpt() {
  for DEP in $TCZS
  do
    if [ -s "$TCEDIR"/optional/"$DEP".dep ]
    then
      for DELETE in $(cat "$TCEDIR"/optional/"$DEP".dep)
      do
        if ( echo "$TCZS" | grep -q "$DELETE" )
        then
          not_needed="$not_needed"" $DELETE"
        fi
      done
    fi
  done

  LIST="$TCZS"

  for NN in `echo $not_needed | tr ' ' '\n' | sort | uniq`
  do
    LIST=`echo $LIST | sed "s/$NN//g"`
  done

  for DEP in $LIST
  do
    (( $C )) && echo -n ${CYAN}
    echo $DEP
    (( $C )) && echo -n ${NORMAL}
  done
}

(( $S )) && chkrpt && echo 1>&0

# Show NOTFOUND

snf() {
  [ "$(echo "$NOTFOUND" | tr -d ' ')" = '' ] || {
  (( $C )) && echo -n ${RED} 1>&2
  echo "These files need to be checked" 1>&2
  (( $C )) && echo -n ${NORMAL} 1>&2
  for NF in $NOTFOUND; do
    (( $C )) && echo -n ${MAGENTA}
    echo $NF
    (( $C )) && echo -n ${NORMAL}
  done | sort | uniq 1>&2
  }
}

(( $N )) && snf

Fixed
Title: Re: A script to check dependencies
Post by: Juanito on December 05, 2016, 01:53:43 AM
Does ldd link to the "optional" extensons ?

These would typically be found under /usr/local/lib/extension_name/ and could be both libs or executables (libexec).
Title: Re: A script to check dependencies
Post by: andyj on December 05, 2016, 03:09:35 AM
I have my own script as well, which is how I learned quickly of the limitations of such scripts. My script will list dependencies to optionally loaded libraries, but as an extention maintainer I recognize the optional from required dependencies. If you are going to try this with extentions you are unfamiliar with proceed cautiously.
Title: Re: A script to check dependencies
Post by: polikuo on December 05, 2016, 03:17:57 AM
If you are going to try this with extentions you are unfamiliar with proceed cautiously.
I'll keep that in mind
Thanks for your advice
Title: Re: A script to check dependencies
Post by: polikuo on December 05, 2016, 05:45:10 AM
Code: [Select]
#!/bin/sh
# author polikuo
# version 1.2
# Dec 4 2016

. /etc/init.d/tc-functions
useBusybox

showhelp() {
  (( $C )) && echo -n ${YELLOW}
  echo 'Usage: '"$(basename $0)" [-option] your_top_directory
  echo '  'Available Options \(Default -ns\)
  echo '  '\-c colorful output \(same as -cns\)
  echo '  '\-f show full dependencies list
  echo '  '\-h show help
  echo '  '\-l show detected tczs \(long\)
  echo '  '\-n show dependencies that are not found
  echo '  '\-s show detected tczs \(short\)
  (( $C )) && echo -n ${NORMAL}
  exit 1
}

unset TCEDIR LIBS NOTFOUND ELF TCZS file DEP DELETE not_needed LIST NN NF option B C D F L O N S T DIR SELF REMOVE

DIR=false
for O in $@;
do
  echo $O | grep -Eqv "^-" && {
    cd $O 2> /dev/null && { DIR=true; break; } || {
      (( $C )) && echo -n ${RED}
      echo can\'t cd to $O
      (( $C )) && echo -n ${NORMAL}
      break
    }
  }
done

(( $DIR )) || showhelp

D=0
C=false
F=false
L=false
N=false
S=false
while getopts cfhlns option
do
case "$option" in
    c ) C=true;;
    f ) F=true; D=$((D+1));;
    h ) showhelp;;
    l ) L=true; D=$((D+1));;
    n ) N=true; D=$((D+1));;
    s ) S=true; D=$((D+1));;
    * ) showhelp;;
esac
done
[ $D -eq 0 ] && { N=true;S=true; }

# Preparing

TCEDIR=/etc/sysconfig/tcedir
ELF="$(find . -not -type d -exec ldd {} + 2> /dev/null | cut -d '(' -f 1 | tr -d '\t' | sort | uniq)"
SELF="$(for B in `find . -not -type d`; do basename $B; done | grep -Ev "\.$")"
LIBS="$(echo "$ELF" | tr -d ' ' | grep -v "notfound" | cut -d '>' -f 2)"
NOTFOUND="$(echo "$ELF" | tr -d ' ' | grep "notfound" | cut -d '=' -f 1)"
TCZS="$(ls -l $LIBS 2> /dev/null | grep /tmp/tcloop/ | cut -d '>' -f 2 | cut -d '/' -f 4 | sort | uniq | sed 's/$/\.tcz/g')"

(( $F )) && {
  (( $C )) && echo -n ${GREEN}
  echo "$ELF" && echo 1>&0
  (( $C )) && echo -n ${NORMAL}
}
(( $L )) && {
  (( $C )) && echo -n ${BLUE}
  echo "$TCZS" && echo 1>&0
  (( $C )) && echo -n ${NORMAL}
}

for REMOVE in $SELF
do
  LIBS="$(echo "$LIBS" | grep -v $REMOVE)"
  NOTFOUND="$(echo "$NOTFOUND" | grep -v $REMOVE)"
done

# checking if TCZS existed

for file in $TCZS
do
  [ -s "$TCEDIR"/optional/"$file" ] || {
    (( $C )) && echo -n ${MAGENTA}
    echo $file not found or size is zero
    (( $C )) && echo -n ${NORMAL}
    echo If you load it elsewhere, please move it and the dep file into your
    (( $C )) && echo -n ${CYAN}
    echo $(ls -l /etc/sysconfig | grep tcedir | awk '{ print $11 "optional/" }')
    (( $C )) && echo -n ${NORMAL}
    exit 1
  }
done

# checking for repeated

chkrpt() {
  for DEP in $TCZS
  do
    if [ -s "$TCEDIR"/optional/"$DEP".dep ]
    then
      for DELETE in $(cat "$TCEDIR"/optional/"$DEP".dep)
      do
        if ( echo "$TCZS" | grep -q "$DELETE" )
        then
          not_needed="$not_needed"" $DELETE"
        fi
      done
    fi
  done

  LIST="$TCZS"

  for NN in `echo $not_needed | tr ' ' '\n' | sort | uniq`
  do
    LIST=`echo $LIST | sed "s/$NN//g"`
  done

  for DEP in $LIST
  do
    (( $C )) && echo -n ${CYAN}
    echo $DEP
    (( $C )) && echo -n ${NORMAL}
  done
}

(( $S )) && chkrpt && echo 1>&0

# Show NOTFOUND

snf() {
  [ "$(echo "$NOTFOUND" | tr -d '[\ \t\n]')" = '' ] || {
  (( $C )) && echo -n ${RED} 1>&2
  echo "These files need to be checked" 1>&2
  (( $C )) && echo -n ${NORMAL} 1>&2
  for NF in $NOTFOUND; do
    (( $C )) && echo -n ${MAGENTA}
    echo $NF
    (( $C )) && echo -n ${NORMAL}
  done | sort | uniq 1>&2
  }
}

(( $N )) && snf

I made some stupid mistakes which caused me a lot of trouble
Fix the NOTFOUND function
Title: Re: A script to check dependencies
Post by: polikuo on December 09, 2016, 12:16:07 AM
Oops, wrong script, my bad :-[

Code: [Select]
#!/bin/sh
# author polikuo
# version 1.3
# Dec 9 2016

. /etc/init.d/tc-functions
useBusybox

showhelp() {
  (( $C )) && echo -n ${YELLOW}
  echo 'Usage: '"$(basename $0)" [-option] your_top_directory
  echo '  'Available Options \(Default -ns\)
  echo '  '\-c colorful output \(same as -cns\)
  echo '  '\-f show full dependencies list
  echo '  '\-h show help
  echo '  '\-l show detected tczs \(long\)
  echo '  '\-n show dependencies that are not found
  echo '  '\-s show detected tczs \(short\)
  (( $C )) && echo -n ${NORMAL}
  exit 1
}

unset TCEDIR LIBS NOTFOUND ELF TCZS file DEP DELETE not_needed LIST NN NF option B C D F L O N S T DIR SELF REMOVE

DIR=false
for O in $@;
do
  echo $O | grep -Eqv "^-" && {
    cd $O 2> /dev/null && { DIR=true; break; } || {
      (( $C )) && echo -n ${RED}
      echo can\'t cd to $O
      (( $C )) && echo -n ${NORMAL}
      break
    }
  }
done

(( $DIR )) || showhelp

D=0
C=false
F=false
L=false
N=false
S=false
while getopts cfhlns option
do
case "$option" in
    c ) C=true;;
    f ) F=true; D=$((D+1));;
    h ) showhelp;;
    l ) L=true; D=$((D+1));;
    n ) N=true; D=$((D+1));;
    s ) S=true; D=$((D+1));;
    * ) showhelp;;
esac
done
[ $D -eq 0 ] && { N=true;S=true; }

# Preparing

TCEDIR=/etc/sysconfig/tcedir
ELF="$(find . -not -type d -exec ldd {} + 2> /dev/null | cut -d '(' -f 1 | tr -d '\t' | sort | uniq)"
SELF="$(for B in `find . -not -type d`; do basename $B; done | grep -Ev "\.$")"
LIBS="$(echo "$ELF" | tr -d ' ' | grep -v "notfound" | cut -d '>' -f 2)"

for REMOVE in $SELF
do
  LIBS="$(echo "$LIBS" | grep -v $REMOVE)"
  NOTFOUND="$(echo "$NOTFOUND" | grep -v $REMOVE)"
done

NOTFOUND="$(echo "$ELF" | tr -d ' ' | grep "notfound" | cut -d '=' -f 1)"
TCZS="$(ls -l $LIBS 2> /dev/null | grep /tmp/tcloop/ | cut -d '>' -f 2 | cut -d '/' -f 4 | sort | uniq | sed 's/$/\.tcz/g')"

(( $F )) && {
  (( $C )) && echo -n ${GREEN}
  echo "$ELF" && echo 1>&0
  (( $C )) && echo -n ${NORMAL}
}
(( $L )) && {
  (( $C )) && echo -n ${BLUE}
  echo "$TCZS" && echo 1>&0
  (( $C )) && echo -n ${NORMAL}
}

# checking if TCZS existed

for file in $TCZS
do
  [ -s "$TCEDIR"/optional/"$file" ] || {
    (( $C )) && echo -n ${MAGENTA}
    echo $file not found or size is zero
    (( $C )) && echo -n ${NORMAL}
    echo If you load it elsewhere, please move it and the dep file into your
    (( $C )) && echo -n ${CYAN}
    echo $(ls -l /etc/sysconfig | grep tcedir | awk '{ print $11 "optional/" }')
    (( $C )) && echo -n ${NORMAL}
    exit 1
  }
done

# checking for repeated

chkrpt() {
  for DEP in $TCZS
  do
    if [ -s "$TCEDIR"/optional/"$DEP".dep ]
    then
      for DELETE in $(cat "$TCEDIR"/optional/"$DEP".dep)
      do
        if ( echo "$TCZS" | grep -q "$DELETE" )
        then
          not_needed="$not_needed"" $DELETE"
        fi
      done
    fi
  done

  LIST="$TCZS"

  for NN in `echo $not_needed | tr ' ' '\n' | sort | uniq`
  do
    LIST=`echo $LIST | sed "s/$NN//g"`
  done

  for DEP in $LIST
  do
    (( $C )) && echo -n ${CYAN}
    echo $DEP
    (( $C )) && echo -n ${NORMAL}
  done
}

(( $S )) && chkrpt && echo 1>&0

# Show NOTFOUND

snf() {
  [ "$(echo "$NOTFOUND" | tr -d '[\ \t\n]')" = '' ] || {
  (( $C )) && echo -n ${RED} 1>&2
  echo "These files need to be checked" 1>&2
  (( $C )) && echo -n ${NORMAL} 1>&2
  for NF in $NOTFOUND; do
    (( $C )) && echo -n ${MAGENTA}
    echo $NF
    (( $C )) && echo -n ${NORMAL}
  done | sort | uniq 1>&2
  }
}

(( $N )) && snf
Title: Re: A script to check dependencies
Post by: polikuo on January 05, 2017, 10:45:11 AM
In case anybody is still interested in this.
Tested on x86_64.
I practically re-write the script.  :P
bug fix, add new features, check copy2fs  8)

Code: [Select]
tc@box:~ $ script [-option] your_top_directory [append your list here ...]
### for example ###
tc@box:~ $ script -s /tmp/tcloop/gnome-session `cat /etc/sysconfig/tcedir/optional/gnome-session.tcz.dep`
adwaita-icon-theme.tcz
cantarell-fonts.tcz
desktop-file-utils.tcz
epiphany.tcz
gdm-gir.tcz
gedit-gir.tcz
geocode-glib-gir.tcz
gnome-backgrounds.tcz
gnome-terminal.tcz
gvfs.tcz
mousetweaks.tcz
nautilus-gir.tcz
telepathy-mission-control.tcz
udev-extra.tcz
udisks-gir.tcz
vte-gir.tcz
yelp.tcz

As you can see, the output list seems to be shorter than original.
However, if you load the list, you'll get the same result.
Why is that happening ?

Take a look at these 2 extensions  :)
  gnome-desktop-gir.tcz
  gnome-shell-gir.tcz

They are both present in gnome-session.tcz.dep
Yet, if you trace the dep files manually, you'll get:
gnome-session
  gdm-gir
    gnome-shell-gir
      mutter-gir
        gnome-desktop-gir

gnome-desktop-gir and gnome-shell-gir are both dependencies of gdm-gir
The chkonboot.sh would't fix this because the script only cat the dep file once
BTW, the new list works as expected.  ;D

My script fix this bug by looping, this will takes a while if you're scanning big extensions, but the result is quite satisfying.
As for the kernel modules, such as alsa-modules-4.2.9-tinycore64.tcz,
KERNEL is auto-renamed to fit your current kernel when scanning, and is auto-renamed to KERNEL when outputting.

Code here:
Code: [Select]
#!/bin/sh
# author polikuo
# version 1.4
# Jan 6 2017

. /etc/init.d/tc-functions
useBusybox
unset A APPEND C COPY2FS D DELETE DETECTED_TCZS DIR EXT F FILE KERNELVER LDD_OUT LIBS LIST LO N NEWLINE NF NN NOTFOUND NOT_NEEDED R RPT RS S SCAN SELF TCEDIR TCZS option

showhelp() {
( $C ) && echo -n ${YELLOW} 1>&2
cat << END_HELP 1>&2
Usage: $(basename $0) [-option] your_top_directory [append your list here ...]
  Available Options (Default -ns)
  -c colorful output (same as -cns)
  -d show only detected tczs, without custom extensions (cyan)
  -f show full detected dependencies with ldd (white)
  -h show this helpinfo (yellow)
  -n show dependencies that are not found (red)
  -r show recursively scanned tczs (full)
  -s show detected tczs (short)
END_HELP
( $C ) && echo -n ${NORMAL} 1>&2
exit 1
}

# check if TCZS existed
chkexist() {
  for FILE in $@
  do
    [ -s "$TCEDIR"/optional/"$FILE" ] || {
      ( $C ) && echo -n ${MAGENTA}
      echo $FILE not found or size is zero
      echo If you load it elsewhere, please move it and the dep file into your
      echo "$TCEDIR""/optional/"
      ( $C ) && echo -n ${NORMAL}
      exit 1
    }
  done
}

# check repeated
chkrpt() {
  for RPT in $TCZS
  do
    if [ -s "$TCEDIR"/optional/"$RPT".dep ]
    then
      for DELETE in $(cat "$TCEDIR"/optional/"$RPT".dep)
      do
        DELETE=${DELETE##*/}
        DELETE=${DELETE%.tcz}
        DELETE=${DELETE//KERNEL/$KERNELVER}
        DELETE=${DELETE}".tcz"
        if ( echo "$TCZS" | grep -q "$DELETE" )
        then
          NOT_NEEDED="$NOT_NEEDED"" $DELETE"
        fi
      done
    fi
  done

  LIST="$TCZS"

  for NN in $(echo $NOT_NEEDED | tr ' ' '\n' | sort -u)
  do
    LIST="$(echo "$LIST" | sed "/^$NN$/d")"
  done

  ( $C ) && echo -n ${CYAN}
  echo "${LIST//$KERNELVER.tcz/KERNEL.tcz}"
  ( $C ) && echo -n ${NORMAL}
}

# recursively scan dependencies
rscan() {
  cd $TCEDIR/optional
  echo -e "$@" | awk '
  function awk_scan(name, depfile, line) {
    if (name) {
      sub(/KERNEL/, KERNELVER, name)
      depfile=name".dep"
      printf "%s\n", name
      while (getline line < depfile > 0)
        awk_scan(line)
      close(depfile)
    }
  }
  BEGIN {KERNELVER="'"$KERNELVER"'"; IDX=0;}
  {awk_scan($1)}
  '
}

# Options
A=false
C=false
D=false
F=false
N=false
R=false
S=false
while getopts cdfhnrs option 2> /dev/null
do
  case "$option" in
    c ) C=true;;
    d ) D=true; A=true;;
    f ) F=true; A=true;;
    h ) showhelp;;
    n ) N=true; A=true;;
    r ) R=true; A=true;;
    s ) S=true; A=true;;
    * ) showhelp;;
  esac
done
( $A ) || { N=true;S=true; }
shift $((OPTIND-1))

# directory to scan
DIR=false
[ -n "$1" ] || showhelp
cd $1 && DIR=true
( $DIR ) || showhelp
shift

# additional custom extensions
NEWLINE="
"
KERNELVER="$(uname -r)"
[ "$#" -ne 0 ] && {
  for EXT in $@
  do
    EXT=${EXT##*/}
    EXT=${EXT%.tcz}
    EXT=${EXT//KERNEL/$KERNELVER}
    EXT=${EXT}".tcz"
    APPEND="$EXT""$NEWLINE""$APPEND"
  done
}

# copy2fs
COPY2FS=false
TCEDIR="$(realpath /etc/sysconfig/tcedir)"
[ -f $TCEDIR/copy2fs.lst ] && {
  ( $C ) && echo -n ${RED} 1>&2
  echo WARNING! copy2fs.lst found in $TCEDIR, please disable it and reboot 1>&2
  ( $C ) && echo -n ${NORMAL} 1>&2
  COPY2FS=true
}
[ -f $TCEDIR/copy2fs.flg ] && {
  ( $C ) && echo -n ${RED} 1>&2
  echo WARNING! copy2fs.flg found in $TCEDIR, please disable it and reboot 1>&2
  ( $C ) && echo -n ${NORMAL} 1>&2
  COPY2FS=true
}
( $COPY2FS ) && exit 1

# main
LDD_OUT="$(find . -not -type d -exec ldd {} + 2> /dev/null)"
LO="$(echo "$LDD_OUT" | cut -d '(' -f 1 | tr -d '[[:blank:]]')"
SELF="$(find . -not -type d -exec sh -c 'basename "$1"' _ {} \;)"
LIBS="$(echo "$LO" | grep '/' | cut -d '>' -f 2 | sort -u)"
( $N ) && NOTFOUND="$(echo "$LO" | grep "notfound$" | sort -u)"

for REMOVE in $SELF
do
  ( $D ) || ( $R ) || ( $S ) && LIBS="$(echo "$LIBS" | grep -v "/$REMOVE$")"
  ( $N ) && NOTFOUND="$(echo "$NOTFOUND" | grep -v "^$REMOVE=>notfound$")"
done

DETECTED_TCZS="$(realpath $LIBS 2> /dev/null | grep '^/tmp/tcloop/' | cut -d '/' -f 4 | sort -u | sed 's/$/\.tcz/g')"
SCAN="$APPEND""$DETECTED_TCZS"
( $R ) || ( $S ) && { TCZS="$(for RS in $SCAN; do rscan $RS;done | sort -u)";chkexist "$TCZS"; }
NOTFOUND="$(echo "$NOTFOUND" | cut -d = -f 1)"

# output
( $D ) && {
  chkexist "$DETECTED_TCZS"
  ( $C ) && echo -n ${CYAN}
  echo "${DETECTED_TCZS//$KERNELVER.tcz/KERNEL.tcz}"
  ( $C ) && echo -n ${NORMAL}
}

( $F ) && {
  echo "$LDD_OUT"
}

( $N ) && [ -n "$NOTFOUND" ] && {
  ( $C ) && echo -n ${RED} 1>&2
  echo These dependencies are missing 1>&2
  echo "$NOTFOUND" 1>&2
  ( $C ) && echo -n ${NORMAL} 1>&2
}

( $R ) && {
  ( $C ) && echo -n ${GREEN}
  echo "${TCZS//$KERNELVER.tcz/KERNEL.tcz}"
  ( $C ) && echo -n ${NORMAL}
}

( $S ) && chkrpt

unset A APPEND C COPY2FS D DELETE DETECTED_TCZS DIR EXT F FILE KERNELVER LDD_OUT LIBS LIST LO N NEWLINE NF NN NOTFOUND NOT_NEEDED R RPT RS S SCAN SELF TCEDIR TCZS option

Feedbacks are welcomed  :D
Title: Re: A script to check dependencies
Post by: patrikg on January 06, 2017, 01:29:14 AM
My comment: Have you seen:
Halma on this forum have the following script thread
http://forum.tinycorelinux.net/index.php/topic,17454.0.html
Title: Re: A script to check dependencies
Post by: polikuo on January 06, 2017, 03:18:40 AM
My comment: Have you seen:
Halma on this forum have the following script thread
http://forum.tinycorelinux.net/index.php/topic,17454.0.html

Hi, thanks for your reply.
I honestly haven't seen the post before.
A great script fetching desired extensions.  :D
Although the script doesn't take care of KERNEL modules...

My goal here is to find dependencies and generate .dep files when creating extensions.
I put the recursive scan function in it to prevent unwanted tczs to be listed in the .dep file.
By unwanted tczs, I'm talking about extensions like gnome-session.

The current dep tree look like this:
gnome-session
 |--gnome-themes-standard -----> redundant, can be removed
 |--gdm-gir
 |   `--gnome-shell-gir
 |        `--gnome-shell
 |             `--gnome-themes-standard -----> already called here
 |--some
 |--other
 `--tczs

Sorry that I didn't made my point clear at my previous post.
Title: Re: A script to check dependencies
Post by: Juanito on January 06, 2017, 04:22:08 AM
gnome-themes-standard, gnome-desktop-gir and gnome-shell-gir removed from gnome-session dep file - thanks
Title: Re: A script to check dependencies
Post by: Juanito on January 07, 2017, 01:02:29 AM
When I run your latest script I get:
Code: [Select]
$ ./dep_check
./dep_check: line 32: syntax error: unexpected "(" (expecting "}")
Title: Re: A script to check dependencies
Post by: polikuo on January 07, 2017, 03:58:25 AM
When I run your latest script I get:
Code: [Select]
$ ./dep_check
./dep_check: line 32: syntax error: unexpected "(" (expecting "}")
Did you accidentally add/delete anything by chance ?
My script will show help info if there are no arguments
I just copy the script from forum, and this is what I get:
Code: [Select]
tc@box:~$ ./dep_check
Usage: dep_check [-option] your_top_directory [append your list here ...]
  Available Options (Default -ns)
  -c colorful output (same as -cns)
  -d show only detected tczs, without custom extensions (cyan)
  -f show full detected dependencies with ldd (white)
  -h show this helpinfo (yellow)
  -n show dependencies that are not found (red)
  -r show recursively scanned tczs (full)
  -s show detected tczs (short)
tc@box:~$ ./dep_check /tmp/tcloop/firefox/
dbus-glib.tcz
gtk2.tcz
gtk3.tcz
libasound.tcz
tc@box:~$ ./dep_check -c /tmp/tcloop/alsa
These dependencies are missing
libsamplerate.so.0
libasound.tcz
ncurses.tcz
tc@box:~$ tce-load -si libsamplerate
tc@box:~$ ./dep_check -c /tmp/tcloop/alsa
libasound.tcz
libsamplerate.tcz
ncurses.tcz
tc@box:~$ ./dep_check -c /tmp/tcloop/alsa alsa-modules-KERNEL input-joystick-4.2.9-tinycore.tcz
alsa-modules-KERNEL.tcz
libasound.tcz
libsamplerate.tcz
ncurses.tcz
Title: Re: A script to check dependencies
Post by: coreplayer2 on January 07, 2017, 08:40:12 AM
This is a great idea thanks for the script.

However...  At the risk of sounding dumb,   I'm having difficulty getting past the help menu.  Actually I haven't successfully made it passed the help menu.  There is always some kind of error.

Code: [Select]
/mnt/sda2/tc7-x86/optional $ getDepList
Usage: getDepList [-option] your_top_directory [append your list here ...]
  Available Options (Default -ns)
  -c colorful output (same as -cns)
  -d show only detected tczs, without custom extensions (cyan)
  -f show full detected dependencies with ldd (white)
  -h show this helpinfo (yellow)
  -n show dependencies that are not found (red)
  -r show recursively scanned tczs (full)
  -s show detected tczs (short)
/mnt/sda2/tc7-x86/optional $ getDepList -f *
/usr/local/bin/getDepList: cd: line 120: can't cd to NAS-Connect.tcz
Usage: getDepList [-option] your_top_directory [append your list here ...]
  Available Options (Default -ns)
  -c colorful output (same as -cns)
  -d show only detected tczs, without custom extensions (cyan)
  -f show full detected dependencies with ldd (white)
  -h show this helpinfo (yellow)
  -n show dependencies that are not found (red)
  -r show recursively scanned tczs (full)
  -s show detected tczs (short)
/mnt/sda2/tc7-x86/optional $ getDepList -f /tmp/tcloop/*
WARNING! copy2fs.lst found in /mnt/sda2/tc7-x86, please disable it and reboot
/mnt/sda2/tc7-x86/optional $ getDepList -f /mnt/sda2/tc7-x86/optional/*
/usr/local/bin/getDepList: cd: line 120: can't cd to /mnt/sda2/tc7-x86/optional/NAS-Connect.tcz
Usage: getDepList [-option] your_top_directory [append your list here ...]
  Available Options (Default -ns)
  -c colorful output (same as -cns)
  -d show only detected tczs, without custom extensions (cyan)
  -f show full detected dependencies with ldd (white)
  -h show this helpinfo (yellow)
  -n show dependencies that are not found (red)
  -r show recursively scanned tczs (full)
  -s show detected tczs (short)
/mnt/sda2/tc7-x86/optional $ getDepList -f *
/usr/local/bin/getDepList: cd: line 120: can't cd to NAS-Connect.tcz
Usage: getDepList [-option] your_top_directory [append your list here ...]
  Available Options (Default -ns)
  -c colorful output (same as -cns)
  -d show only detected tczs, without custom extensions (cyan)
  -f show full detected dependencies with ldd (white)
  -h show this helpinfo (yellow)
  -n show dependencies that are not found (red)
  -r show recursively scanned tczs (full)
  -s show detected tczs (short)
/mnt/sda2/tc7-x86/optional $ getDepList -r /tmp/tcloop/*
Xdialog.tcz
Xlibs.tcz
...
xz.tcz
zsync.tcz
/usr/local/bin/getDepList: line 204: Feedbacks: not found
/mnt/sda2/tc7-x86/optional $ getDepList -f /tmp/tcloop/*

/usr/local/bin/getDepList: line 204: Feedbacks: not found
/mnt/sda2/tc7-x86/optional $
/mnt/sda2/tc7-x86/optional/oss $ getDepList -r *
usr.tcz not found or size is zero
If you load it elsewhere, please move it and the dep file into your
/mnt/sda2/tc7-x86/optional/
/tmp/tcloop/firefox/usr/local/firefox $ getDepList -f *
/usr/local/bin/getDepList: cd: line 120: can't cd to Throbber-small.gif
Usage: getDepList [-option] your_top_directory [append your list here ...]
  Available Options (Default -ns)
  -c colorful output (same as -cns)
  -d show only detected tczs, without custom extensions (cyan)
  -f show full detected dependencies with ldd (white)
  -h show this helpinfo (yellow)
  -n show dependencies that are not found (red)
  -r show recursively scanned tczs (full)
  -s show detected tczs (short)
/tmp/tcloop/firefox/usr/local/firefox $
/tmp/tcloop/firefox/usr/local/firefox $ getDepList -f ./firefox
/usr/local/bin/getDepList: cd: line 120: can't cd to ./firefox
Usage: getDepList [-option] your_top_directory [append your list here ...]
  Available Options (Default -ns)
  -c colorful output (same as -cns)
  -d show only detected tczs, without custom extensions (cyan)
  -f show full detected dependencies with ldd (white)
  -h show this helpinfo (yellow)
  -n show dependencies that are not found (red)
  -r show recursively scanned tczs (full)
  -s show detected tczs (short)
/tmp/tcloop/firefox/usr/local/firefox $

closest I've been to success but still not without error is:
Code: [Select]
/tmp/tcloop/firefox/usr/local/firefox $ getDepList /tmp/tcloop/firefox/
dbus-glib.tcz
gtk3.tcz
libasound.tcz
nvidia-352.63-KERNEL.tcz
/usr/local/bin/getDepList: line 204: Feedbacks: not found
/tmp/tcloop/firefox/usr/local/firefox $

Ok lets examine this.
yet firefox will not run without these deps loaded
cairo.tcz
gamin.tcz

The only list I've been able to show is what looks like a partial list of extensions in my tcz store.
So far I've not been able to show a list of missing (or otherwise) deps of any installed extensions

Maybe I misunderstand the intention of the script??
Maybe I misunderstand the usage?

Speaking of usage,
Code: [Select]
Usage: getDepList [-option] your_top_directory [append your list here ...]Why not set a default option if no option is given?
The default directory appears to be /tmp/tcloop where all extensions are loop mounted too, in which case why not use it as the default if none is specified?
When you say "your_top_directory" you mean " / "  ?
When you say "[append your list here ...]" please explain, list of what?  Extensions? executables ?



To sum up, the usage isn't exactly intuitive and I receive an error indicating how the usage is incorrect at every attempt without a correct usage example.

AIUI the script will check for missing dependencies.  However, if deps are missing the extension will refuse to load.  so how exactly am I supposed to load the extension..?  A catch 22?

Maybe I'm doing this all wrong?
Title: Re: A script to check dependencies
Post by: polikuo on January 07, 2017, 11:11:37 AM
To sum up, the usage isn't exactly intuitive and I receive an error indicating how the usage is incorrect at every attempt without a correct usage example.

Sorry about that, not exactly an English speaker here.  :-[
I'll try to re-explain my idea.

Let's say today I get bored, and feel like making a new extension. (for instance: ibus.tcz)

First, we compile it from source.
./configure ~~ make ~~ make install DESTDIR=/tmp/ibus-base

After all the stripping, we end up with a "top_directory" (/tmp/ibus-base),
which contains some libraries and some python scripts, make sense  :D

The main idea of the script is to find all the "shared libraries" required by the program.
To do that with my script, just type /path/to/the/script /tmp/ibus-base
Code: [Select]
tc@box:~/ $ get-dep-list /tmp/ibus-base
dconf.tcz
libnotify.tcz
You can have a colorful output by passing "-c", by the way.  :)

My script first validate that the first argument it gets is a directory, and cd to it.
No, I never use /tmp/tcloop as the default directory.
There never was a "default directory" to begin with.  ::)

That's why you get  :-\
/mnt/sda2/tc7-x86/optional $ getDepList -f *
/usr/local/bin/getDepList: cd: line 120: can't cd to NAS-Connect.tcz
/mnt/sda2/tc7-x86/optional $ getDepList -f /mnt/sda2/tc7-x86/optional/*
/usr/local/bin/getDepList: cd: line 120: can't cd to /mnt/sda2/tc7-x86/optional/NAS-Connect.tcz
/tmp/tcloop/firefox/usr/local/firefox $ getDepList -f *
/usr/local/bin/getDepList: cd: line 120: can't cd to Throbber-small.gif
/tmp/tcloop/firefox/usr/local/firefox $ getDepList -f ./firefox
/usr/local/bin/getDepList: cd: line 120: can't cd to ./firefox
Apparently you don't have a directory named "firefox" in your "/tmp/tcloop/firefox/usr/local/firefox"  ;)

In order to scan all the "shared libraries", I use "find -exec ldd {} +" to do the trick.
Thanks andyj who point out that
1. Not all dependencies are libraries, just as not all extensions are libraries. Be careful when you say extensions seem to have unneeded dependecies.
2. Watch out for circular (mutual) dependencies. Freetype <-> harfbuzz and cyrus-sasl <-> openldap are two that I know of off the top of my head.
3. Many dependencies are optional. This is particularly true for programs that use plugins / extensions / modules like Apache, PHP, LDAP, rsyslog, Postgresql, etc.
4. This requires that copy2fs.flg or copy2fs.lst not be used. Might be a problem in some cases.

Due to the copy2fs feature, the contents in a extension get copied to the RAM.
That extension will be dismissed because I check /tmp/tcloop/extension_name to figure out which tcz contains the library.
That's why you get  :-\
WARNING! copy2fs.lst found in /mnt/sda2/tc7-x86, please disable it and reboot

Since ibus depends on iso-codes and python, I need to add that to the "list-to-scan".  8)
I can append the list by typing /path/to/the/script /tmp/ibus-base iso-codes python (with or without ".tcz")
Code: [Select]
tc@box:~/ $ get-dep-list /tmp/ibus-base iso-codes python
dconf.tcz
iso-codes.tcz
libnotify.tcz
python.tcz
Actually I'm considering merging ibus.tcz and ibus-gir.tcz since they depends on each other.

I can redirect the list to a file and move on packing the extension.
Code: [Select]
tc@box:~/ $ get-dep-list /tmp/ibus-base iso-codes python > /tmp/ibus.tcz.dep

In your case, you should do /path/to/the/script /tmp/tcloop/firefox cairo.tcz gamin.tcz hicolor-icon-theme.tcz
Or you can be lazy and do /path/to/the/script /tmp/tcloop/firefox `cat firefox.dep`
(When I do this test, I boot with bootcode "base", and then only load firfox)
Code: [Select]
These dependencies are missing
libXt.so.6
libgdk-x11-2.0.so.0
libgtk-x11-2.0.so.0
dbus-glib.tcz
gamin.tcz
gtk3.tcz
hicolor-icon-theme.tcz
libasound.tcz

In fact, my script also suggest adding gtk2.tcz to the dep file.
Althought, we all know that firefox goes well without it  :P
Anyway~, after loading gtk2 you'll see
Code: [Select]
dbus-glib.tcz
gamin.tcz
gtk2.tcz
gtk3.tcz
hicolor-icon-theme.tcz
libasound.tcz
cairo.tcz is already a dependence of gtk3.tcz, that's why it's not on the list.
If you expand the dependence tree, you'll see it.  ;)
My script take care of that by looping (not a good idea for big extensions).  :'(

As for nvidia-352.63-KERNEL.tcz (renamed cause the script, but tce-load can detect that as well)  :D
Somehow ldd links these files to /usr/lib/libnvidia-glsi.so.352.63 (I knew that long ago, just couldn't figure out why) ???
  crashreporter
  libmozgk.so
  libxul.so
  updater

Finally, I have absolutely no clue where does the "Feedbacks" comes from, I never put such string in it.  :o
/mnt/sda2/tc7-x86/optional $ getDepList -f /tmp/tcloop/*
/usr/local/bin/getDepList: line 204: Feedbacks: not found
/tmp/tcloop/firefox/usr/local/firefox $ getDepList /tmp/tcloop/firefox/
dbus-glib.tcz
gtk3.tcz
libasound.tcz
nvidia-352.63-KERNEL.tcz
/usr/local/bin/getDepList: line 204: Feedbacks: not found

I hope that answers most of the questions, let me know if I miss something.  8)
Title: Re: A script to check dependencies
Post by: coreplayer2 on January 07, 2017, 03:28:02 PM
Thanks polikuo for the info, I'm getting used to this script.   
Although am sure I'll have forgot by next time, which is why I like default settings
Another reason for defaults in usage is that it makes you feel comfortable when there's no immediate error.
I hate a script/program which I initiated telling me I'm stupid for not providing any argument, even if I am!! 
I'd much prefer to have a default option, then if the output needs tweaking I know I can run -h option to modify the results.

FYI 
Code: [Select]
tc@box:/tmp/tcloop/firefox/usr/local/firefox$ getDepList -n .*
These dependencies are missing
libgdk-x11-2.0.so.0
libgtk-x11-2.0.so.0
/usr/local/bin/getDepList: line 204: Feedbacks: not found
tc@box:/tmp/tcloop/firefox/usr/local/firefox$
I'm guessing this is one of the limitations of the script. 
Because latest firefox is gtk3 now.
Code: [Select]
tc@box:/tmp/tcloop/firefox/usr/local/firefox$ ldd libmozgtk.so | grep 'gtk'
        libgtk-3.so.0 => /usr/local/lib/libgtk-3.so.0 (0xb728a000)
tc@box:/tmp/tcloop/firefox/usr/local/firefox$

AIUI firefox is built with lib's offering backwards compatibly for gtk2 plugins (specifically flashplayer) 
gtk2 is a dep of flash.tcz and is therefore not required to be loaded by firefox.


I also found that
Code: [Select]
/usr/local/bin/getDepList: line 204: Feedbacks: not foundcomes from libxul.so (a firefox lib)
another erroneous item to disregard..

The script attempts to process non related files like *.pdf and some text files which throws errors..

Title: Re: A script to check dependencies
Post by: polikuo on January 07, 2017, 07:55:27 PM
Hi coreplayer2
/usr/local/bin/getDepList: line 204: Feedbacks: not found
I can't replicate it, which version of firefox your're using ?  ???
I use getfirefox_Latest.sh on both 32&64 platform, bootcode lang=zh_TW, version 50.1.0

The script attempts to process non related files like *.pdf and some text files which throws errors..
In my script, I have
Code: [Select]
LDD_OUT="$(find . -not -type d -exec ldd {} + 2> /dev/null)"That shouldn't throw any error messages.
I can't replicate the "Feebacks error" and fix the script.  :P

P.S.
Don't use wildcard character (*) for directory.
If you create a script getARGS with 'echo "$@"' in it, you'll see:
Code: [Select]
/tmp/tcloop/firefox/usr/local/firefox$ /tmp/getARGS .*
. ..
What will happen here is that you'll cd to . (i.e. /tmp/tcloop/firefox/usr/local/firefox)
Anything after the first ARG, the script will assume it's an tcz extension.
Usually, you'll get "...tcz not found" message, but since you pass "-n" to the script,
it just ignore all the none-necessary looping and print out:
Code: [Select]
These dependencies are missing
libgdk-x11-2.0.so.0
libgtk-x11-2.0.so.0
Title: A script to check dependencies
Post by: coreplayer2 on January 07, 2017, 08:04:40 PM
Ah ha!! It just clicked

The path to file under test ends in a directory, not a file !!

I knew I was doing this all wrong



Sent from my iPhone using Tapatalk
Title: Re: A script to check dependencies
Post by: polikuo on January 07, 2017, 08:13:34 PM
Ah ha!! It just clicked

The path to file under test ends in a directory, not a file !!

I knew I was doing this all wrong



Sent from my iPhone using Tapatalk

Glad that we're on the same wavelength  :)
Title: Re: A script to check dependencies
Post by: coreplayer2 on January 07, 2017, 08:51:44 PM
Well...   whether I'll be able to change habits is another matter..

Code: [Select]
/tmp/tcloop/alsa $ getDepList -n $PWD
These dependencies are missing
libsamplerate.so.0
/usr/local/bin/getDepList: line 204: Feedbacks: not found
/tmp/tcloop/alsa $

yet the "Feedbacks" is still present
:p
Title: Re: A script to check dependencies
Post by: polikuo on January 07, 2017, 09:04:05 PM
Well...   whether I'll be able to change habits is another matter..

Code: [Select]
/tmp/tcloop/alsa $ getDepList -n $PWD
These dependencies are missing
libsamplerate.so.0
/usr/local/bin/getDepList: line 204: Feedbacks: not found
/tmp/tcloop/alsa $

yet the "Feedbacks" is still present
:p

My script only has 201 lines
Did you accidentally copied "Feedbacks are welcomed  :D" from my previous post ?
If that's the case, delete it ~
Title: Re: A script to check dependencies
Post by: coreplayer2 on January 07, 2017, 09:15:41 PM
That is it exactly, and there is no "exit"
Title: Re: A script to check dependencies
Post by: Rich on January 07, 2017, 09:26:09 PM
Hi coreplayer2
That's what the  [Select]  to the right of where it says  Code:  is for. Click the  [Select]  and it highlights what's inside the
code tags. Then just  Ctrl-C  to copy it to the clipboard.
Title: Re: A script to check dependencies
Post by: coreplayer2 on January 07, 2017, 10:26:09 PM
OkI think we have figured this out

Code: [Select]
tc@box:~$ depList -n /tmp/tcloop/alsa
These dependencies are missing
libsamplerate.so.0
tc@box:~$

replaced FeedBacks with a successful exit status
Code: [Select]
exit 0
The more I look at this, the more I'm convinced the default arguments if none given should be -cdn
Code: [Select]
tc@box:~$ depList -cdn /tmp/tcloop/alsa
libasound.tcz
ncurses.tcz
These dependencies are missing
libsamplerate.so.0
tc@box:~$

For me, I find typing the path to top level directory for the build goes against the grain.  It's more intuitive and natural to enter the directory in question then run the script.

this is considerably harder
Code: [Select]
tc@box:~$ depList -cdn /tmp/tcloop/alsa
libasound.tcz
ncurses.tcz
These dependencies are missing
libsamplerate.so.0
tc@box:~$

than
Code: [Select]
/tmp/tcloop/alsa $ depList -cdn *
libasound.tcz
ncurses.tcz
These dependencies are missing
libsamplerate.so.0
/tmp/tcloop/alsa $

The latter works fine however :)
Title: Re: A script to check dependencies
Post by: coreplayer2 on January 07, 2017, 10:27:46 PM
Hi coreplayer2
That's what the  [Select]  to the right of where it says  Code:  is for. Click the  [Select]  and it highlights what's inside the
code tags. Then just  Ctrl-C  to copy it to the clipboard.
Thanks,  we learn something new every day :)
Title: Re: A script to check dependencies
Post by: polikuo on January 07, 2017, 10:58:06 PM
The more I look at this, the more I'm convinced the default arguments if none given should be -cdn
Actually, that's how my script look like at the very beginning, but bmarkus pointed out that
Using > formatting color codes are saved too.
It is better to make it by default and use a -c option to create a colored list if someone need it.
That's why the -c option is no longer one of the default arguments.  :P

Code: [Select]
/tmp/tcloop/alsa $ depList -cdn *
libasound.tcz
ncurses.tcz
These dependencies are missing
libsamplerate.so.0
/tmp/tcloop/alsa $

The latter works fine however :)
That's because most packages only has "usr"  under it's "root directory",
if it has more then one directory (such as etc), you'll cd to etc and get "usr.tcz not found" error message.

Anyway~~
Feel free to modify it to suit your need  ;)
This script is made to help package maintainers to figure out what dependencies are required, and remove all the redundant ones.
(such as cairo.tcz, already a dependence of gtk3.tcz)
Title: Re: A script to check dependencies
Post by: coreplayer2 on January 07, 2017, 11:17:30 PM
Thanks for the insight.

I was just admiring your work and it seemed a pity to modify it just to suite little old me.   But my systems have such long paths to their optional directories  where I build extensions,  it would be near impossible or tedious to say the least to type in the full path every time (even with tab completion).  Even more difficult to remember I need to type the full path each time  :p
Once configure and make has finished the project, I load up a desktop environment asap.


I already created an extension so that we need only enter the directory then run it.   Is an easy fix to add default arguments and use current path, I'll look at that..

thanks for the script, it's really helpful
Title: Re: A script to check dependencies
Post by: Juanito on January 08, 2017, 10:44:26 PM
Did you accidentally add/delete anything by chance ?

Hmm - it seems the problem is with fifth, if use opera-9 to copy and paste it works.
Title: Re: A script to check dependencies
Post by: andyj on January 09, 2017, 03:44:43 AM
It would be an actual step in the "grand unified extension build script" dream if we had an extension checker script we could all get behind. We have two kinds of extension dependency problems, those with direct dependencies that we can "measure" using ldd, and those we just have to know about like bash or tzdata or gnome-icon-theme and the like. I like others have a script to solve the first problem. I don't find the '-f' option in the script which is the topic of this thread of much use, it needs to be distilled. To that end here's my script if you want to incorporate it. Or someone else's like it. Or not. It's more in the busybox style, in that you need to create symlinks to get the different functionality. It uses shell recursion to walk the library dependency tree, so it's kinda slow, and painfully so with X/GTK apps. But it seems to give the right answers so I keep using it. The names of the symlinks correspond to the choices in the main case block.

Code: [Select]
#!/bin/sh

list_libs(){
ldd "$2" | grep '=>' | while read SLIB SEP DLIB ADDR; do
if [ "$DLIB $ADDR" = "not found" ] ; then
echo -e "\\n***ERROR*** $SLIB not found ***\\n"
else
DEPTH=$(($1 + 1))
case "$DLIB" in
*/local/* ) EXT=$(readlink -f "$DLIB" | cut -d/ -f4) ;;
* ) EXT="xxx${DEPTH}" ;;
esac
eval "qq$DEPTH=$EXT"
CIRC=0
for LEVEL in $(seq $1); do
eval "DEP=\$qq$LEVEL"
[ "$EXT" = "$DEP" ] && CIRC=$LEVEL
echo -e -n \\t
done
echo -n "$DEPTH ==>> $DLIB"
case "$ELF" in
*"/$EXT/"* ) echo " *** $EXT SELF-REFERENCING ***"
continue ;;
* ) [ "$EXT" != "xxx$DEPTH" -a $CIRC -eq 0 ] && echo -n "  extension: $EXT.tcz" ;;
esac
echo
if [ $CIRC -gt 0 ] ; then
if [ "$EXT" != "$DEP" ] ; then
echo -e "\\n*** CIRCULAR REFERENCES FOUND BETWEEN $EXT and $DEP ***\\n"
return
fi
else
list_libs $DEPTH "$DLIB"
fi
fi
done
}

ME=${0##*/}
DEPTH=0
for ELF in $@; do
echo $ELF
case $ME in
lib-tree) list_libs 0 "$ELF" ;;
lib-deps) list_libs 0 "$ELF" | grep extension | cut -d\  -f6 | sort | uniq -c | sort -n ;;
lib-tcz)  for TLIB in $(find "$ELF" -type f)
        do file -b "$TLIB" | grep -q '^ELF ' && list_libs 0 "$TLIB"
  done | tee /tmp/lib-tcz.out | awk '/  extension: / {printf "%03i %s\n", $1,$5}' \
| sort -u | sort -k2 | uniq -f1 -u | awk '/^001 / {print $2}' ;;
esac
echo -e \\n
done
Title: Re: A script to check dependencies
Post by: polikuo on January 09, 2017, 06:56:50 AM
I don't find the '-f' option in the script which is the topic of this thread of much use, it needs to be distilled.
Personally, I find it helpful remastering an iso image when copying files, especially when you need some custom packages.
I can trick the script by myscript -f /path/to/an/empty/directory `cat list-of-wanted-packages`, and get a full list.

To that end here's my script if you want to incorporate it.
Thanks,  we learn something new every day :)
It's more in the busybox style, in that you need to create symlinks to get the different functionality.
I learn something new today, I never knew "while" and "read" can be combined and have several variables.  :D
Also, a script called from different symlinks can have different result, knowledge acquired! It does share a similarity with busybox.  ;D
Your script is amazing, I'll need some time to digest that.  :)
Thanks for the script, will work on it when I have time for it.  ;)
Life gets in the way.  :P

Oh, before I go, I notice that you use "readlink" for the linking.
Personally I prefer using "realpath" instead, cause some packages might have libraries that link to each other by tce.installed script.
(Haven't seen one yet, but just in case)

For instance:
/usr/local/lib/libA.so         -->   /tmp/tcloop/libA/usr/local/lib/libA.so
/usr/local/lib/libA.so.1      -->   /usr/local/lib/libA.so

readlink /usr/local/lib/libA.so
/tmp/tcloop/libA/usr/local/lib/libA.so   -->   Good, that's what we want.
readlink /usr/local/lib/libA.so.1
/usr/local/lib/libA.so                           -->   OMG, I need to read again ??
realpath /usr/local/lib/libA.so.1
/tmp/tcloop/libA/usr/local/lib/libA.so   -->   Straight to the target.  :D
Title: Re: A script to check dependencies
Post by: coreplayer2 on January 09, 2017, 07:41:44 AM
Personally, I find it helpful remastering an iso image when copying files, especially when you need some custom packages.
I can trick the script by myscript -f /path/to/an/empty/directory `cat list-of-wanted-packages`, and get a full list.
If you remaster, whether a one off or multiple repetitive ISO's; with or without custom extensions, then I urge you to look at ezremaster.tcz 
:)
Title: Re: A script to check dependencies
Post by: polikuo on January 09, 2017, 08:16:46 AM
If you remaster, whether a one off or multiple repetitive ISO's; with or without custom extensions, then I urge you to look at ezremaster.tcz 
:)

Thanks, I do know it, I just prefer remastering manually.
It provides the most flexibility.  ;)
Title: Re: A script to check dependencies
Post by: andyj on January 09, 2017, 08:34:24 AM
The -f option to readlink is important. That should get you to the actual file.

For checking whether or not I have all the required extensions (unresolved libraries) loaded (in an ESXi VM in my case) I use this:

Code: [Select]
#!/bin/sh
#
for a in $(find /tmp/tcloop -type f)
        do file -b $a | grep -q '^ELF ' && ldd $a | grep -q found && echo $a
done

To find out if I have extensions loaded I don't need (unreferenced extensions) I use this:

Code: [Select]
#!/bin/sh
#
{ tce-status -i ;
for b in $(for a in $(find /tmp/tcloop -type f)
        do file -b $a | grep -q '^ELF ' && ldd $a | grep /local/ | cut -d\  -f3 | cut -d/ -f5
        done | sort -u)
        do find /tmp/tcloop -name $b | cut -d/ -f4
done | sort -u ; } | sort | uniq -u

Again, I use these for jogging my memory. They are by no means definitive.
Title: Re: A script to check dependencies
Post by: patrikg on January 23, 2017, 11:11:21 AM
I could be cool if all script's that getting the dep, also make a graphviz dot file.
To visualize the dependency chart map.
Title: Re: A script to check dependencies
Post by: andyj on January 23, 2017, 08:34:39 PM
The first script I posted will do a library dependency tree if called as lib-tree. If you want a .tcz dependency tree you could just grep the output for tcz. Not sure why though, except that it's more educational if not particularly useful.
Title: Re: A script to check dependencies
Post by: polikuo on June 27, 2018, 08:24:35 AM
I have refactored the script for performance boost and added some new functions.

The latest script can be found here (https://raw.githubusercontent.com/polikuo/practice/master/get-dep-list).

The purpose of this script is to help package maintainers to find out which extensions are needed under a certain directory.
For instance, I compile openbox and make install to /tmp/openbox-root.
Then, run the script as below.
Code: [Select]
tc@box:/tmp $ get-dep-list /tmp/openbox-root
WARNING! python script detected
imlib2.tcz
libICE.tcz
libSM.tcz
libXcursor.tcz
libXinerama.tcz
libXrandr.tcz
librsvg.tcz
As you can see, the deps are printed, you can redirect it to openbox.tcz.dep
Since /usr/local/libexec/openbox-xdg-autostart is a python script, you'll get a warning.
However, python.tcz is not appended automatically in the list because usually these scripts are optional functions.
To append it, just run the script this way: "get-dep-list /tmp/openbox-root python"

The new functions are shown below:
Quote
Usage: get-dep-list [-option] directory [append your list here ...]
Available Options (default -ns)
  -c     colorful output
  -d     show only the detected tczs, without custom extensions (blue)
  -g     graph .tcz.tree (magenta)
  -h     show this helpinfo (yellow)
  -n     show dependencies that can not be found (red)
  -p     scan provides.db for dependencies
  -r     show recursively scanned tczs (green)
  -s     show shortened tczs (cyan)
  -t DIR set tcedir to DIR (default /etc/sysconfig/tcedir)
The "-d" function prints only the "ldd-matched" tczs.
The "-g" function prints the dep tree of the shortened tczs.
The "-p" function use provides.db as reference if there are matched missing dependencies.
The "-t" function switch you tcedir to another directory.
All the other functions are inherited from previous versions.

P.S. I think this thread should be moved to "Programming & Scripting - Unofficial".