WelcomeWelcome | FAQFAQ | DownloadsDownloads | WikiWiki

Author Topic: depfinder_sorted: script to find dependencies, and print sorted output  (Read 2366 times)

Offline halma

  • Full Member
  • ***
  • Posts: 164
Hi all,

i spend some time to write this Code for playing some around with  Tinycore, with this Code it's possible to find the all dependencies, (the location of the extension must be in the tinycore default path), of a given extension like "Bash" and it will print them out sorted as its need to mount in the right order or tinycore not support, unmount.
this Script will print out:
Code: [Select]
./f_find_dependencies_sorted.sh bash
Quote
bash readline ncurses

As the output shows, bash needs "readline" and "ncurses", the last word is the first extention to mount, example: mount ncurses, then mount readline, then mount bash, to get bash working as example.

This script needs bash installed.
Code: [Select]
#!/bin/bash

#************************************************#
#        f_find_dependencies_sorted.sh           #
#               written by Halma                #
#                July 20, 2017                   #
#                                                #
#           Find dependencies of extension      #
#           and print them out sorted           #
#************************************************#

#TinyCoreLinux EXTENSIONSARRAY Folder
TINYCOREEXTENSIONSFOLDER=/etc/sysconfig/tcedir/optional

#arg check
package=$1
if [ -z "$package" ]; then
  echo "[Error]: No package name specified"
  exit 1
fi

# --------------------------------------------------------- #
# f_finddepfile()    #
# this function will try to find dep files of extension    #
# --------------------------------------------------------- #
#function
f_finddepfile() {

var_path_to_file=""
var_depfile_found=false

if [ -f $TINYCOREEXTENSIONSFOLDER/$var_extension.tcz.dep ]; then
var_path_to_file=$TINYCOREEXTENSIONSFOLDER
var_depfile_found=true
fi
}

# --------------------------------------------------------- #
# f_sortoutput()    #
# this function will sort the dependencies                 #
# in a sorted row as needed to load                        #
# --------------------------------------------------------- #
#function
f_sortoutput() {

List="${HELPERARRAY[@]}"
NewList=""

#run till List is zero
while [ $(echo "${#List}") -gt 1 ] ; do

leftvaluefound=false
rightvaluefound=false

#get first column
firstcolumn=$(echo "$List" | awk '{print $1}')

#split colum
leftvalue="${firstcolumn%_@_*}"
rightvalue="${firstcolumn#*_@_}"

#add $leftvalue to NewList if not exists
if !  $(echo "$NewList" | grep "$leftvalue" > /dev/null) ; then
#prevent whitespace on adding at starting of a NewList
if [ $( echo "$NewList" == "" ) ]; then
#echo "add1 leftvalue $leftvalue"
NewList="$NewList$leftvalue" #remove whitespace
#set true if added to NewList
leftvaluefound=true
else
#echo "add1 leftvalue $leftvalue"
NewList="$NewList $leftvalue"
#set true if added to NewList
leftvaluefound=true
fi
fi

#add $rightvalue to NewList if not exists
if ! echo "$NewList" | grep "$rightvalue" >/dev/null ; then
#echo "add2 rightvalue $leftvalue"
NewList="$NewList $rightvalue"
#set true if added to NewList
rightvaluefound=true
#delete first field from $List to prevent double matching #To remove all whitespace (including tabs) from left to first word, enter: sed -e 's/^[ \t]*//'
List=$(echo "$List" | awk '{$1 = ""; print $0}' | sed -e 's/^[ \t]*//')

#search for more Strings in $List containing "$rightvalue"
for i in $List; do
if [ "${i%_@_*}" == "$rightvalue" ]; then
#add ${i#*_@_} to NewList
NewList="$NewList ${i#*_@_}"
#delete field from $List to prevent double matching
List=$(echo "$List" | sed 's/'$i'//')
fi
done
fi

#if both left and right values allready exists in the Newlist delete from List to prevent looping
if [ "$leftvaluefound" == false ] && [ "$rightvaluefound" == false ]; then
List=$(echo "$List" | awk '{$1 = ""; print $0}' | sed -e 's/^[ \t]*//')
fi
done

echo "$NewList"

}

# --------------------------------------------------------- #
# f_find_dependencies()    #
# this function will find all dependencies                 #
# of a given extension                         #
# --------------------------------------------------------- #
#function
function f_find_dependencies () {

declare -a EXTENSIONSARRAY  # indexed array for the EXTENSIONSARRAY
declare -a HELPERARRAY    # indexed HELPERARRAY array for the EXTENSIONSARRAY

EXTENSIONSARRAY=( "$1" )
HELPERARRAY=()
#echo "$EXTENSIONSARRAY"

while [ $(echo ${#EXTENSIONSARRAY[@]}) -gt 0 ] ; do
for var_extension in ${EXTENSIONSARRAY[@]}; do

#search for a depfile
f_finddepfile

#
#--- if a dep file was found
#

if [ "$var_depfile_found" == "true" ]; then

#case
case "$var_path_to_file " in
/*)
for var_dep_file in $var_extension; do

while read line ; do
#check for empty line
[ -z "$line" ] && continue
#add extension to EXTENSIONSARRAYarray
#echo "add depencie: ${line%%.*}"
EXTENSIONSARRAY+=(${line%%.*})
HELPERARRAY+="${var_extension}_@_${line%%.*} "
done < $var_path_to_file/$var_dep_file.tcz.dep;

#remove current array-element from array EXTENSIONSARRAY[0] #http://stackoverflow.com/a/34199081
EXTENSIONSARRAY_TEMP=("${EXTENSIONSARRAY[0]}")
result=("${EXTENSIONSARRAY[@]}")

#remove array element [0]
for element in "${EXTENSIONSARRAY_TEMP[@]}"; do
result=(${result[@]/*${element}*/}) #/*${element}*/
done

#(re)set the array EXTENSIONSARRAY=( "${result[@]}" )
EXTENSIONSARRAY=( "${result[@]}" )

done;;
esac

fi

#
#--- if no dep file was found
#
if [ "$var_depfile_found" == "false" ]; then

#remove current array-element from array EXTENSIONSARRAY[0] #http://stackoverflow.com/a/34199081
EXTENSIONSARRAY_TEMP=("${EXTENSIONSARRAY[0]}")
result=("${EXTENSIONSARRAY[@]}")

#remove array element [0]
for element in "${EXTENSIONSARRAY_TEMP[@]}"; do
result=(${result[@]/*${element}*/}) #/*${element}*/
done

#(re)set the array EXTENSIONSARRAY=( "${result[@]}" ) #if just one array element is left, reset the array =()
if [ $( echo "${#EXTENSIONSARRAY[@]}") -eq 1 ]; then
EXTENSIONSARRAY=()
else
EXTENSIONSARRAY=( "${result[@]}" )
fi

fi
done #end for var_extension in ${EXTENSIONSARRAY[@]}; do
done

#
#--- Done, print out
#
if [ ! ${#HELPERARRAY[@]} -eq 0 ]; then

#sort the output
f_sortoutput
fi
}

# run the script with the given arg=extension
f_find_dependencies "$1"

usage of your own risk, no warranty for this code.
any suggestions are welcome  :)

please use the second attachment, i dont know how to delete the first attachment, if an mod can do delete the first attachment, it would be nice
thx
« Last Edit: January 20, 2017, 01:36:01 PM by Rich »
1 + 2 = 6  cause  10 - 6 = 78 ;-) lol

Offline polikuo

  • Hero Member
  • *****
  • Posts: 714
Re: depfinder_sorted: script to find dependencies, and print sorted output
« Reply #1 on: January 21, 2017, 06:39:00 AM »
If I understand you correctly, you're trying to "recursively scan all deps" of a given extension.
You can find the "recursive_scan_dep()" function inside the "tce-load" script.  :P

No hard feeling, after copying your script, I test it with "compiletc" and see how fast does it goes.
Code: [Select]
time ./f_find_dependencies_sorted.sh compiletc > /dev/null
real    0m 7.06s
user    0m 2.25s
sys     0m 1.65s
time ./recursive_scan_dep compiletc > /dev/null
real    0m 0.73s
user    0m 0.07s
sys     0m 0.08s
It takes way longer than the one in tce-load.  :(

You'll need some modification to use the fuction thought.
(I just dig out this script written long ago, I don't use it anymore.)
Code: [Select]
#!/bin/sh

SUPPRESS=FALSE
KERNELVER="$(uname -r)"

recursive_scan_dep() {
  echo -e "$@"|awk '
  function recursive_scan(name, optional, mirror, _, depfile, line, i) {
    gsub(/[\t ]+/, "", name)
    if (name) {
      sub(/\-KERNEL\.tcz/, "-"KERNELVER".tcz", name)
      if (name in MARK) {
        if (MARK[name] == 2) {
          if (! SUPPRESS)
          system("echo Warning loop dependency: "name" 1>&2")
        } else {
          RESULT[++IDX]="@#"name
        }
      } else {
        IDX+=1
        RESULT[IDX]=name
        IRANGE[name"#1"]=IDX
        depfile=optional"/"name".dep"
        if (mirror && (system("test ! -f "depfile) == 0 || system("test ! -f "optional"/"name) == 0))
        if (system("rm -f "depfile"; wget -c -P "optional" "mirror"/"name".dep 2>/dev/null") == 0 && ! SUPPRESS)
        system("echo "name".dep OK 1>&2")
        MARK[name]=2
        if (mirror || system("test -f "optional"/"name) == 0) {
          while (getline line < depfile > 0)
          recursive_scan(line, optional, mirror)
          close(depfile)
        }
        MARK[name]=1
        IRANGE[name"#2"]=IDX
      }
    }
  }
  function output(idx1, idx2, _, name, i, refname) {
    for (i=idx2; i>=idx1; i--) {
      name=RESULT[i]
      if (! (name in PRINTED)) {
        PRINTED[name]=1
        if (substr(name, 1, 2) == "@#") {
          refname = substr(name, 3)
          output(IRANGE[refname"#1"]+0, IRANGE[refname"#2"])
        } else {
          print name
        }
      }
    }
  }
  BEGIN {KERNELVER="'"$KERNELVER"'"; SUPPRESS="'"$SUPPRESS"'"; IDX=0;}
  {pi=IDX; recursive_scan($1, $2 ? $2 : ".", $3); output(pi+1, IDX); delete PRINTED;}
  '
}

showhelp() {
  echo 'Usage: '"$(basename $0)" [-option] name1 name2 name3 ... 1>&2
  cat << ENDHELP 1>&2
Options:
  -d <directory>  specify where to scan
  -h              show this help
ENDHELP
}

# main

while getopts "d:h" option; do
  case "$option" in
    d ) DIR=${OPTARG%/};;
    h ) showhelp;exit 1;;
  esac
done

[ -n "$DIR" ] && shift 2
DIR=${DIR:-/etc/sysconfig/tcedir/optional}
TO_SCAN="$@"
cd $DIR || { echo can\'t find directory $DIR 1>&2; exit 1; }

for i in $TO_SCAN; do
  TRIM=${i##*/}
  TRIM=${TRIM%.tcz}
  e=$TRIM".tcz"
  [ ! -e $e ] && { echo file $DIR'/'$e not found 1>&2; exit 1; }
  recursive_scan_dep ${e}
done | sort | uniq

The output style is different from yours, but it does the same job.
This script can be shrunk down if you just want to run it locally.

P.S.
This script depends on busybox only, which means nothing needs to be loaded.

Offline halma

  • Full Member
  • ***
  • Posts: 164
Re: depfinder_sorted: script to find dependencies, and print sorted output
« Reply #2 on: January 22, 2017, 05:38:01 AM »
Quote
The output style is different from yours, but it does the same job.
The difference of the output from my Code and the scan_recursive_dep() function is, that the my Code will print out "reverse" the dependencies, how the extensions needed to be mounted(the main extension installed with his dependencies), in the right order, and thats a big difference as an sorted [a-zA-Z] output.

Code: [Select]
./recursive_scan_dep.sh compiletc
binutils.tcz
bison.tcz
cloog.tcz
compiletc.tcz
diffutils.tcz
e2fsprogs_base-dev.tcz
file.tcz
findutils.tcz
flex.tcz
gawk.tcz
gcc.tcz
gcc_base-dev.tcz
gcc_libs-dev.tcz
gcc_libs.tcz
glibc_base-dev.tcz
gmp.tcz
grep.tcz
isl.tcz
linux-3.16.2_api_headers.tcz
m4.tcz
make.tcz
mpc.tcz
mpfr.tcz
ncurses.tcz
patch.tcz
pkg-config.tcz
readline.tcz
sed.tcz
util-linux_base-dev.tcz
zlib_base-dev.tcz
real 0m 0.10s
user 0m 0.00s
sys 0m 0.01s


./f_find_dependencies_sorted.sh compiletc
compiletc bison diffutils file findutils flex gawk mpfr readline gcc binutils cloog gcc_libs-dev mpc grep m4 make patch pkg-config sed e2fsprogs_base-dev glibc_base-dev gcc_base-dev linux-3 util-linux_base-dev zlib_base-dev ncurses isl gmp
real 0m 1.03s
user 0m 0.05s
sys 0m 0.21s

take a look at f_sortoutput(), this function is called as the last function of this code to sort the dependencies as needed to mount in the right order,
if you want only sorted [a-zA-Z] output, just replace line199 (f_sortoutput) with
Code: [Select]
echo "${HELPERARRAY[@]}" | tr " " "\n" | awk NF | awk -F'@_' '{print $2}' | awk '!a[$0]++'then you have mostly the same usage time as the scan_recursive_dep() function if you take care about the time/speed running this code.
« Last Edit: January 22, 2017, 05:48:19 AM by halma »
1 + 2 = 6  cause  10 - 6 = 78 ;-) lol