Tiny Core Linux

General TC => Programming & Scripting - Unofficial => Topic started by: Jason W on July 25, 2009, 07:33:55 AM

Title: extension audit script
Post by: Jason W on July 25, 2009, 07:33:55 AM
I am going to be auditing extensions more closely, and basically using a script such as this one.  It would behoove extension submitters to run this script in the directory where their extension files are before submitting to help ensure the integrity of the repo.  This basically checks the md5s, checks to see if the tce and tcz extensions are valid tar.gz or mountable files, and checks the info files for basic info.  Suggestions for features or improvement are welcome.  I will be adding improvements in time but this I think is a good starting point.  Proper list files are created by this script.

Errors that are caught before submission save a lot of time for everyone compared to those found later, so thanks in advance for checking extensions before submission.   And thanks for the continued hard work that is being put into contributions.

TC version 3 Script attached.

Code: [Select]
#!/bin/sh
# Extension audit script

. /etc/init.d/tc-functions

MIRROR=`cat /opt/tcemirror`
VERSION="3.x"

THISDIR=`pwd`

if [ ! -f /usr/local/tce.installed/file ]; then
echo "${RED}Install the file.tcz extension.${NORMAL}"
exit 1
fi

if [ ! -f /usr/local/tce.installed/diffutils ]; then
echo "${RED}Install the diffutils.tcz extension.${NORMAL}"
exit 1
fi

if [ ! -f /usr/local/tce.installed/squashfs-tools-4.x ]; then
echo "${RED}Install the squashfs-tools extension.${NORMAL}"
exit 1
fi

if [ ! -f /usr/local/tce.installed/wget ]; then
echo "${RED}Install wget extension.${NORMAL}"
exit 1
fi




checkstartup() {
if [ -d "$TMP2"/usr/local/tce.installed/ ]; then
     if [ `ls "$TMP2"/usr/local/tce.installed/ | wc -l` == "1" ]; then

OUT=$(ls -d -l "$TMP2"/usr/local/tce.installed)
      PERM=$(echo $OUT | cut -d" " -f1)
      OWNER=$(echo $OUT | cut -d" " -f3-4)

      [ "$PERM" == "drwxrwxr-x" ] || [ "$PERM" == "drwxrwsr-x" ] || export FILE=1
      [ "$OWNER" == "root staff" ] || export FILE=1

    if [ -f "$TMP2"/usr/local/tce.installed/"$BASENAME" ]; then
if [ ! -x "$TMP2"/usr/local/tce.installed/"$BASENAME" ]; then
export FILE=1
echo "${YELLOW}Will rebuild ${BLUE}"$F"${YELLOW} for wrong /usr/local/tce.installed perms.${NORMAL}"
echo "$F" >> /tmp/submitqc/wrongstartscriptperms
fi
    fi


    if [ ! -z `ls "$TMP2"/usr/local/tce.installed/` ] && [ ! -f "$TMP2"/usr/local/tce.installed/"$BASENAME" ]; then

echo "${YELLOW}Will rebuild ${BLUE}"$F"${YELLOW} for wrong /usr/local/tce.installed script name.${NORMAL}"
echo "$F" >> /tmp/submitqc/wrongstartscriptname
export FILE=1

    fi


if [ "$FILE" == "1" ]; then

rebuild
fi
unset FILE
             else
echo "${BLUE}"$F"${RED} has more than one /usr/local/tce.installed script.  Please remake.${NORMAL}"
unset FILE
     fi


fi
}


checkmaintainer() {
unset NAMES
if  wget -O /tmp/submitqc/."$F".info "$MIRROR""$VERSION"/tcz/"$F".info > /dev/null 2>&1; then
MAINTAINER=`grep "Extension_by" /tmp/submitqc/."$F".info | cut -d: -f2`
SUBMITTER=`grep "Extension_by" "$F".info | cut -d: -f2`
echo "Extension: "$F"" >> /tmp/submitqc/tcemaintainer
echo "Maintainer is "$MAINTAINER"" >> /tmp/submitqc/tcemaintainer
echo "Submitter is "$SUBMITTER"" >> /tmp/submitqc/tcemaintainer
echo " " >> /tmp/submitqc/tcemaintainer
echo " "
echo "${BLUE}"$F"${YELLOW} is maintained by ${GREEN}"$MAINTAINER"${YELLOW}. You are ${GREEN}"$SUBMITTER"${YELLOW}.  Make sure the maintainer is aware you are updating his extension.${NORMAL}"
fi
echo " "
echo "$F" > /tmp/submitqc/.extname

echo "EXT: "$F" - SIMILARLY NAMED EXTENISIONS:" >> /tmp/submitqc/similarextensions
> /tmp/submitqc/.temp
for I in `grep $(basename "$F" .tcz) /tmp/submitqc/.info.lst`; do echo "$I".tcz >> /tmp/submitqc/.temp; done
for I in `cat /tmp/submitqc/.info.lst `; do grep "$I" /tmp/submitqc/.extname >/dev/null && echo "$I".tcz >> /tmp/submitqc/.temp; done

cat /tmp/submitqc/.temp | uniq >> /tmp/submitqc/similarextensions

NAMES=`cat /tmp/submitqc/.temp | uniq`
if [ ! -z "$NAMES" ]; then
   echo "${BLUE}"$F"${YELLOW}  has some similarly named extensions in the repo.  Make sure yours does not \
   collide with these:${NORMAL}"
   echo " "
   echo "${GREEN}"$NAMES"${NORMAL}"
   echo " " >> /tmp/submitqc/similarextensions
   echo " "
fi
}

checkdiff() {
[ -f /tmp/submitqc/listfilediffs/"$F".difflist ] && rm /tmp/submitqc/listfilediffs/"$F".difflist
[ -d /tmp/submitqc/listfilediffs/ ] || mkdir -p /tmp/submitqc/listfilediffs/
#> /tmp/submitqc/listfilediffs/"$F".difflist
if  wget -O /tmp/submitqc/."$F".repolist "$MIRROR""$VERSION"/tcz/"$F".list > /dev/null 2>&1; then
diff -NBa -U 0  /tmp/submitqc/."$F".repolist "$F".list > /tmp/submitqc/listfilediffs/"$F".difflist
 
  if [ -s /tmp/submitqc/listfilediffs/"$F".difflist ]; then
echo " "
sed -i '1,3d' /tmp/submitqc/listfilediffs/"$F".difflist

sed -i '1i ---' /tmp/submitqc/listfilediffs/"$F".difflist
sed -i '1i Minus means file is in the existing repo ext. and not in yours' /tmp/submitqc/listfilediffs/"$F".difflist
sed -i '1i Plus means file is in your ext. and not in repos' /tmp/submitqc/listfilediffs/"$F".difflist
    echo " "
   echo ${BLUE}"$F": "${YELLOW}There are filename differences in your extension and the one\
existing in the repo,they are noted above and stored in ${BLUE}/tmp/submitqc/listfilediffs/"$F".difflist.${NORMAL}"


  fi

 sleep 5
fi
[ -f /tmp/submitqc/."$F".repolist ] && rm /tmp/submitqc/."$F".repolist


}

checkappend() {
if find "$TMP2" -maxdepth 1 | grep "_1"; then
export APPEND=1
fi

if [ "$APPEND" == "1" ]; then
echo "${BLUE}"$F"${RED} has unwanted data appended to it. Please remake.${NORMAL}"
echo "$F" >> /tmp/submitqc/appendeddata
unset APPEND
sleep 3
fi
}

checkblock() {
if ! unsquashfs -s "$F" | grep "Block size" | grep "4096" > /dev/null; then
echo "${BLUE}"$F"${YELLOW} is not 4096 block size.  Rebuilding with -b 4096.${NORMAL}"
rebuild
fi
}


rebuild() {
mkdir "$F".123456789
cp -a "$TMP2"/* "$F".123456789/
chmod 775 "$F".123456789/
#if [ "$BADPERM" == "1" ]; then
#for I in `find "$F".123456789/ -type d`; do stat -c '%a' "$I" | grep 755 >/dev/null || \
#stat -c '%a' "$I" | grep 775 >/dev/null || stat -c '%a' "$I" | grep 2755 >/dev/null || stat -c '%a' "$I" | grep 2775 >/dev/null || chmod 755 "$I" ; done
  #fi

if [ ! -z `ls "$F".123456789/usr/local/tce.installed/` ] && [ ! -f "$F".123456789/usr/local/tce.installed/"$BASENAME" ]; then
   
mv `find "$F".123456789/usr/local/tce.installed/ -not -type d | tail -n 1` "$F".123456789/usr/local/tce.installed/"$BASENAME"
   
fi

if [ -d "$F".123456789/usr/local/tce.installed ]; then
chmod -R 775 "$F".123456789/usr/local/tce.installed
chown -R root:staff "$F".123456789/usr/local/tce.installed
fi
sudo busybox umount -d "$TMP2"

mksquashfs "$F".123456789 "$F".new -b 4096
mv "$F".new "$F"
rm -r "$F".123456789
sudo busybox mount -o loop "$F" "$TMP2" > /dev/null 2>&1
[ -f "$F".md5.txt ] && rm "$F".md5.txt



}

checkbasedirperm() {
if ! su tc -c "ls "$THISDIR"/"$TMP2"" > /dev/null 2>&1; then
echo "${YELLOW}Rebuilding "$F" for unreadable base squashfs directory.${NORMAL}"
rebuild
fi
}


checkdirperms() {
unset BADPERM
for I in `find "$TMP2" -type d`; do
if ! stat -c '%a' "$I" | grep 755 > /dev/null && \
! stat -c '%a' "$I" | grep 775 >/dev/null && ! stat -c '%a' "$I" | grep 2755 >/dev/null && ! stat -c '%a' "$I" | grep 2775 >/dev/null; then
#export BADPERM=1
DIR=`echo "$I" | sed "s:"$TMP2"::"`
echo ""$F":  `stat -c '%a' "$I"`  "$DIR"" >> /tmp/submitqc/baddirperms
echo "${BLUE}"$F": ${GREEN}"$DIR"${YELLOW} directory has suspicious permissionss: ${GREEN}`stat -c '%a' "$I"`${NORMAL}"
fi
done
sleep 3

}


echo_green() {
if [ -f "$F".md5.txt ]; then
echo "${BLUE}"$F"${GREEN} is a valid tcz file. Checking MD5 for ${BLUE}"$F":${NORMAL}"
md5sum -c "$F".md5.txt
if [ ! "$?" == "0" ]; then
echo "${RED}Md5sum failed for "$F"${NORMAL}"
echo "$F" >> /tmp/wrongmd5
fi
else
echo "${BLUE}"$F"${GREEN} is a valid tcz file.  Creating MD5 for ${BLUE}"$F":${NORMAL}"
md5sum "$F" > "$F".md5.txt
fi


cd "$TMP2"
find `ls` -not -type d > ../"$F".list
cd ..

zsyncmake "$F" > /dev/null 2>&1


}





checkdep() {
if `ls *.dep > /dev/null 2>&1`; then
[ "$(ls -A /tmp/submitqc/missingdeps/)" ] && rm /tmp/submitqc/missingdeps/*
 for I in `ls *.dep`; do
for F in `cat "$I"`; do

if echo "$F" | grep "2.6.33.3" > /dev/null 2>&1; then
echo "${RED}The ${BLUE}"$F"${RED} entry in ${BLUE}"$I" ${RED}needs to use the KERNEL variable instead of kernel version number.${NORMAL}"
echo "The "$F" entry in "$I" needs to use the KERNEL variable instead of kernel version number." >> /tmp/submitqc/missingdeps/"$I"
echo " " >> /tmp/submitqc/missingdeps/"$I"
elif echo "$F" | grep "KERNEL" > /dev/null 2>&1; then
   
    if B=`echo "$F" | sed 's:KERNEL:2.6.33.3-tinycore:g'` && wget --spider "$MIRROR""$VERSION"/tcz/"$B" \
> /dev/null 2>&1; then
sleep .001
            elif B=`echo "$F" | sed 's:KERNEL:2.6.33.3-tinycore64:g'` && wget --spider "$MIRROR""$VERSION"/tcz/"$B" \
> /dev/null 2>&1; then
sleep .001
    else
echo ""$F" is missing in the repo as a dependency of "$I"" >> /tmp/submitqc/missingdeps/"$I"
echo "${BLUE}"$F"${RED} is missing in the repo as a dependency of ${BLUE}"$I"${NORMAL}"
echo " " >> /tmp/submitqc/missingdeps/"$I"
    fi
    else
    if ! wget --spider "$MIRROR""$VERSION"/tcz/"$F" \
> /dev/null 2>&1; then
echo ""$F" is missing in the repo as a dependency of "$I"" >> /tmp/submitqc/missingdeps/"$I"
echo "${BLUE}"$F"${RED} is missing in the repo as a dependency of ${BLUE}"$I"${NORMAL}"
echo " " >> /tmp/submitqc/missingdeps/"$I"
if [ -f "$F" ]; then
  echo "${BLUE}"$F"${GREEN}, however is present in this batch of extensions being checked for upload.${NORMAL}"
          echo ""$F", however, is present in this batch of extensions being checked for upload." >> /tmp/submitqc/missingdeps/"$I"
fi
    fi
   
fi   

done
 done
[ "$(ls -A /tmp/submitqc/missingdeps/)" ] || echo "${GREEN}Dep files seem to have no detectable errors.${NORMAL}"
fi
}

checkinfo() {
for I in `ls *.info`; do

NAME=`basename "$I" .info`
INFONAME=`cat "$I" | grep "Title" | awk '{print $2}'`

if [ ! "$NAME" == "$INFONAME" ]; then
echo "${RED}Name field in ${BLUE}"$I"${RED} is incorrect.${NORMAL}"
echo "Name: field is not valid in "$I"." >> /tmp/submitqc/wronginfofile
fi


echo "The following errors are found in "$I". No news is good news:"
[ -z `awk '/Change-log:/ { print $2 }' "$I"` ] && echo "Change-log: field is not valid in "$I"." && echo "Change-log: field is not valid in "$I"." >> /tmp/submitqc/wronginfofile
[ -z `awk '/Comments:/ { print $2 }' "$I"` ] && echo "Comments: field is not valid in "$I"." && echo "Comments: field is not valid in "$I"." >> /tmp/submitqc/wronginfofile
[ -z `awk '/Original-site:/ { print $2 }' "$I"` ] && echo "Original-site: field is not valid in "$I"." && echo "Original-site: field is not valid in "$I"." >> /tmp/submitqc/wronginfofile
[ -z `awk '/Author:/ { print $2 }' "$I"` ] && echo "Author: field is not valid in "$I"." && echo "Author: field is not valid in "$I"." >> /tmp/submitqc/wronginfofile
[ -z `awk '/Title:/ { print $2 }' "$I"` ] && echo "Title: field is not valid in "$I"." && echo "Title: field is not valid in "$I"." >> /tmp/submitqc/wronginfofile
[ -z `awk '/Description:/ { print $2 }' "$I"` ] && echo "Description: field is not valid in "$I"." && echo "Description: field is not valid in "$I"." >> /tmp/submitqc/wronginfofile
[ -z `awk '/Version:/ { print $2 }' "$I"` ] && echo "Version: field is not valid in "$I"." && echo "Version: field is not valid in "$I"." >> /tmp/submitqc/wronginfofile
[ -z `awk '/Copying-policy:/ { print $2 }' "$I"` ] && echo "Copying-policy: field is not valid in "$I"." && echo "Copying-policy: field is not valid in "$I"." >> /tmp/submitqc/wronginfofile
[ -z `awk '/Extension_by:/ { print $2 }' "$I"` ] && echo "Extension_by: field is not valid in "$I"." && echo "Extension_by: field is not valid in "$I"." >> /tmp/submitqc/wronginfofile
[ -z `awk '/Size:/ { print $2 }' "$I"` ] && echo "Size: field is not valid in "$I"." && echo "Size: field is not valid in "$I"." >> /tmp/submitqc/wronginfofile
[ -z `awk '/Current:/ { print $2 }' "$I"` ] && echo "Current: field is not valid in "$I"." && echo "Current: field is not valid in "$I"." >> /tmp/submitqc/wronginfofile
file "$I" | grep "CRLF" && dos2unix -u "$I" && echo "${YELLOW}Fixing CRLF line terminators in ${BLUE}"$I"${YELLOW}.  Please don't use Windows apps to edit Linux files.${NORMAL}" \
&& echo ""$I" has CRLF line terminators in it.  Please don't use Windows apps to edit Linux files." >> /tmp/submitqc/wronginfofile

sleep 1
done
}



if [ ! -z "$1" ]; then

[ -f /tmp/submitqc/.info.lst ] && rm  /tmp/submitqc/.info.lst
wget -O /tmp/submitqc/.info.lst.gz "$MIRROR""$VERSION"/tcz/info.lst.gz > /dev/null 2>&1
gunzip /tmp/submitqc/.info.lst.gz
sed -i 's:.tcz::g' /tmp/submitqc/.info.lst


if [ "$1" == "checkdep" ]; then
[ -d /tmp/submitqc/missingdeps ] || mkdir -p /tmp/submitqc/missingdeps
checkdep
exit 0
fi

if [ "$1" == "checkinfo" ]; then
[ -d /tmp/submitqc ] || mkdir /tmp/submitqc
checkinfo
exit 0
fi

if [ "$1" == "checkmaintainer" ]; then
[ -d /tmp/submitqc ] || mkdir /tmp/submitqc
> /tmp/submitqc/tcemaintainer
for F in `ls *.tcz`; do
checkmaintainer
done
exit 0
fi

fi

[ -d /tmp/submitqc ] && rm -r /tmp/submitqc

mkdir -p /tmp/submitqc/missingdeps

[ -f /tmp/submitqc/.info.lst ] && rm  /tmp/submitqc/.info.lst
wget -O /tmp/submitqc/.info.lst.gz "$MIRROR""$VERSION"/tcz/info.lst.gz > /dev/null 2>&1
gunzip /tmp/submitqc/.info.lst.gz
sed -i 's:.tcz::g' /tmp/submitqc/.info.lst

> /tmp/submitqc/corrupttcz
> /tmp/submitqc/wrongblocksize
> /tmp/submitqc/wrongmd5
> /tmp/submitqc/tcemaintainer
> /tmp/submitqc/similarextensions
> /tmp/submitqc/appendeddata
> /tmp/submitqc/wrongstartscriptperms
> /tmp/submitqc/wrongstartscriptname
> /tmp/submitqc/wronginfofile
> /tmp/submitqc/baddirperms


for F in `ls | xargs file | grep ".tcz" | grep "Squashfs" | cut -f 1 -d :`; do
TMP2=""$F".tmpdir12345678"
if df | grep "$TMP2" > /dev/null 2>&1; then
umount "$TMP2"
fi
[ -d "$TMP2" ] && rm -rf "$TMP2"
mkdir "$TMP2"
BASENAME=`basename "$F" .tcz`
       

if `sudo busybox mount -o loop "$F" "$TMP2" > /dev/null 2>&1`; then
checkappend
checkstartup
checkbasedirperm
checkdirperms
    if [ -f "$F".list ]; then
checkdiff
fi
if [ -f "$F".info ]; then
checkmaintainer
fi
checkblock
echo_green



busybox umount -d "$TMP2"
sleep 2
rm -rf "$TMP2"
echo " "
echo ${GREEN}##############################################${NORMAL}
echo ${GREEN}##############################################${NORMAL}
else
echo "${BLUE}"$F"${RED} is a corrupt tcz file, please remake.${NORMAL}"
echo "$F" >> /tmp/submitqc/corrupttcz
rm -rf "$TMP2"
fi
   

done
echo " "
checkdep
echo " "
checkinfo
sudo chown tc:staff *.tcz*
sudo chmod 664 *.tcz*

if [ ! -z `cat /tmp/submitqc/corrupttcz` ]; then
echo "${RED}YOU HAVE CORRUPT TCZ FILES. PLEASE REVIEW \
${GREEN}/tmp/submitqc/corrupttcz${NORMAL}"
fi

Title: Re: extension audit script
Post by: samedirection on July 25, 2009, 10:32:47 AM
It could also warn if there are any funny permissions, or files installed outside of the usual areas (just warn, since either may be intended)
Title: Re: extension audit script
Post by: danielibarnes on July 25, 2009, 11:33:44 AM
I've been meaning to do something like this myself, but haven't had time yet. I'm glad you got it started. A few thoughts:

Title: Re: extension audit script
Post by: Jason W on July 25, 2009, 12:22:48 PM
File permissions, correct md5.txt, validity of tce/tcz files is the most important.  But I would like to automate whatever can be done if it can be done safely and reliably.  I included some more checks, and will make it more elegent and modular in time, like checking md5 only, creating list files or whatever.  Clobbering directories will be addressed.  Personally, I think dep files are best made by hand, but if there is a quick and simple way to check for misspelling of entries that would be considered.  Calculating sizes will be considered.

It would be good to make the boring, tedious part of extension preparing as easy as possible.  But I am also careful not to go down a slippery slope of complexity as remembered with tce-uninstall.
Title: Re: extension audit script
Post by: SvOlli on August 13, 2009, 11:46:19 AM
Maybe it would be wise to put the script in the "compiletc" extension or a stand-alone one?
Title: Re: extension audit script
Post by: Jason W on August 13, 2009, 01:08:26 PM
Perhaps, but it is easier to update being posted here. 

I attached the script in an attachment to the original post in this thread for easy download.  Simply adding executable permissions is required.
Title: Re: extension audit script
Post by: Jason W on October 11, 2009, 09:15:04 AM
The script has been updated for tcz only.  Please use it before submitting. 
Title: Re: extension audit script
Post by: Kingdomcome on October 11, 2009, 11:52:58 PM
the stat applet is no longer available in the base....
http://forum.tinycorelinux.net/index.php?topic=3365.0
Title: Re: extension audit script
Post by: Jason W on October 12, 2009, 03:39:33 AM
A check for coreutils has been added.
Title: Re: extension audit script
Post by: Jason W on October 12, 2009, 12:07:39 PM
Checking for the case of tc:staff ownership rather than the case against root ownership of tce.installed and such has been added.
Title: Re: extension audit script
Post by: Jason W on October 16, 2009, 08:57:11 AM
Updated to check for user root, group staff, and 775 perms of /usr/local/tce.* directories.  
Title: Re: extension audit script
Post by: Kingdomcome on October 16, 2009, 01:16:50 PM
any consideration given to adding this script to the compiletc extension?  Retrieving the script for the forums is quite difficult in TC/MC booted base norestore and only loading compile deps.
Title: Re: extension audit script
Post by: Jason W on October 16, 2009, 02:36:43 PM
This script has seen about 4 or 5 updates today, so it would be a burden to include it in an extension at this point.  Perhaps when it stabilizes it can find a home.

Oh, and added one more thing to check for - CRLF line terminators in the info file that cause it to appear funny in the appbrowser. 
Title: Re: extension audit script
Post by: jpeters on October 18, 2009, 07:13:29 PM
Since this is used also for checking tce dir;  needed extensions to run script?

Title: Re: extension audit script
Post by: Jason W on October 18, 2009, 07:18:40 PM
For running against a tce directory, "base norestore" would be the way to do it and then install coreutils into /tmp/tce.  I can drop the perl dependency as dos2unix is in base which I will use instead.  Stat was moved into coreutils so coreutils has to remain a dep. 
Title: Re: extension audit script
Post by: jpeters on October 18, 2009, 10:57:21 PM
For running against a tce directory, "base norestore" would be the way to do it and then install coreutils into /tmp/tce.  I can drop the perl dependency as dos2unix is in base which I will use instead.  Stat was moved into coreutils so coreutils has to remain a dep.  

might add that it creates .list files in the tce dir (or wherever it's placed). 
Title: Re: extension audit script
Post by: jpeters on October 19, 2009, 01:43:54 AM
"fix" script
Here's a quick hack that fixed extension permissions in my tce dir.  eg, "fix gnumeric"

Finds extension in tce_dir and sets tce.menu, tce.installed. tce.icons  to root:staff/775.
Rebuilds and copies to tce_dir. Deletes  old md5 file.

Rerunning audit creates new md5
 
Code: [Select]
#!/bin/ash

TCEDIR="$(cat /opt/.tce_dir)"
[ -d /tmp/extract ] || sudo mkdir /tmp/extract
[ -d /tmp/work ] || sudo mkdir /tmp/work

cd ${TCEDIR}


APP="$(busybox find ${1}* | busybox grep -v .dep | busybox grep -v .txt | busybox grep -v .list )"  
if [ -z ${APP} ]; then
  echo "no match in tce dir"
  exit 1
fi
 

sudo mount ${APP} /tmp/extract -o loop,ro
sudo cp -Rp /tmp/extract/* /tmp/work
cd /tmp/work/usr/local
if [ -d tce.installed ]; then
   sudo chown -R root:staff tce.installed
   sudo chmod  775  tce.installed
   sudo chmod  644  tce.installed/*
fi
if [ -d tce.menu ]; then
   sudo chown -R root:staff tce.menu
   sudo chmod  775  tce.menu
   sudo chmod  644  tce.menu/* 
fi
if [ -d tce.icons ]; then
   sudo chown -R root:staff tce.icons
   sudo chmod  775  tce.icons
   sudo chmod  644  tce.icons/*
fi

cd /tmp/work
sudo mksquashfs  ./  /tmp/${APP}
sudo cp -p /tmp/${APP}  ${TCEDIR}


sudo rm ${TCEDIR}/${APP}.md5.txt


 

###### cleanup

sudo umount /tmp/extract
sudo rmdir /tmp/extract
sudo rm -R /tmp/work
sudo rm /tmp/${APP}

exit 0
Title: Re: extension audit script
Post by: Jason W on October 19, 2009, 04:34:02 AM
I will add a switch to prevent creating of list files as they would interfere with tce2tcz when run on the tce directory.  I didn't initially intend for this to be an audit tool for the tce directory but rather for extensions ready to be submitted.  But it does serve as a quick check for existing extensions in the tce dir especially in regards to the tce.installed thing.

I did use a "fix it" script on my mirror, using a check like in the audit script.
Title: Re: extension audit script
Post by: jpeters on October 19, 2009, 07:56:56 AM
I will add a switch to prevent creating of list files as they would interfere with tce2tcz when run on the tce directory.  I didn't initially intend for this to be an audit tool for the tce directory but rather for extensions ready to be submitted.  But it does serve as a quick check for existing extensions in the tce dir especially in regards to the tce.installed thing.

I did use a "fix it" script on my mirror, using a check like in the audit script.

I already altered update and load scripts in case people want them (only 4K for a record of extension files without an internet connection).

EDIT:  added 644 perm to tce/*
Title: Re: extension audit script
Post by: jpeters on November 15, 2009, 05:17:50 PM
Is this script still working with updated squashfs?  I tried it with a newly compiled dialog.tcz, and it complained that the file was corrupted, although it worked fine.  

edit:  seems to work after I rebooted and tried again.
Title: Re: extension audit script
Post by: Jason W on November 21, 2009, 06:12:03 PM
Updated to check for 4096 block size and echo a red error message on failing md5sum.
Title: Re: extension audit script
Post by: jpeters on November 21, 2009, 11:16:04 PM
Great tool, thanks!
Title: Re: extension audit script
Post by: Jason W on January 09, 2010, 05:01:15 PM
Updated to check for unwanted appended data and recreate list files even if they are present.
Title: Re: extension audit script
Post by: robc on January 12, 2010, 11:26:39 AM
How about auto editing the info file? this can be done for the size and the extension name...

Change Title field:
Code: [Select]
sed -i 's!^Title:.*$!Title:\t\t$F!' $F.info
Change Size field:
Code: [Select]
sed -i 's!^Size:.*$!Size:\t\t`du -h $F | cut -f 1`!' $F.info
Title: Re: extension audit script
Post by: Jason W on January 12, 2010, 11:53:19 AM
Sure, I will add it in.
Title: Re: extension audit script
Post by: robc on January 19, 2010, 09:22:48 PM
The squashfs block 4096 check needs to have unsquashfs, this check needs to be added
Code: [Select]
if [ ! -f /usr/local/tce.installed/squashfs-tools-4.0 ]; then
echo "Install squashfs-tools-4.0.tcz before proceeding."
exit 1
fi

The chown line for the tce* directories needs to have a sudo.

The -noappend flag can be added to the mksquashfs command and the rm line can be removed.
Title: Re: extension audit script
Post by: Jason W on January 27, 2010, 06:00:03 AM
Thanks, I will work that in too.  Will get to both soon.
Title: Re: extension audit script
Post by: Jason W on January 27, 2010, 06:42:43 AM
Done
Title: Re: extension audit script
Post by: Jason W on March 06, 2010, 07:06:15 PM
Added a check for missing entries in dep files.
Title: Re: extension audit script
Post by: jls on March 07, 2010, 01:44:15 AM
 line 82: syntax error: unexpected "then" (expecting "done")
Title: Re: extension audit script
Post by: Jason W on March 07, 2010, 02:03:55 AM
I copied and pasted it and it works fine here.
Title: Re: extension audit script
Post by: jls on April 02, 2010, 06:26:37 AM
recopied and repasted but:
Code: [Select]
/home/tc/.local/bin/tce-check: line 82: syntax error: unexpected "then" (expecting "done"can u publish this script also as an attachment of the post?
Thanx
Title: Re: extension audit script
Post by: jls on April 02, 2010, 06:29:26 AM
wheneve u modify this script and also the others, plz add another post saying that u modified it
Title: Re: extension audit script
Post by: jls on April 02, 2010, 06:32:14 AM
indentation is not perfect: look at line 80
Title: Re: extension audit script
Post by: Jason W on April 02, 2010, 03:34:29 PM
Attachment made.  I have made some portability changes.  At one time I was bugfixing quite a bit and didn't want to announce every little change.  But since the script is pretty much completed, I will announce any new features or feature removal.
Title: Re: extension audit script
Post by: jls on April 03, 2010, 02:12:54 AM
Code: [Select]
tc-ita.tcz is a corrupt tcz file, please remake.
The following errors are found in tc-ita.tcz.info. No news is good news:

changing the line
Code: [Select]
TMP2=`mktemp -d tmp.XXXXX`like this:
Code: [Select]
TMP2=`mktemp -d`it works
Title: Re: extension audit script
Post by: curaga on April 03, 2010, 06:02:21 AM
There should probably be six X's instead of five.
Title: Re: extension audit script
Post by: Jason W on April 03, 2010, 07:58:10 AM
`mktemp -d` makes 10 characters, changed to mktemp -d.
Title: Re: extension audit script
Post by: Jason W on April 03, 2010, 09:33:59 AM
`mktemp -d` was causing problems with me, so using manual directory creation, along with other bug fixes.
Title: Re: extension audit script
Post by: jls on April 03, 2010, 09:58:44 AM
u could post your script also in the wiki
Title: Re: extension audit script
Post by: ^thehatsrule^ on April 03, 2010, 04:32:56 PM
Scripts should not be posted in the wiki - a link back to the thread should be the way to go.
Title: Re: extension audit script
Post by: robc on May 04, 2010, 10:42:53 AM
If Bash isn't installed then there is an error so the Bash check is useless. Try this:
Code: [Select]
#!/bin/sh

if [ ! `which bash` ]; then
echo "Install bash."
exit 1
fi

bash

RED=`echo -e '\e[1;31m'`
YELLOW=`echo -e '\e[1;33m'`
WHITE=`echo -e '\e[00m'`
GREEN=`echo -e '\e[0;32m'`

if [ ! `which file` ]; then
echo "Install the file extension."
exit 1
fi

if [ ! `which mksquashfs` ]; then
echo "Install the squashfs-tools extension."
exit 1
fi

And the info file check can be changed to:
Code: [Select]
for I in `ls *.info`; do

INFOERR=""

echo "Checking for errors in "$I"..."
[ -z `awk '/Title:/ { print $2 }' "$I"` ] && INFOERR=`echo -e "\tTitle: field is not valid in "$I"."\n`
[ -z `awk '/Description:/ { print $2 }' "$I"` ] && INFOERR=$INFOERR`echo -e "\tDescription: field is not valid in "$I"."\n`
[ -z `awk '/Version:/ { print $2 }' "$I"` ] && INFOERR=$INFOERR`echo -e "\tVersion: field is not valid in "$I"."\n`
[ -z `awk '/Author:/ { print $2 }' "$I"` ] && INFOERR=$INFOERR`echo -e "\tAuthor: field is not valid in "$I"."\n`
[ -z `awk '/Original-site:/ { print $2 }' "$I"` ] && INFOERR=$INFOERR`echo -e "\tOriginal-site: field is not valid in "$I"."\n`
[ -z `awk '/Copying-policy:/ { print $2 }' "$I"` ] && INFOERR=$INFOERR`echo -e "\tCopying-policy: field is not valid in "$I"."\n`
[ -z `awk '/Size:/ { print $2 }' "$I"` ] && INFOERR=$INFOERR`echo -e "\tSize: field is not valid in "$I"."\n`
[ -z `awk '/Extension_by:/ { print $2 }' "$I"` ] && INFOERR=$INFOERR`echo -e "\tExtension_by: field is not valid in "$I"."\n`
[ -z `awk '/Comments:/ { print $2 }' "$I"` ] && INFOERR=$INFOERR`echo -e "\tComments: field is not valid in "$I"."\n`
[ -z `awk '/Change-log:/ { print $2 }' "$I"` ] && INFOERR=$INFOERR`echo -e "\tChange-log: field is not valid in "$I"."\n`
[ -z `awk '/Current:/ { print $2 }' "$I"` ] && INFOERR=$INFOERR`echo -e "\tCurrent: field is not valid in "$I"."\n`
file "$I" | grep "CRLF" && dos2unix -u "$I" && INFOERR=$INFOERR`echo -e "\t${YELLOW}Fixed CRLF line terminators in "$I".  Please don't use Windows apps to edit Linux files.${WHITE}"\n`

if [ -n $INFOERR ]; then
echo "${RED}$INFOERR"
else
echo -e "\bNo Errors found."
fi

sleep 1
done
Title: Re: extension audit script
Post by: Jason W on May 04, 2010, 10:54:52 AM
Thanks, I will add that in soon.
Title: Re: extension audit script
Post by: Jason W on June 16, 2010, 09:13:28 AM
Dropped bash requirement, added robc's suggestion and also added the listing of .list file differences from the extension being audited and the one in the repo to draw attention to list file content differences when updating existing extensions.  Particular attention in list file diffs is those of /usr/local/lib/lib.so.X number differences that often result in breakage of dependent apps when the extension being updated is a library one.  This just raises a yellow flag on file content differences for the submitter to determine the relevance. 

A version for 2.x and 3.x are attached to original post.
Title: Re: extension audit script
Post by: Jason W on June 16, 2010, 03:41:50 PM
robc-

I undid your info changes as I saw errors and don't have time to troubleshoot.  If you would confirm it working or bugfix it I would be glad to include it in.
Title: Re: extension audit script
Post by: Jason W on August 31, 2010, 05:56:05 PM
In addition to some fixes, I added a function that echoes both your name from your info file and the extension maintainers name if the extension you are checking already exists in the repo.  If the names don't match, you need to contact the extension maker if you desire to update the existing extension.  Please use it and make sure that you are not sending an intended new submission or an update for an extension already existing by someone else.  There are 2509 tcz extensions in the repo, and it is easy to overlook one, and this tool will help in that regard.  

I use this script on every extension that comes through, but always more efficient to catch one's own errors before they are sent in.
Title: Re: extension audit script
Post by: Jason W on September 01, 2010, 07:18:31 PM
Added some small features and improvements, also using /opt/tcemirror for wget command to allow more universal access.  I don't plan on announcing every small change, but I think for now it is where I want it.

Some files that are created to tell you of issues:

/tmp/submitqc/corrupttcz  - lists corrupt tcz extensions
/tmp/submitqc/wrongblocksize  - lists extensions that are not 4096 block size.
/tmp/submitqc/wrongmd5  -   lists extensions that fail md5sum
/tmp/submitqc/tcemaintainer  -  lists extensions being audited that have existing maintainers and maintainer names.
Title: Re: extension audit script
Post by: Jason W on September 02, 2010, 04:14:17 PM
Added an additional check for extensions in the repo with similar names, so one can make sure that their extension does not already exist in a similarly named format.  For example, running the script on emelfm2-locale.tcz will list these extensions as existing in similar named format:

emelfm.tcz emelfm2.tcz emelfm2-locale.tcz

Then the submitter can look to see if any of those existing extensions are the same as he is about to upload, or if his extension will in any way conflict with those that are already there.  

This information is stored in this file on each run of the audit script:

/tmp/submitqc/similarextensions

Running the script on emelfm.tcz, emelfm2.tcz, and emelfm2-locale.tcz produced this output in the similarextensions file that can be studied after the script is run, handy in the case of auditing a long list of extensions at one time:

Code: [Select]
emelfm.tcz: similar extensions
emelfm2-locale.tcz
emelfm2.tcz
emelfm.tcz
 
emelfm2-locale.tcz: similar extensions
emelfm2-locale.tcz
emelfm2.tcz
emelfm.tcz
 
emelfm2.tcz: similar extensions
emelfm2-locale.tcz
emelfm2.tcz
emelfm.tcz



Also, here is the output of the tcemaintainer file in that same directory:

Code: [Select]
Extension: emelfm.tcz
Maintainer is  juanito
Submitter is  juanito
 
Extension: emelfm2-locale.tcz
Maintainer is  Jason W
Submitter is  Jason W
 
Extension: emelfm2.tcz
Maintainer is  Jason W
Submitter is  Jason W
 
Title: Added to the wiki
Post by: peterc on September 02, 2010, 11:01:26 PM
I just came across this page today, after having submitted several extensions over the past three months. I can't help but wonder who else has missed this. I have therefore added a note to the wiki (http://wiki.tinycorelinux.com/Creating+Extensions#Testing (http://wiki.tinycorelinux.com/Creating+Extensions#Testing)), since that is where people will be looking for information on creating an extension, not the forums.
Title: Re: extension audit script
Post by: Jason W on September 03, 2010, 05:12:43 AM
Thanks!

I also made this a sticky so it doesn't get buried.

Title: Re: extension audit script
Post by: Jason W on September 21, 2010, 06:02:28 PM
I added the check for the KERNEL variable to see if either a 32 bit or 64 bit version exists of entries in dep files, all entries in dep files that do not exist in the repo are stored in /tmp/submitqc/missingdeps.
Title: Re: extension audit script
Post by: Jason W on October 20, 2010, 02:27:17 PM
Added a check for the readability as user TC of the base squashfs directory, to prevent non-readable extensions from getting through. 
Title: Re: extension audit script
Post by: Jason W on October 30, 2010, 06:17:37 PM
Added checking for suspicious directory permissions, echoed in the terminal and
stored in /tmp/submitqc/badperm. 
Title: Re: extension audit script
Post by: Jason W on November 11, 2010, 08:09:33 PM
Added check for wrong startup script name, results echoed in

> /tmp/submitqc/wrongstartscriptname

The extension is automatically rebuilt with proper script name if there is only one file in tce.installed yet it is
not the same name as the extension.  If there are two or more files in tce.installed, then no
automatic rebuilding and renaming takes place since this condition can't be reliably corrected.
Title: Re: extension audit script
Post by: jls on November 17, 2010, 06:30:11 AM
Code: [Select]
tc@box1:/mnt/hda3/jls/estensioni/pygobject-2.7$ submit3
/home/tc/.local/bin/submit3: line 407: stat: not found
Title: Re: extension audit script
Post by: Jason W on November 17, 2010, 07:16:24 AM
Stat is part of coreutils, but I have also seen that error message as when I boot I see /usr/local/tce.installed/coreutils but the extension itself is not loaded.  It is likely a bug in an extension somewhere and I will look more into it later today, have not had the chance to yet.
Title: Re: extension audit script
Post by: robc on December 03, 2010, 10:52:36 AM
I think this script should be put in the repo as an extension. That way the deps will be installed with it. Also it will allow for further automation of build scripts.
Title: Re: extension audit script
Post by: Jason W on December 03, 2010, 11:34:53 AM
Though I keep my remaster, fetch-and-package, and other scripts in this section and out of the repo, the audit script is different as it is used as part of the submission process and recommended for extension makers to either use or at least be aware of what is needed to be correct and factor that into their own routines.

With it in the repo, it will be updated each time tce-update is run if I have made any changes to it.  And deps will be automatically installed and ready to go. 

I have a few minor adjustments to make to it and I will ponder placing it in the repo. 
Title: Re: extension audit script
Post by: Jason W on December 06, 2010, 07:15:40 PM
Now in the repo as submitqc.tcz, and the script name is submitqc.
Title: Re: extension audit script
Post by: Rich on February 11, 2012, 01:19:25 PM
Hi Jason W
Quote
The graphics-2.6.33.3-tinycore.tcz entry in nvidia-96.43.20-module-2.6.33.3-tinycore.tcz.dep needs to use the KERNEL variable instead of kernel version number.
This message could be a little clearer, since you have to replace more than just the number:
Quote
The graphics-2.6.33.3-tinycore.tcz entry in nvidia-96.43.20-module-2.6.33.3-tinycore.tcz.dep needs to have "version-tinycore" replaced with the word "KERNEL" .
The script also ran into trouble trying to fix one of my errors:
Quote
chown: changing ownership of `nvidia-96.43.20-glx.tcz.123456789/usr/local/tce.installed/nvidia-96.43.20-glx': Operation not permitted
chown: changing ownership of `nvidia-96.43.20-glx.tcz.123456789/usr/local/tce.installed': Operation not permitted
I'm guessing there should be a sudo in front of the chmod and chown commands.
Due to cutting and pasting I introduced a typo in my dependency file:
nvidia-96.43.20-module-KERNEL.tcz.dep  instead of  nvidia-96.43.20-module-KERNEL.tcz
The dependency check did not flag it as a missing dependency.

Overall, a very helpful utility, thank you.
Title: Re: extension audit script
Post by: Jason W on February 11, 2012, 04:25:24 PM
Thanks, I have not tended to the script in quite a while, but I will give it some attention soon.
Title: Re: extension audit script
Post by: coreplayer2 on February 12, 2012, 10:47:40 AM
Thanks for the extension, I use it as a extension check whether submitted or not.     Are you making one for scm's? 
Title: Re: extension audit script
Post by: Jason W on February 12, 2012, 12:00:37 PM
Actually, with scms the checks would be fewer and simpler.  I will surely make a script for it soon.
Title: Re: extension audit script
Post by: Jason W on February 12, 2012, 04:47:22 PM
I have not actually announced it, but I have let Robert Schumann, aka gutmensch, handle the main part of the tcz repo since I am occupied with the scms.  And the best help I can give overall is not to meddle and let him tend to it unless my help is solicited.

But I will update the audit script, I plan to simplify it a little, since it really does not need to try to verify .desktop files for one, as it is always giving false negatives on that.  I am considering adding in the 'check for library breakage' routine, but as it is now the data for that is made on my own mirror at home, and is not real time with ibiblio.  Which means it can do as much harm as good if someone has a good extension that will not break the repo but the data is old and falsely states that breakage would occur.  It is good for my personal use, and perhaps to run against existing stuff knowing that the data used may not be current.   But if not in real time with the repo, it doesn't need to be in the audit routine.
Title: Re: extension audit script
Post by: Jason W on February 12, 2012, 05:41:18 PM
Updated, simplified dep check, removed suspicious directory perms check as there are many different dir perms for different apps expected - cannot check for them all, removed desktop file check as again unexpected entries are admissible.
Title: Re: extension audit script
Post by: jls on May 29, 2012, 06:02:08 PM
The script doesn't recognizes me as package maintainer, maybe because there are spaces in my name?
Title: Re: extension audit script
Post by: Jason W on May 29, 2012, 06:59:55 PM
It seems to work fine here, with the spaces

Code: [Select]

root@box:/home/tc# submitqc4
There are extensions without md5.txt files : mldonkey.tcz. Do you want to proceed and create md5.txt files for these and check the existing ones?  y/n  y
 
mldonkey.tcz is maintained by  jls_legalize <unsenepopiu at tin dot it>. You are  jls_legalize <unsenepopiu at tin dot it>.  Make sure the maintainer is aware you are updating his extension.
 
mldonkey.tcz  has some similarly named extensions in the repo.  Make sure yours does not    collide with these:
 
mldonkey.tcz
 
The following errors are found in mldonkey.tcz.info. No news is good news:
mldonkey.tcz is a valid tcz file.  Creating MD5 for mldonkey.tcz:
 
#######

Title: Re: extension audit script
Post by: Alatun on September 18, 2015, 05:44:59 AM
There is a link in the wiki to this thread, howto to submit extensions (http://wiki.tinycorelinux.net/wiki:creating_extensions).

It's a bit cumbersome to read thru this thread, to filter out the relevant information.

Suggestion: put the most recent info in the first posting or put the most recent info into the wiki.

When using the submitqc6 script as user "tc" it complains about missing permissions to remove files in the temporary mounted directory and finally fails to umount the extension.

When running this script as root (sudo) it seems ok. Is this by design? If so, it should be documented.
Title: Re: extension audit script
Post by: gordon64 on September 18, 2015, 07:10:31 AM
Hi Alatun

from the wiki
Quote
If you prefer the build script way, it is easy to build root:root by running the script as root.

and lower down...
Quote
sudo chown -R root:staff /tmp/package/usr/local/tce.installed
sudo chmod -R 775 /tmp/package/usr/local/tce.installed

This shows you that you have 2 choices in building and they are
build as root or
build as local user but fix permission, ownership using sudo

so choose your build method and then either submitqc6 will run as root
or run it as sudo.

good luck
Title: Re: extension audit script
Post by: Alatun on September 18, 2015, 08:26:44 AM
Hi Alatun

from the wiki
Quote
If you prefer the build script way, it is easy to build root:root by running the script as root.

and lower down...
Quote
sudo chown -R root:staff /tmp/package/usr/local/tce.installed
sudo chmod -R 775 /tmp/package/usr/local/tce.installed

This shows you that you have 2 choices in building and they are
build as root or
build as local user but fix permission, ownership using sudo

so choose your build method and then either submitqc6 will run as root
or run it as sudo.

good luck

Hm, the stuff you are quoting deals with "Custom Startup Scripts".

I was refering to the submitqc6 script (a script to check the quality of extensions),
which is no custom startup script, so I'm confused what you like to tell me???

Title: Re: extension audit script
Post by: nitram on September 18, 2015, 10:54:39 AM
When using the submitqc6 script as user "tc" it complains about missing permissions to remove files in the temporary mounted directory and finally fails to umount the extension... When running this script as root (sudo) it seems ok. Is this by design? If so, it should be documented.

I experience the same behaviour/error and end up running submitqc6 with sudo, then all is good. If this is correct, it would be useful if the submitqc6 info file was updated, which presently reads: "submitqc6" is the command to run.
Title: Re: extension audit script
Post by: dentonlt on October 03, 2015, 06:38:09 PM
Seeing as there is still interest (after 6 years!), I took on a rewrite of this script starting around June/July. The first version hit the repo about a week or so ago. New version, with many bug fixes, is now in the submission queue as of this morning (2014/10/04).

submitqc is now cross-repo, tested on 5.x/6.x and x86/x86_64. Run without sudo. [fingers crossed]
Use 'submitqc --help' to see the options.
Checks include ... well, lots of things. Open the script and read the Change Log if you're interested.

Kudos to gordonselfish/gordon64 and coreplayer2 for their bug checking assistance & advice thus far.

Lots left to do/fix, but I hope this update is a good representative of JasonW's original intent.
Title: Re: extension audit script
Post by: nitram on October 03, 2015, 07:28:56 PM
Thanks for your hard work dentonlt.
Quote
Lots left to do/fix, but I hope this update is a good representative of JasonW's original intent.
Is the script still in testing or can it now be used to formally submit an extension?
Title: Re: extension audit script
Post by: dentonlt on October 04, 2015, 07:47:22 PM
Quote
Is the script still in testing or can it now be used to formally submit an extension?

To submit? no. To check your extensions, yes. There are still bugs in it, though, so it would be smart to check over your extension yourself, too.

At the least, the current script can help identify some common errors in submissions. Hope it's good for that.
Title: Re: extension audit script
Post by: dentonlt on October 11, 2015, 02:56:23 PM
Update 20151004 hit the repo today. Next update (to submit in the next week?) adds library checking to help identify missing files in dep list. Knock on wood.

Sent from my HTC_0P6B6 using Tapatalk

Title: Re: extension audit script
Post by: dentonlt on October 16, 2015, 10:07:24 PM
20151017 update is now in the submission queue.

Major change is that I've added dynamic library checking. This scans the extension binaries for dynamic lib requirements, then compares this against the libraries included in extension & the extension dep list. As possible, missing libs are resolved against the local repo and then suggested additions to dep list are made.

Also added --core, --arch, --license, and --changes cli options. Cross-repo maintainers may find the former useful.

I've run this script against much of my local repo (500 extensions in each of x86 and x86_64) and it seems stable. Bug reports and patches welcome, of course.
Title: Re: extension audit script
Post by: _madRat on February 29, 2016, 06:49:25 AM
Is the simple way to disable "Copyright notice" warning in submitqc script - it can not found license file in package.tcz, because it as doc files are located in package-doc.tcz.
Title: Re: extension audit script
Post by: Misalf on February 29, 2016, 09:13:01 AM
Note:
Quote
Please check the license condition. Although we can strip a number of documents from the main tcz and have a separate doc.tcz, this is not always possible. [...]
http://wiki.tinycorelinux.net/wiki:creating_extensions#license
Title: Re: extension audit script
Post by: dentonlt on March 16, 2016, 03:41:39 PM
Is the simple way to disable "Copyright notice" warning in submitqc script - it can not found license file in package.tcz, because it as doc files are located in package-doc.tcz.

_madRat, Misalf: thanks for mentioning this, and the warning re license checking. I don't currently have flags/args for disabling specific tests, but that's probably a good idea for the future. I'll add it to the queue.

Overall, just use good judgement when you're submitting. Some errors/warnings will be acceptable depending upon the package you're submitting. The script currently warns you to contact the maintainer even if you're the current maintainer, for example ...

Other news: thanks to andyj for his notes on armv7 - this lets submitqc run on arm now. I have no arm hardware for testing, however - the script may eat your data! Bug reports and test results much needed and welcome.

submitqc submitted today for 8 repos: 5.x, 6.x, 7.x on x86, x86_64, and armv7.
Title: Re: extension audit script
Post by: Misalf on March 16, 2016, 05:03:04 PM
dentonlt, first of, thanks for contributing and maintaining submitqc.
I find text coloring in the output of scripts is very useful, however, I noticed there is none in the in the current submitqc, contrary to former versions. While there are lots of warnings which might be considered redundant, depending an what extensions are checked (not of your concern), it can be quite cumbersome to check the hole output by reading every line, compared to 'viewing' (f.e. red=error, yellow=warning, green=success etc.).

Any plans for fancy colors for those read-impaired people like me?
Title: Re: extension audit script
Post by: dentonlt on May 17, 2016, 03:38:24 PM
dentonlt, first of, thanks for contributing and maintaining submitqc.
I find text coloring in the output of scripts is very useful, however, I noticed there is none in the in the current submitqc, contrary to former versions. While there are lots of warnings which might be considered redundant, depending an what extensions are checked (not of your concern), it can be quite cumbersome to check the hole output by reading every line, compared to 'viewing' (f.e. red=error, yellow=warning, green=success etc.).

Any plans for fancy colors for those read-impaired people like me?
Hey, misalf:

Damn, so sorry that I missed your post - thought I had the tapatalk notifications on!

In short, use --color or -c. Hoping you already discovered that. Cheers ~
Title: Re: extension audit script
Post by: Misalf on May 17, 2016, 07:48:57 PM
Ah, K then. Will make an alias or something.
Thanks!
Title: Re: extension audit script
Post by: polikuo on December 31, 2016, 05:31:18 AM
Happy New Year people !  :D

submitqc is a great script, a very useful tool that saves us lots of trouble.

I've been making extensions by following the instructions on wiki.
http://wiki.tinycorelinux.net/wiki:creating_extensions (http://wiki.tinycorelinux.net/wiki:creating_extensions)

Quote
sudo chown -R root:staff /tmp/package/usr/local/tce.installed
sudo chmod -R 775 /tmp/package/usr/local/tce.installed

However, every time I make an extension, I get the "wrongstartscriptperms" message.

I never take much heed since the script "fixes" the permission automatically.

Today, however, I got curious and checked the script.

Code: [Select]
tc@box:~$ head -n 619 /usr/local/bin/submitqc | tail -n 71
# if ext has a startup file, check startup name, permissions, location, etc.
checkstartup() {
 
  echo -n "${BLUE}${SCRIPT}: ${F} startup files ok? ${NORMAL}"

  if [ ! -d usr/local/tce.installed/ ]; then
    echo "${GREEN}None present. Ok, I think.${NORMAL}"
    return # no startup script
  fi

  # count files in startup script dir; get name of startup script
  SSDIR="usr/local/tce.installed/"
  SS="$(ls -A $SSDIR)"
  COUNT="$(echo "$SS" | wc -l)"

  # is there only 1 startup file, named same as extension?
  case $COUNT in
    0)  echo -e "${RED}Has startup folder but no script. Repair manually.${NORMAL}"
  echo -e "${F}: startup folder without script. Repair manually." \
>> /tmp/submitqc/startupscript
        ;;
    1)  if [ "${SS}" != "${BASENAME}" ]; then
          echo -ne "\n${YELLOW}script name ($SS) is incorrect. Renaming ${BASENAME}. ${NORMAL}"
          echo "$F startup named $SS renamed $BASENAME" >> /tmp/submitqc/wrongstartscriptname
          SS="$(find $SSDIR -not -type d | tail -n 1)"
          mv "${SS}" "${SSDIR}${BASENAME}"
          echo "${YELLOW}Done.${NORMAL}"
          CHANGES=1
        fi
        ;;
    *)  echo -e "${RED}Multiple startup files, none named $F. Repair manually.${NORMAL}"
  echo -e "${F}: multiple startup files, none named $F. Repair manually." \
>> /tmp/submitqc/startupscript
        ;;
  esac

  DIRERROR=
 
  if [ ! "$(stat -c%a ${SSDIR})" = 775 ]; then
    sudo chmod 775 "usr/local/tce.installed"
    echo -ne "\n\t${YELLOW}Startup directory permissions were corrected.${NORMAL}"
    DIRERROR=1
  fi

  if [ "$(stat -c'%U %G' ${SSDIR})" != "root staff" ]; then
    sudo chown root:staff "${SSDIR}"
    echo -ne "\n\t${YELLOW}Startup directory ownership was corrected.${NORMAL}"
    DIRERROR=1
  fi
 
  if [ ! "$(stat -c%a ${SSDIR}/$BASENAME)" = 755 ]; then
    sudo chmod 755 "${SSDIR}/$BASENAME"
    echo -ne "\n\t${YELLOW}Startup script permissions were corrected.${NORMAL}"
    DIRERROR=1
  fi

  if [ ! "$(stat -c'%U %G' ${SSDIR}/$BASENAME)" = "tc staff" ]; then
    sudo chown tc:staff "${SSDIR}/$BASENAME"
    echo -ne "\n\t${YELLOW}Startup script ownership was corrected.${NORMAL}"
    DIRERROR=1
  fi

  if [ $DIRERROR ]; then
    echo "$F" >> /tmp/submitqc/wrongstartscriptperms
    CHANGES=1
    echo
  else
    echo "${GREEN}Ok.${NORMAL}"
  fi

} # end checkstartup()

"tce.installed" --> "drwxrwxr-x root:staff"
"start-up-script" --> "-rwxr-xr-x tc:staff"

 :o

I thought their owner should both be "root:staff"
with permission "775"
while every thing else as "root:root"

I'm confused.  :-\

P.S.
I'm on x86_64 and the submitqc.tcz.md5.txt look like this:
7b157080f71142c84a5faf2cc66b3308  submitqc.tcz

Most extensions on repo have this problem
Title: Re: extension audit script
Post by: Juanito on December 31, 2016, 05:37:08 AM
Only the /usr/local/tce.installed folder needs to be root:staff - the files in the folder can be tc:staff
Title: Re: extension audit script
Post by: Misalf on December 31, 2016, 05:58:40 AM
The  tce.installed  directory is used for two things: To indicate which extensions are loaded and to run an extension's startup script if present.

staff  needs write permission for the  tce.installed  directory so that there can be created (empty) files for extensions that don't come with a startup script.
The startup scripts are ran as root though. So I guess the startup scripts could be  root:root  but it's not that important?

Is this actually a problem or just a discontinuity?
Title: Re: extension audit script
Post by: polikuo on December 31, 2016, 07:12:25 AM
Only the /usr/local/tce.installed folder needs to be root:staff - the files in the folder can be tc:staff

The  tce.installed  directory is used for two things: To indicate which extensions are loaded and to run an extension's startup script if present.

staff  needs write permission for the  tce.installed  directory so that there can be created (empty) files for extensions that don't come with a startup script.
The startup scripts are ran as root though. So I guess the startup scripts could be  root:root  but it's not that important?

Is this actually a problem or just a discontinuity?

IMHO, since the wiki says that you should set it as "root:staff"
We should keep it consistent  :)
BTW, some users don't use "tc" as their username  ;)
(submitqc might not work if a user set his username differently, not a big deal thought)
I think it would be better to modify the script to fit the wiki's instructions
Title: Re: extension audit script
Post by: Misalf on December 31, 2016, 07:25:55 AM
The user "tc" should be always present though. Similar to "Default" on Windows.
Anyway, I agree that "tc" probably shouldn't be used for permissions if not necessary.
Title: Re: extension audit script
Post by: Sashank999 on July 02, 2020, 07:34:19 AM
Hi !

I have a tortoise like internet speed and I am tired of submitqc fetching the info.lst.gz.

I highly suggest to update the script so that it uses recently fetched info.lst.gz and also check the remote copy's zsync file so that it only downloads the blocks that are different. I also suggest md5 checking for both local and remote info.lst.gz.

Also when checking the extension, it shows :
        Copyright notice not found at usr/local/share/doc/ClipIt
        Confirm that this is not needed.
        Checking repo for copyright statement in ClipIt.tcz ...
        ClipIt.tcz not found on server.

Hence I request to also add --no-copyright or --copyright= options so that the script identifies the copyright file.
Title: Re: extension audit script
Post by: Sashank999 on July 04, 2020, 09:56:30 PM
Hi !

I think that it is better for the script to update the size field in .info file after stripping of binaries, if the binaries are unstripped.
Title: Re: extension audit script
Post by: aus9 on July 04, 2020, 10:15:28 PM
Hi Sashank999
if I run
Code: [Select]
submitqc --libsand have no size in the info files....submitqc will add them.

I prefer to have my build script "strip" where ever  possible rather than use the switch supplied by submitqc.
eg as per wiki we have....multiple mentions of strip
https://www.linuxsecrets.com/tinycorelinux-wiki/wiki:creating_extensions.html

but I prefer ...I build to /tmp
Code: [Select]
P=TCE-name
SRC=some source name
cd /tmp
# wget  command to get source
# unpack
mkdir $P
cd $SRC
# compile commands
make install-strip DESTDIR=/tmp/$P
cd /tmp

I know you have already read Rich script
http://forum.tinycorelinux.net/index.php?action=dlattach;topic=23844.0;attach=5365
Title: Re: extension audit script
Post by: Sashank999 on July 04, 2020, 10:34:51 PM
Hi aus9

I think it is better to add it to submitqc.
Title: Re: extension audit script
Post by: Sashank999 on July 14, 2020, 05:37:47 AM
Hello all !

I have created a patch for submitqc that totally integrates infocreator script inside submitqc. No need of infocreator script. Just apply the patch and run submitqc. I have only changed the "checkinfo()" function inside the original submitqc. I encourage everyone to try it  :D
Title: Re: extension audit script
Post by: dentonlt on July 31, 2020, 08:53:03 PM
Hey, all ~ happy isolation-days greetings from Australia =) The pandemic seems to have created some at-home time to revisit old projects!

Sashank999 - thanks for the infocreator work. I'll have to read all the code to see how you've done it - thanks for posting the patch. Obviously, it's a space that deserved some attention! Unless somebody thinks that we shouldn't include some info file options/cleanup inside of submitqc, I think it makes sense to merge/include this.

Thanks also for suggesting not downloading the info.lst.gz if not required. I don't see any arguments against that one.

@aus9, thanks for your build script framework, too. I'll have to go back to submitqc and check how/when I'm running strip.

Re the stripping,

Seeing as it's been ages since I submitted a submitqc update ... it may take me a bit to remind myself how it all fits together! I'll grab the current version from the repo, read the current docs, and make a formal update/submission in the days ahead.

Hope you're all well out there. Stay safe =)

~ Denton
Title: Re: extension audit script
Post by: aus9 on July 31, 2020, 09:20:35 PM
Hi welcome back.

My only issue with
Code: [Select]
submitqc --libs is I am unable to work out the correct dir own and perms
and then owner and perms for /usr/local/tce.installed/filename
when you look at your system be it TC64 or TC32 there appears to be some randomness.
Title: Re: extension audit script
Post by: Sashank999 on July 31, 2020, 10:36:55 PM
Oops, look like I submitted the wrong patch before. I actually stopped working on it as the last one gone wrong.

I think I should start working on it. Since I know how (some) python modules look, I have also added python things into my info script. I need to work on integrating them into submitqc.
Title: Re: extension audit script
Post by: aus9 on August 05, 2020, 05:31:14 PM
@dentonlt

Due to a recent stuff up by me, where I failed to notice I was a missing   
Quote
:
in my info file. Here is a test to prove current submitqc does not check all of the info file structure,
for you consideration.

Code: [Select]
tce-load -i squashfs-tools submitqc
cd /tmp
rm -rf submitqc zzz
P=zzz
mkdir -p $P/usr/local/$P
mksquashfs $P $P.tcz
md5sum $P.tcz > $P.tcz.md5.txt
cd $P
find usr -not -type d > /tmp/$P.tcz.list
sed 's|usr|/usr|g' -i /tmp/$Z.tcz.list
cd /tmp

echo 'Title:          zzz.tcz
Description:    z
Version:        z
Author:         z
Original-site:  z
Copying-policy: z
Size:           4.0K
Extension_by:   z
Tags            z     
Comments:       z

Change-log:     2020/08/06
Current:        2020/08/06  ' > $P.tcz.info

submitqc --libs

# output  SNIP
# submitqc: There are warnings or errors for your extensions. See:  /tmp/submitqc/copyright

I was missing : after tags

I am aware that some people prefer the EOF method of generating an info file where top 2 lines read
cat >> $P.tcz.info <<'EOF'
Title:             something.tcz
SNIP bottom 2 lines read
Current:        2020/08/06 something
EOF

thanks for reading.
Title: Re: extension audit script
Post by: aus9 on August 14, 2020, 05:21:48 AM
I had a look at coreplayers's firefox get latest script. My coding skills need improving so I may be wrong

It appears there is agreement with the create TCE wiki that dir is root:staff for /usr/local/tce.installed
then the script under that dir is tc:staff

which means this part of the wiki may be wrong
Quote
sudo chown -R root:staff /tmp/package/usr/local/tce.installed

I can not remember if I originally wrote that....but feel free to blame me, as I would have based it on something someone said to me. If its wrong, lets fix. If its right then maybe submitqc is wrong?
Title: Re: extension audit script
Post by: Juanito on August 14, 2020, 06:02:54 AM
What is important is that /usr/local/tce.installed is root:staff - the file in it can be $USER:staff or root:staff
Title: Re: extension audit script
Post by: dentonlt on December 21, 2020, 01:13:54 AM
Damn, I really need to schedule time that's not on school holidays ...
Currently trying to get a gtkpod extension done for TC11 so that I can get my stuff off an old iPod. I'll keep reading through all this to integrate changes.

For all the Christmas-y types out there, Merry Christmas =D
Title: Re: extension audit script
Post by: dentonlt on December 21, 2020, 01:33:01 AM
What is important is that /usr/local/tce.installed is root:staff - the file in it can be $USER:staff or root:staff

Umm ... since $USER is pretty much anybody, I guess I have two options for the startup script:

I'm thinking the first one is more conservative, so I'm going to do that for now. Happy to take advice/direction otherwise.

I'm also thinking there isn't ever a need for a maintainer to provide an extension startup script that must be run by any user other than tc or root. I am admittedly really rusty, though. Maybe I'm missing a whole slew of cases where a non-root non-tc username is advised?
Title: Re: extension audit script
Post by: dentonlt on December 21, 2020, 01:43:33 AM
@dentonlt

Due to a recent stuff up by me, where I failed to notice I was a missing   
Quote
:
in my info file. Here is a test to prove current submitqc does not check all of the info file structure,
for you consideration.

Ahhh, you're very detailed, @aus9! It does mean it probably misses other things, too.
I'll browse both Sashank999's code and the current submitqc. I think I also have some sample/test info files that I used in 2016 ... somewhere ...
Title: Re: extension audit script
Post by: vinceASPECT on December 21, 2020, 02:09:20 AM
HEllo forum,

THat feature which consolidates any extensions dep's (in tcl control panel)  seems good.
(check deps)

thx

v :) :) :)