extension audit script

[dentonlt]

20151017 update is now in the submission queue.

Major change is that I've added dynamic library checking. This scans the extension binaries for dynamic lib requirements, then compares this against the libraries included in extension & the extension dep list. As possible, missing libs are resolved against the local repo and then suggested additions to dep list are made.

Also added --core, --arch, --license, and --changes cli options. Cross-repo maintainers may find the former useful.

I've run this script against much of my local repo (500 extensions in each of x86 and x86_64) and it seems stable. Bug reports and patches welcome, of course.

[_madRat]

Is the simple way to disable "Copyright notice" warning in submitqc script - it can not found license file in package.tcz, because it as doc files are located in package-doc.tcz.

[Misalf]

Note:

Please check the license condition. Although we can strip a number of documents from the main tcz and have a separate doc.tcz, this is not always possible. [...]

http://wiki.tinycorelinux.net/wiki:creating_extensions#license

[dentonlt]

Is the simple way to disable "Copyright notice" warning in submitqc script - it can not found license file in package.tcz, because it as doc files are located in package-doc.tcz.

_madRat, Misalf: thanks for mentioning this, and the warning re license checking. I don't currently have flags/args for disabling specific tests, but that's probably a good idea for the future. I'll add it to the queue.

Overall, just use good judgement when you're submitting. Some errors/warnings will be acceptable depending upon the package you're submitting. The script currently warns you to contact the maintainer even if you're the current maintainer, for example ...

Other news: thanks to andyj for his notes on armv7 - this lets submitqc run on arm now. I have no arm hardware for testing, however - the script may eat your data! Bug reports and test results much needed and welcome.

submitqc submitted today for 8 repos: 5.x, 6.x, 7.x on x86, x86_64, and armv7.

[Misalf]

dentonlt, first of, thanks for contributing and maintaining submitqc.
I find text coloring in the output of scripts is very useful, however, I noticed there is none in the in the current submitqc, contrary to former versions. While there are lots of warnings which might be considered redundant, depending an what extensions are checked (not of your concern), it can be quite cumbersome to check the hole output by reading every line, compared to 'viewing' (f.e. red=error, yellow=warning, green=success etc.).

Any plans for fancy colors for those read-impaired people like me?

[dentonlt]

dentonlt, first of, thanks for contributing and maintaining submitqc.
I find text coloring in the output of scripts is very useful, however, I noticed there is none in the in the current submitqc, contrary to former versions. While there are lots of warnings which might be considered redundant, depending an what extensions are checked (not of your concern), it can be quite cumbersome to check the hole output by reading every line, compared to 'viewing' (f.e. red=error, yellow=warning, green=success etc.).

Any plans for fancy colors for those read-impaired people like me?

Hey, misalf:

Damn, so sorry that I missed your post - thought I had the tapatalk notifications on!

In short, use --color or -c. Hoping you already discovered that. Cheers ~

[Misalf]

Ah, K then. Will make an alias or something.
Thanks!

[polikuo]

Happy New Year people ! 

submitqc is a great script, a very useful tool that saves us lots of trouble.

I've been making extensions by following the instructions on wiki.
http://wiki.tinycorelinux.net/wiki:creating_extensions

sudo chown -R root:staff /tmp/package/usr/local/tce.installed
sudo chmod -R 775 /tmp/package/usr/local/tce.installed

However, every time I make an extension, I get the "wrongstartscriptperms" message.

I never take much heed since the script "fixes" the permission automatically.

Today, however, I got curious and checked the script.

Code: [Select]

tc@box:~$ head -n 619 /usr/local/bin/submitqc | tail -n 71
# if ext has a startup file, check startup name, permissions, location, etc.
checkstartup() {
 
  echo -n "${BLUE}${SCRIPT}: ${F} startup files ok? ${NORMAL}"

  if [ ! -d usr/local/tce.installed/ ]; then
    echo "${GREEN}None present. Ok, I think.${NORMAL}"
    return # no startup script
  fi

  # count files in startup script dir; get name of startup script
  SSDIR="usr/local/tce.installed/"
  SS="$(ls -A $SSDIR)"
  COUNT="$(echo "$SS" | wc -l)"

  # is there only 1 startup file, named same as extension?
  case $COUNT in
    0)  echo -e "${RED}Has startup folder but no script. Repair manually.${NORMAL}"
  echo -e "${F}: startup folder without script. Repair manually." \
>> /tmp/submitqc/startupscript
        ;;
    1)  if [ "${SS}" != "${BASENAME}" ]; then
          echo -ne "\n${YELLOW}script name ($SS) is incorrect. Renaming ${BASENAME}. ${NORMAL}"
          echo "$F startup named $SS renamed $BASENAME" >> /tmp/submitqc/wrongstartscriptname
          SS="$(find $SSDIR -not -type d | tail -n 1)"
          mv "${SS}" "${SSDIR}${BASENAME}"
          echo "${YELLOW}Done.${NORMAL}"
          CHANGES=1
        fi
        ;;
    *)  echo -e "${RED}Multiple startup files, none named $F. Repair manually.${NORMAL}"
  echo -e "${F}: multiple startup files, none named $F. Repair manually." \
>> /tmp/submitqc/startupscript
        ;;
  esac

  DIRERROR=
 
  if [ ! "$(stat -c%a ${SSDIR})" = 775 ]; then
    sudo chmod 775 "usr/local/tce.installed"
    echo -ne "\n\t${YELLOW}Startup directory permissions were corrected.${NORMAL}"
    DIRERROR=1
  fi

  if [ "$(stat -c'%U %G' ${SSDIR})" != "root staff" ]; then
    sudo chown root:staff "${SSDIR}"
    echo -ne "\n\t${YELLOW}Startup directory ownership was corrected.${NORMAL}"
    DIRERROR=1
  fi
 
  if [ ! "$(stat -c%a ${SSDIR}/$BASENAME)" = 755 ]; then
    sudo chmod 755 "${SSDIR}/$BASENAME"
    echo -ne "\n\t${YELLOW}Startup script permissions were corrected.${NORMAL}"
    DIRERROR=1
  fi

  if [ ! "$(stat -c'%U %G' ${SSDIR}/$BASENAME)" = "tc staff" ]; then
    sudo chown tc:staff "${SSDIR}/$BASENAME"
    echo -ne "\n\t${YELLOW}Startup script ownership was corrected.${NORMAL}"
    DIRERROR=1
  fi

  if [ $DIRERROR ]; then
    echo "$F" >> /tmp/submitqc/wrongstartscriptperms
    CHANGES=1
    echo
  else
    echo "${GREEN}Ok.${NORMAL}"
  fi

} # end checkstartup()

"tce.installed" --> "drwxrwxr-x root:staff"
"start-up-script" --> "-rwxr-xr-x tc:staff"

 

I thought their owner should both be "root:staff"
with permission "775"
while every thing else as "root:root"

I'm confused. 

P.S.
I'm on x86_64 and the submitqc.tcz.md5.txt look like this:
7b157080f71142c84a5faf2cc66b3308  submitqc.tcz

Most extensions on repo have this problem

[Juanito]

Only the /usr/local/tce.installed folder needs to be root:staff - the files in the folder can be tc:staff

[Misalf]

The  tce.installed  directory is used for two things: To indicate which extensions are loaded and to run an extension's startup script if present.

staff  needs write permission for the  tce.installed  directory so that there can be created (empty) files for extensions that don't come with a startup script.
The startup scripts are ran as root though. So I guess the startup scripts could be  root:root  but it's not that important?

Is this actually a problem or just a discontinuity?

[polikuo]

Only the /usr/local/tce.installed folder needs to be root:staff - the files in the folder can be tc:staff

The  tce.installed  directory is used for two things: To indicate which extensions are loaded and to run an extension's startup script if present.

staff  needs write permission for the  tce.installed  directory so that there can be created (empty) files for extensions that don't come with a startup script.
The startup scripts are ran as root though. So I guess the startup scripts could be  root:root  but it's not that important?

Is this actually a problem or just a discontinuity?

IMHO, since the wiki says that you should set it as "root:staff"
We should keep it consistent 
BTW, some users don't use "tc" as their username 
(submitqc might not work if a user set his username differently, not a big deal thought)
I think it would be better to modify the script to fit the wiki's instructions

[Misalf]

The user "tc" should be always present though. Similar to "Default" on Windows.
Anyway, I agree that "tc" probably shouldn't be used for permissions if not necessary.

[Sashank999]

Hi !

I have a tortoise like internet speed and I am tired of submitqc fetching the info.lst.gz.

I highly suggest to update the script so that it uses recently fetched info.lst.gz and also check the remote copy's zsync file so that it only downloads the blocks that are different. I also suggest md5 checking for both local and remote info.lst.gz.

Also when checking the extension, it shows :
        Copyright notice not found at usr/local/share/doc/ClipIt
        Confirm that this is not needed.
        Checking repo for copyright statement in ClipIt.tcz ...
        ClipIt.tcz not found on server.

Hence I request to also add --no-copyright or --copyright= options so that the script identifies the copyright file.

[Sashank999]

Hi !

I think that it is better for the script to update the size field in .info file after stripping of binaries, if the binaries are unstripped.

[aus9]

Hi Sashank999
if I run

Code: [Select]

submitqc --libsand have no size in the info files....submitqc will add them.

I prefer to have my build script "strip" where ever  possible rather than use the switch supplied by submitqc.
eg as per wiki we have....multiple mentions of strip
http://wiki.tinycorelinux.net/doku.php?id=wiki:creating_extensions.html

but I prefer ...I build to /tmp

Code: [Select]

P=TCE-name
SRC=some source name
cd /tmp
# wget  command to get source
# unpack
mkdir $P
cd $SRC
# compile commands
make install-strip DESTDIR=/tmp/$P
cd /tmp
I know you have already read Rich script
http://forum.tinycorelinux.net/index.php?action=dlattach;topic=23844.0;attach=5365