Tiny Core Linux
Tiny Core Extensions => TCE Talk => Topic started by: Sashank999 on October 02, 2020, 10:45:10 PM
-
Hi !
I just wanna know about checklibs function in submitqc. I am scratching my head to understand it but I don't get it. I have some ideas on analysing binaries and libs to get the deps. I can implement them in submitqc. Please explain that function if you know about it.
Also, why 2 functions - checkremotedep and checkmirror when both functions do the same ? - Checking if the dep exists on mirror and comparing them.
-
Hi Sashank999
This line:
find -type f -exec file {} \; | grep 'shared libs' | cut -f1 -d: | xargs ldd | \
awk '{print $1}' | sed 's/.*\///' | sort | uniq > $REQLOGis supposed to find all executable files (programs and shared libraries) and run ldd on them to find all dependencies. The
results are sorted, duplicates removed, and the list is saved in $REQLOG.
Any dependencies found in the package being submitted are removed from the list.
If there are still entries in the list, the repo is checked and any entries found there are removed from the list.
If there are still entries in the list, those get flagged as missing dependencies.
That's basically how it should work. I don't believe it does work. I think this part of the find command is incorrect:
| grep 'shared libs'I don't think any results will be returned with that. I think it should read:
| grep 'ELF'so it finds compiled programs and libraries.
-
I am actually asking about the INCLOG,REPOLOG and REQLOG variables.
And, I agree with you. As binaries don't give output "shared libs" with file command, we should grep for ELF as both binaries and .so (shared libs) give output ELF.
I also strongly suggest that ldd should be replaced with readelf -d as readelf gives only direct dependencies whereas ldd lists tree deps also.
-
Hi Sashank999
REQLOG List of all of the dependencies of all of the binaries in the package being checked.
INCLOG List of all of the binaries included in the package.
REPOLOG List of all of the packages dependencies found in the repository.
REQLOG - (INCLOG + REPOLOG) should equal zero if all dependencies exist.
... And, I agree with you. As binaries don't give output "shared libs" with file command, ...
And as far as I could determine, shared libraries don't give that output either.
I also strongly suggest that ldd should be replaced with readelf -d as readelf gives only direct dependencies whereas ldd lists tree deps also.
ldd is the tool of choice for dependency checks. Although I can't provide an example of the top of my head, my concern
would be missing some corner cases using readelf.
-
Hi Sashank999
OK, after some poking around I've learned a few things.
This is the response from file on libc under TC4:
tc@box:~$ file /lib/libc-2.13.so
/lib/libc-2.13.so: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.22, stripped
tc@box:~$So it seems file used to return shared libs , though the type is actually shared object.
This is the response from file on libc under TC10:
tc@E310:~$ file /lib/libc-2.28.so
/lib/libc-2.28.so: ELF 32-bit LSB pie executable, Intel 80386, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 4.8.17, stripped
tc@E310:~$In this case the type is listed as pie executable.
This article goes into the responses file provides depending on compile/link options:
https://stackoverflow.com/questions/34519521/why-does-gcc-create-a-shared-object-instead-of-an-executable-binary-according-to/55704865#55704865
With that out of the way, just grepping for ELF won't work because object files also use the ELF format:
tc@E310:~$ file BuildTCZs/GPicView/gpicview-0.2.5/src/gpicview-image-view.o
BuildTCZs/GPicView/gpicview-0.2.5/src/gpicview-image-view.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped
tc@E310:~$
I believe this will work:
find -type f -exec file {} \; | grep 'ELF' | grep -E 'executable|shared object' | cut -f1 -d: | xargs ldd | awk '{print $1}' | sed 's/.*\///' | sort | uniq
-
I don't know about PIE(I am a real noob when it comes to C or C++).
Well, object files end with .o so "grep -v '\.o$'" might work to filter them out from the list of files even before executing the "file" command. As we are filtering out .o files, we could eliminate "grep -E 'executable|shared object'" and using :
find -type f -exec file {} \; | grep 'ELF' | cut -f1 -d: | xargs ldd | awk '{print $1}' | sed 's/.*\///' | sort | uniqAlso, using "file" command to find .so libs in submitqc is time taking(IMO). We could simply "grep '\.so$'" on list of files to that.
What about my other question ?
Also, why 2 functions - checkremotedep and checkmirror when both functions do the same ? - Checking if the dep exists on mirror and comparing them.
-
Hi Sashank999
... Also, why 2 functions - checkremotedep and checkmirror when both functions do the same ? - Checking if the dep exists on mirror and comparing them.
More than 1 individuals worked on that script. Maybe someone didn't realize the function already existed. Or maybe the
2 functions have subtle differences due to different needs.
-
Looks like dentonlt accidentally added it.
I still don't get why you don't like readelf...
ldd is the tool of choice for dependency checks. Although I can't provide an example of the top of my head, my concern
would be missing some corner cases using readelf.
link=https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html#AEN115
Beware: do not run ldd on a program you don't trust. As is clearly stated in the ldd(1) manual, ldd works by (in certain cases) by setting a special environment variable (for ELF objects, LD_TRACE_LOADED_OBJECTS) and then executing the program. It may be possible for an untrusted program to force the ldd user to run arbitrary code (instead of simply showing the ldd information). So, for safety's sake, don't use ldd on programs you don't trust to execute.
which I think is something that should be known.
-
since there seems to be some interest in improving submitqc, maybe we should add the script to tinycore git?
-
since there seems to be some interest in improving submitqc, maybe we should add the script to tinycore git?
That would be great.
It'll be easier to keep track to all of the changes.
-
I think it should have its own repo.
-
Hi Sashank999
... Well, object files end with .o so "grep -v '\.o$'" might work to filter them out from the list of files even before executing the "file" command. As we are filtering out .o files, we could eliminate "grep -E 'executable|shared object'" ...
I don't know what other unwanted file extensions (or names with no extensions) could pop up, do you?
We want 2 types of ELF files, so that's what it makes sense to search for.
... Also, using "file" command to find .so libs in submitqc is time taking(IMO). We could simply "grep '\.so$'" on list of files to that. ...
That will catch all of the .so files but miss all of the programs.
... I still don't get why you don't like readelf...
I guess the real questions are:
Is it sufficient to only check whether the direct dependencies are available?
Do we need to check for the entire dependency tree?
Maybe someone else could chime in.
link=https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html#AEN115
Beware: do not run ldd on a program you don't trust. As is clearly stated in the ldd(1) manual, ldd works by (in certain cases) by setting a special environment variable (for ELF objects, LD_TRACE_LOADED_OBJECTS) and then executing the program. It may be possible for an untrusted program to force the ldd user to run arbitrary code (instead of simply showing the ldd information). So, for safety's sake, don't use ldd on programs you don't trust to execute.
So, by using readelf to determine dependencies you won't get any bad behavior from the "untrusted program".
Now that you have resolved the dependencies, what do you think will happen when you run the "untrusted program"?
If the program is untrusted, would you be packaging it for the repository?
If you knew in advance that a program is untrusted, you would not be performing any of those activities with it.
-
Hi Guys !
@Juanito @curaga @polikuo Yes, it can be added to the TCL Repo. Creating a new separate repo for just 1 script looks unnecessary.
I don't know what other unwanted file extensions (or names with no extensions) could pop up, do you?
We want 2 types of ELF files, so that's what it makes sense to search for.
Well, look at the object we are grepping - '\.o$' - I don't think any others will pop up because we are
- adding \ to that special . to make it an ordinary one
- using $ to designate the end of line
- using single quotes so that $ doesn't get substituted
That will catch all of the .so files but miss all of the programs.
We are already using 'file' and 'grep ELF' to get the programs.
I guess the real questions are:
Is it sufficient to only check whether the direct dependencies are available?
Do we need to check for the entire dependency tree?
Maybe someone else could chime in.
Yes. As we have dependency chaining system (I don't remember what it is called), I think direct deps check is enough.
I don't think we need to check for entire dependency tree.
I think it would be great if Juanito, dentonlt, curaga participated here along with you.
So, by using readelf to determine dependencies you won't get any bad behavior from the "untrusted program".
Now that you have resolved the dependencies, what do you think will happen when you run the "untrusted program"?
If the program is untrusted, would you be packaging it for the repository?
If you knew in advance that a program is untrusted, you would not be performing any of those activities with it.
Ok. Lets assume that the program is for RPi and due to network access being unavailable on RPi (just assume it), we would need to get it on another system. Then ldd might not work. readelf might work. See this : https://superuser.com/a/239694 .
-
The hypothetical questions here that are not really relevant. submitqc is only going to be ran by the person submitting the extension. That removes the whole trust concern, and also it highly unlikely that you are going to be building on a system without network access.
As for dependencies, normally we include just the first layer in the dep file, but if a second layer made it in, that's not the end of the world. As for ldd/readelf, it makes no difference to me.
-
Hi @Paul_123
Of course its not the end of the world(obviously we don't want that) but repo maintainers consider that.
I also got this command working on creating .tcz.list without unsquashing it. Any suggestions on it ?
unsquashfs -ll ${F} | grep -v '^l' | tee /tmp/submitqc/.${F}.list # For checking with the given one
if diff -u /tmp/submitqc/.${F}.list ${F}.list ; then
echo -en "${RED}${F}.tcz.list is incorrect. ${GREEN}Corrected.${NORMAL}"
cp -f /tmp/submitqc/.${F}.list ./${F}.list
fi
-
This will also create a clean list:
$ for file in *.tcz; do unsquashfs -ll -d '' "$file" | grep -v '^d' | sed -e 's#.* /#/#' -e 's# -> .*##' -e 1,3d > "$file".list ; done
-
Hi Juanito
You left out a pipe through the sort command. I periodically find .list files that are improperly or reverse sorted. Or does
that command guarantee a sorted output?
-
I always create the list from the file system before squashing. Just straight output from
find * -not -type d
Which will definitely clash with the script doing a plain diff comparison.
submitqc never ran properly for arm, so I never even looked at it. Where is the latest iteration of the script.?
-
Hi Sashank999
... I don't know what other unwanted file extensions (or names with no extensions) could pop up, do you?
We want 2 types of ELF files, so that's what it makes sense to search for.
Well, look at the object we are grepping - '\.o$' - I don't think any others will pop up because we are
- adding \ to that special . to make it an ordinary one
- using $ to designate the end of line
- using single quotes so that $ doesn't get substituted ...
I guess my wording wasn't clear. I wasn't questioning your syntax. I was questioning your assumption that all object files
would always have a .o extension. The -o (output) option lets you specify any name and extension you want:
gcc -c xyz.c -o abc.obj
... Yes. As we have dependency chaining system (I don't remember what it is called), I think direct deps check is enough.
I don't think we need to check for entire dependency tree. ...
Since submitqc is intended to check the extension being submitted, checking direct dependencies to make sure the .dep
file is OK makes sense. Since binutils is a dependency of submitqc , readelf is already present anyway.
... I also got this command working on creating .tcz.list without unsquashing it. Any suggestions on it ?
unsquashfs -ll ${F} | grep -v '^l' | tee /tmp/submitqc/.${F}.list # For checking with the given one
if diff -u /tmp/submitqc/.${F}.list ${F}.list ; then
echo -en "${RED}${F}.tcz.list is incorrect. ${GREEN}Corrected.${NORMAL}"
cp -f /tmp/submitqc/.${F}.list ./${F}.list
fi
Only one. Just create the list automatically and skip the remaining steps since that's what the list has to look like anyway.
The script is already quite long. The conditional logic to fix and spit out a message saying "you did it wrong and i fixed it"
adds no value. It simply adds noise to an already lengthy script.
I would suggest the same holds true of other items that are automatically fixed, such as MD5s, permissions, etc.
-
Hi Paul_123
I do something similar:
tc@E310:~/BuildTCZs/GPicView/package/gpicview$ find . -not -type d | cut -c 2- | sort
/usr/local/bin/gpicview
/usr/local/share/applications/gpicview.desktop
/usr/local/share/doc/gpicview/COPYING
/usr/local/share/gpicview/pixmaps/object-flip-horizontal.png
/usr/local/share/gpicview/pixmaps/object-flip-vertical.png
/usr/local/share/gpicview/pixmaps/object-rotate-left.png
/usr/local/share/gpicview/pixmaps/object-rotate-right.png
/usr/local/share/gpicview/ui/pref-dlg.ui
/usr/local/share/pixmaps/gpicview.png
tc@E310:~/BuildTCZs/GPicView/package/gpicview$The cut removes the leading dot but leaves a leading slash. Your version omits the leading slash.
-
I guess that is another case of standardization......from what I've seen the leading slash is always dropped
-
Hi Paul_123
I've come across:
1. Without leading slash.
2. With leading slash.
3. With leading dot and slash.
-
Guys, we have to follow some standards and submitqc should be the qualication test for those standards.
Some of those standards (I think):
1. Always maintain the leading slashes in tcz.list. They should not have ./ or start with letters directly.
2. Check the order of fields in info file and also check if all fields are non-empty.
3. Check if there are child deps in dep file.
4. Check the content and validity(if its a valid gzip) of tgz to be submitted.
5. Add syntax checking (sh -n <SCRIPT>) to startup scripts and build scripts.
6. Check for files that should be in -dev.tcz (.h, .a, .la, .m4, .pc), -doc.tcz (.1 - .9), -locale.tcz (if there are files in usr/local/share/locale/*), -gir.tcz (.gir and .typelib).
7. And basically any other suggestions the repo maintainers, admins or other volunteers give here.
-
It took a while, but submitqc has been added to tinycorelinux git: https://github.com/tinycorelinux/submitqc
-
It took a while, but submitqc has been added to tinycorelinux git: https://github.com/tinycorelinux/submitqc
What about aarch64 ?
--- /usr/local/bin/submitqc 2019-02-18 18:15:53.000000000 +0800
+++ /tmp/submitqc 2020-10-03 11:42:16.585520507 +0800
@@ -182,6 +182,7 @@
i686) BASELIBS="$BASELIBS ld-linux.so.2 linux-gate.so.1"; ARCH="x86" ;;
x86) BASELIBS="$BASELIBS ld-linux.so.2 linux-gate.so.1" ;;
armv7l) BASELIBS="$BASELIBS ld-linux-armhf.so.3 linux-vdso.1"; ARCH="armv7";;
+ aarch64) BASELIBS="$BASELIBS ld-linux-aarch64.so.1 linux-vdso.1"; ARCH="aarch64";;
*) echo "$SCRIPT: Unrecognized architecture '$ARCH'. See --help." && exit 1 ;;
esac
-
change made - thanks