Understanding submitqc

[Sashank999]

Hi !

I just wanna know about checklibs function in submitqc. I am scratching my head to understand it but I don't get it. I have some ideas on analysing binaries and libs to get the deps. I can implement them in submitqc. Please explain that function if you know about it.

Also, why 2 functions - checkremotedep and checkmirror when both functions do the same ? - Checking if the dep exists on mirror and comparing them.

[Rich]

Hi Sashank999
This line:

Code: [Select]

find -type f -exec file {} \; | grep 'shared libs' | cut -f1 -d: | xargs ldd | \
    awk '{print $1}' | sed 's/.*\///' | sort | uniq > $REQLOGis supposed to find all executable files (programs and shared libraries) and run  ldd  on them to find all dependencies. The
results are sorted, duplicates removed, and the list is saved in  $REQLOG.

Any dependencies found in the package being submitted are removed from the list.
If there are still entries in the list, the repo is checked and any entries found there are removed from the list.
If there are still entries in the list, those get flagged as missing dependencies.

That's basically how it should work. I don't believe it does work. I think this part of the  find  command is incorrect:

Code: [Select]

| grep 'shared libs'I don't think any results will be returned with that. I think it should read:

Code: [Select]

| grep 'ELF'so it finds compiled programs and libraries.

[Sashank999]

I am actually asking about the INCLOG,REPOLOG and REQLOG variables.
And, I agree with you. As binaries don't give output "shared libs" with file command, we should grep for ELF as both binaries and .so (shared libs) give output ELF.
I also strongly suggest that ldd should be replaced with readelf -d as readelf gives only direct dependencies whereas ldd lists tree deps also.

[Rich]

Hi Sashank999
REQLOG    List of all of the dependencies of all of the binaries in the package being checked.
INCLOG     List of all of the binaries included in the package.
REPOLOG  List of all of the packages dependencies found in the repository.

REQLOG - (INCLOG + REPOLOG)  should equal zero if all dependencies exist.

... And, I agree with you. As binaries don't give output "shared libs" with file command, ...

And as far as I could determine, shared libraries don't give that output either.

I also strongly suggest that ldd should be replaced with readelf -d as readelf gives only direct dependencies whereas ldd lists tree deps also.

ldd is the tool of choice for dependency checks. Although I can't provide an example of the top of my head, my concern
would be missing some corner cases using readelf.

[Rich]

Hi Sashank999
OK, after some poking around I've learned a few things.
This is the response from file on libc under TC4:

Code: [Select]

tc@box:~$ file /lib/libc-2.13.so
/lib/libc-2.13.so: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.22, stripped
tc@box:~$So it seems file used to return  shared libs , though the type is actually  shared object.

This is the response from file on libc under TC10:

Code: [Select]

tc@E310:~$ file /lib/libc-2.28.so
/lib/libc-2.28.so: ELF 32-bit LSB pie executable, Intel 80386, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 4.8.17, stripped
tc@E310:~$In this case the type is listed as  pie executable.

This article goes into the responses  file  provides depending on compile/link options:
https://stackoverflow.com/questions/34519521/why-does-gcc-create-a-shared-object-instead-of-an-executable-binary-according-to/55704865#55704865

With that out of the way, just grepping for  ELF  won't work because object files also use the ELF format:

Code: [Select]

tc@E310:~$ file BuildTCZs/GPicView/gpicview-0.2.5/src/gpicview-image-view.o
BuildTCZs/GPicView/gpicview-0.2.5/src/gpicview-image-view.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped
tc@E310:~$
I believe this will work:

Code: [Select]

find -type f -exec file {} \; | grep 'ELF' | grep -E 'executable|shared object' | cut -f1 -d: | xargs ldd | awk '{print $1}' | sed 's/.*\///' | sort | uniq

[Sashank999]

I don't know about PIE(I am a real noob when it comes to C or C++).
Well, object files end with .o so "grep -v '\.o$'" might work to filter them out from the list of files even before executing the "file" command. As we are filtering out .o files, we could eliminate "grep -E 'executable|shared object'" and using :

Code: [Select]

find -type f -exec file {} \; | grep 'ELF' | cut -f1 -d: | xargs ldd | awk '{print $1}' | sed 's/.*\///' | sort | uniqAlso, using "file" command to find .so libs in submitqc is time taking(IMO). We could simply "grep '\.so$'" on list of files to that.

What about my other question ?

Also, why 2 functions - checkremotedep and checkmirror when both functions do the same ? - Checking if the dep exists on mirror and comparing them.

[Rich]

Hi Sashank999

... Also, why 2 functions - checkremotedep and checkmirror when both functions do the same ? - Checking if the dep exists on mirror and comparing them.

More than 1 individuals worked on that script. Maybe someone didn't realize the function already existed. Or maybe the
2 functions have subtle differences due to different needs.

[Sashank999]

Looks like dentonlt accidentally added it.
I still don't get why you don't like readelf...

ldd is the tool of choice for dependency checks. Although I can't provide an example of the top of my head, my concern
would be missing some corner cases using readelf.

link=https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html#AEN115
Beware: do not run ldd on a program you don't trust. As is clearly stated in the ldd(1) manual, ldd works by (in certain cases) by setting a special environment variable (for ELF objects, LD_TRACE_LOADED_OBJECTS) and then executing the program. It may be possible for an untrusted program to force the ldd user to run arbitrary code (instead of simply showing the ldd information). So, for safety's sake, don't use ldd on programs you don't trust to execute.

which I think is something that should be known.

[Juanito]

since there seems to be some interest in improving submitqc, maybe we should add the script to tinycore git?

[polikuo]

since there seems to be some interest in improving submitqc, maybe we should add the script to tinycore git?

That would be great.

It'll be easier to keep track to all of the changes.

[curaga]

I think it should have its own repo.

[Rich]

Hi Sashank999

... Well, object files end with .o so "grep -v '\.o$'" might work to filter them out from the list of files even before executing the "file" command. As we are filtering out .o files, we could eliminate "grep -E 'executable|shared object'" ...

I don't know what other unwanted file extensions (or names with no extensions) could pop up, do you?
We want 2 types of ELF files, so that's what it makes sense to search for.

... Also, using "file" command to find .so libs in submitqc is time taking(IMO). We could simply "grep '\.so$'" on list of files to that. ...

That will catch all of the  .so  files but miss all of the programs.

... I still don't get why you don't like readelf...

I guess the real questions are:
Is it sufficient to only check whether the direct dependencies are available?
Do we need to check for the entire dependency tree?
Maybe someone else could chime in.

link=https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html#AEN115
Beware: do not run ldd on a program you don't trust. As is clearly stated in the ldd(1) manual, ldd works by (in certain cases) by setting a special environment variable (for ELF objects, LD_TRACE_LOADED_OBJECTS) and then executing the program. It may be possible for an untrusted program to force the ldd user to run arbitrary code (instead of simply showing the ldd information). So, for safety's sake, don't use ldd on programs you don't trust to execute.

So, by using  readelf  to determine dependencies you won't get any bad behavior from the "untrusted program".
Now that you have resolved the dependencies, what do you think will happen when you run the "untrusted program"?
If the program is untrusted, would you be packaging it for the repository?
If you knew in advance that a program is untrusted, you would not be performing any of those activities with it.

[Sashank999]

Hi Guys !

@Juanito @curaga @polikuo Yes, it can be added to the TCL Repo. Creating a new separate repo for just 1 script looks unnecessary.

I don't know what other unwanted file extensions (or names with no extensions) could pop up, do you?
We want 2 types of ELF files, so that's what it makes sense to search for.

Well, look at the object we are grepping - '\.o$' - I don't think any others will pop up because we are
- adding \ to that special . to make it an ordinary one
- using $ to designate the end of line
- using single quotes so that $ doesn't get substituted

That will catch all of the  .so  files but miss all of the programs.

We are already using 'file' and 'grep ELF' to get the programs.

I guess the real questions are:
Is it sufficient to only check whether the direct dependencies are available?
Do we need to check for the entire dependency tree?
Maybe someone else could chime in.

Yes. As we have dependency chaining system (I don't remember what it is called), I think direct deps check is enough.
I don't think we need to check for entire dependency tree.
I think it would be great if Juanito, dentonlt, curaga participated here along with you.

So, by using  readelf  to determine dependencies you won't get any bad behavior from the "untrusted program".
Now that you have resolved the dependencies, what do you think will happen when you run the "untrusted program"?
If the program is untrusted, would you be packaging it for the repository?
If you knew in advance that a program is untrusted, you would not be performing any of those activities with it.

Ok. Lets assume that the program is for RPi and due to network access being unavailable on RPi (just assume it), we would need to get it on another system. Then ldd might not work. readelf might work. See this : https://superuser.com/a/239694 .

[Paul_123]

The hypothetical questions here that are not really relevant.  submitqc is only going to be ran by the person submitting the extension.  That removes the whole trust concern, and also it highly unlikely that you are going to be building on a system without network access.

As for dependencies, normally we include just the first layer in the dep file, but if a second layer made it in, that's not the end of the world.  As for ldd/readelf, it makes no difference to me.

[Sashank999]

Hi @Paul_123

Of course its not the end of the world(obviously we don't want that) but repo maintainers consider that.

I also got this command working on creating .tcz.list without unsquashing it. Any suggestions on it ?

Code: [Select]

unsquashfs -ll ${F} | grep -v '^l' | tee /tmp/submitqc/.${F}.list # For checking with the given one
if diff -u /tmp/submitqc/.${F}.list ${F}.list ; then
echo -en "${RED}${F}.tcz.list is incorrect. ${GREEN}Corrected.${NORMAL}"
cp -f /tmp/submitqc/.${F}.list ./${F}.list
fi