Proposal for upgrading .tcz on .tcz.dep changed

[GNUser]

Hi Rich. Thank you for confirming my suspicion. And for being such a good archeologist.

I don't mind using a bigger gun to cover against a wider range of possible interference. But I terribly dislike redundancy--not only because of bloat, but also because it makes it difficult to guess what I was thinking when I look at a script again much later. I updated the script (including the link in Reply #40) accordingly.

[curaga]

Hi curaga
If I place an updated .dep file in staging and then run update repo, does
dep.db.gz get updated then?

I'm not sure putting just a dep file in staging would work, but the same step that generates provides.db also generates dep.db.

[patrikg]

Here is the (hopefully) final version and time to relax

Something of that kind, I guess ;-)

By the way, isn't using GNU diff a shooting crows with the nuke? Haven't You tried

Code: [Select]

[ "$(cat file1)" == "${cat file2}" ]
?

I think I have seen a better way of this and faster.
You not need to load cat for doing this.

Code: (bash) [Select]

$(cat file1)
You can do this.

Code: (bash) [Select]

$(< file1)
Very clever, but I don't know if this works with every shell ?

[GNUser]

Hi patrickg.

You not need to load cat for doing this.

Code: (bash) [Select]

$(cat file1)
You can do this.

Code: (bash) [Select]

$(< file1)
Very clever, but I don't know if this works with every shell ?

It does not seem to work, at least not in Busybox ash. Try this with files foo.dep and bar.dep:

Code: [Select]

[ "$(< foo.dep)" = "$(< bar.dep)" ] && echo "files are identical" || echo "files are different"The goal is to see "files are identical" if the two files contain the same words (i.e., extensions) and differ only by whitespace (i.e., leading whitespace, trailing whitespace, extra lines, extra lines with spaces). The code snippet above results in "files are identical" all the time, even if the two files contain totally different words.

I'm okay with using cat and echo for this. It gets the job done, requires nothing outside base system, and does not cross over beyond clever into cryptic.

[Rich]

Hi curaga

I'm not sure putting just a dep file in staging would work, but the same step that generates provides.db also generates dep.db.

OK, I just tried it and  dep.db.gz  does get updated when only a  .dep  file is present.

[patrikg]

Sorry for confusing, it does work in bash.

[GNUser]

Hi patrikg. No it doesn't, unfortunately. I just tested it in bash: If foo.dep and bar.dep differ only by whitespace (empty lines, for example), then I get "files are different" when I use the code snipped I posted in Reply #48. This is not what we need.

It was a good thought, though. Thanks for pitching in.

[GNUser]

Hi jazzbiker. Believe it or not, I found yet another opportunity for increased efficiency in the update-everything script, which will further decrease your execution time.

TL;DR explanation:
In version 6.1 of the update-everything script, tce-audit builddb is always one of the steps. However, it only needs to run if .dep files have changed, so that tce-audit fetchmissing works properly. Version 7.0 of the script (available at the link in Reply #40) takes this into account.

Detailed explanation:
tce-audit builddb produces three files:
/tmp/audit_results.txt
/etc/sysconfig/tcedir/optional/tce.db
/etc/sysconfig/tcedir/optional/tce.lst.

It turns out that tce-update does not utilize any of these files. I confirmed this not only by examining the tce-update script, but also by running numerous tests. As far as tce-update is concerned, it's all about the .md5.txt files

Based on earlier tests, I know that if .dep files have changed and your system is now missing some dependencies, then tce-audit builddb needs to run before tce-audit fetchmissing or else the missing dependencies are not fetched.

So here is the current (version 7.0) logic of the script:

Are system's .dep files current?
    If yes: Run tce-update
    If no: Update the .dep files that are outdated, run tce-audit builddb, run tce-audit fetchmissing, run tce-update

[patrikg]

Strange it works for me in bash, and for more confusing things in bash scripts, i think you missing some chars in the if/test statement. You have to evaluate twice i think in the if/test statement. And i think you missing one equal sign.

So i should try something like this.

Code: (bash) [Select]

[[ "$(< foo.dep)" == "$(< bar.dep)" ]]

And for more confusing things in the shell, i think the char [ is a command, alias to test.
So maybe it's fine with only one equal sign, but with the if statement i think you need to use dubble equal signs
to not assign the variable with some value.(if using only one equal sign).

And all this is in bash shell. Not ash.

Like i said before I am not any programmer.

Happy hacking.

[GNUser]

Hi patrikg. You are right that [ is an alias for the test command.
[[ is only available in bash. It does have more features than test.
In all fairness I did not test your suggestion in bash with [[, only in bash with [
I'll take your word for it that your suggestion works with bash's [[, but I don't want to drag in bash as a dependency for the script.

[ foo == bar ] is a bashism, see https://mywiki.wooledge.org/Bashism. Some POSIX shells can work with that, but the correct POSIX syntax for comparison is [ foo = bar ]

I know that in many contexts == is for comparison and = is for assignment, but not in this context.

[jazzbiker]

Hi jazzbiker. Believe it or not, I found yet another opportunity for increased efficiency in the update-everything script, which will further decrease your execution time.

Hi GNUser,

Believe it or not, but the new version is really faster :-) I've run it in the same conditions with the same tce14 directory content:

Code: [Select]

real    0m 11.31s
user    0m 4.65s
sys     0m 4.67s

to be compared with the measurements in the post #36.

Well done!

[patrikg]

Now i have tested i little bit with busybox ash.
I thinks it not working.

See this.

Code: (bash) [Select]

~ $ [ "$(</etc/passwd)" = "$(</etc/passwd)" ] && echo Working
Working
~ $ [ "$(</etc/passwd)" = "qwertyu" ] && echo Working
~ $

And for more test.

Code: (bash) [Select]

[patrik@hp-arch-new ~]$ ./busybox ash
~ $ echo "test" > test1
~ $ echo "test" > test2
~ $ [ "$(<test1)" = "$(<test2)" ] && echo Working
Working
~ $ echo "test1" > test3
~ $ [ "$(<test1)" = "$(<test3)" ] && echo Working
Working
~ $ cat test3
test1
~ $ [ "$(cat test1)" = "$(cat test3)" ] && echo Working
~ $ [ "$(cat test1)" = "$(cat test2)" ] && echo Working
Working
~ $

I am not any programmer and i don't understand whats happening with this!! You are correct, this is not working.

Code: (bash) [Select]

[patrik@hp-arch-new ~]$ ./busybox
BusyBox v1.35.0 (2022-01-17 19:57:02 CET) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2015.
Licensed under GPLv2. See source distribution for detailed


[jazzbiker]

Hi GNUser,

Your current 7.0 version calls tce-update. tce-update includes check_dependencies() function, which does noop after fetchmissing. I've commented calls of check_dependencies() in original tce-update and here are the measurements:

Code: [Select]

real    0m 10.44s
user    0m 4.29s
sys     0m 4.23s

Edit: I may be wrong, but I have the feeling that check_dependencies() in tce-update is the reminder from the ancient times when tce-load was not performing recursive loading.

[GNUser]

Hi patrikg. I think you are missing a nuance here. The .dep files being compared need to be considered identical if they differ only by whitespace. Please take a look earlier in this thread where jazzbiker points out the need for this.

Hi jazzbiker. It's true that tce-update does not need to look for missing dependencies if fetchmissing has already taken care of that. Unfortunately tce-update does not currently give us the option of skipping check_dependencies (short of commenting-out the call to the function as you did).

[GNUser]

Hi jazzbiker. Thank you for finding the possibility of yet another efficiency. I'm going to submit a pull request to add --skip-dependency-check option to tce-update. Then, logic of update-everything can be upgraded to this:

Are system's .dep files current?
    If yes: Run "tce-update --skip-dependency-check"
    If no: Update the .dep files that are outdated, run tce-audit builddb, run tce-audit fetchmissing, run "tce-update --skip-dependency-check"

This should achieve a further decrease in your execution time