dCore Import Debian Packages to Mountable SCE extensions > dCore X86
Why does sce-update take so much time (even more than sce-import)?
Jason W:
One question, do you have GNU awk installed? GNU awk is almost twice as fast as Busybox awk in getting package info during sce-import and sce-update, just did some tests on it. And awk is what is used in extra repo functions. Awk is always used in sce-import/sce-update, but grep is used to quickly get a snippet of the Packages files in the main and security repo before awk is run on it. Grep gives an about 20% performance increase over using only the awk routine when GNU awk is installed.
I will think of how I can truncate extra repo files during use like the main and security ones are done on the server to only include what is needed. That will save some time.
Thanks
jls:
Hi
do you mean the gawk package?
Jason W:
Hi. Yes, gawk is the Debian/Ubuntu package name.
sm8ps:
Right you are, Jason, GNU Awk does make a big difference compared to Busybox Awk which had been in use before. Here are my updated results. Sorry to repeat most of it but I believe it is easier to read the data when it is all in one place.
Case A) only three (non-relevant) PPAs: case B) the same three PPAs plus 12 Ubuntu repositories. The tests were always run at least twice and showed to be consistent after the DEBINX-file had been updated. In retrospect it would have been better to exclude all extra repositories but the influence of the PPAs does not seem very important.
Old sce-update with Busybox Akw for reference:
--- Code: ---sce-update -rn X-LIST A) 4'56" B) 13'30"
sce-update -crn X-LIST A) 3'30" B) 7'00"
--- End code ---
New sce-update with Busybox Akw:
--- Code: ---sce-update -rn X-LIST A) 2'38" B) 13'18"
sce-update -crn X-LIST A) 0'47" B) 6'48"
--- End code ---
New sce-update with GNU Awk:
--- Code: ---sce-update -rn X-LIST A) 2'40" B) [b]9'05"[/b]
sce-update -crn X-LIST A) 0'41" B) [b]4'17"[/b]
--- End code ---
Time for sce-import:
--- Code: ---sce-import -rpln X-LIST A) 1'51" B) 4'49"
--- End code ---
A) As observed before, the overall time is reduced to about 55% from the original state for standard repositories due to the new sce-update routine. The checking time (option -c) is reduced to about 20% which is pretty spectacular.
B) Using now also GNU Awk reduces the overall time for several big extra repositories to about 65% and the checking time to about 60% which is quite an improvement given the absolute values.
A) Comparing to sce-import, which had been my original motivation, the checking time of sce-update has decreased to about 40% for standard repositories.
B) With several big extra repositories, it is reduced to about 90%. Comparing to the almost 150% from the original state this is impressive and makes sce-update much more performant and thus usable than before.
Many, many thanks for your efforts, Jason!
sm8ps:
I got to test the new debGet* routines that will be in the next release-candidate. They have a similar effect for extra repositories as the new sce-update routine had for the standard repositories. Since the latter case can be considered all solved, I shall only state the results for my former case B) with the extra Ubuntu repositories. This is still with GNU Awk installed but I think this does not matter anymore.
I compare the checking time with the total update time.
* Old sce-update with Busybox Akw
* New sce-update with Busybox Akw
* New sce-update with GNU Awk
* New sce-update with GNU Awk and new debGet* routines
--- Code: --- Check Update
1.) 7'00" 13'30"
2.) 6'48" 13'18"
3.) 4'17" 9'05"
4.) 2'20" 5'22"
--- End code ---
--- Code: ---sce-import:
1.-3.) 4'49"
4) 3'14"
--- End code ---
The results were consistent across three runs. The difference should agree with the time for sce-import which does hold true indeed! Only note that that time has decreased by about 35% as well! I believe this is due to the new debGet* routines. Otherwise this change would point to a flaw in my measurings.
At first, I suspected that the pruning of the repo files only has to be performed once and the fact that I had to run the checking before the actual update is responsible for the miracle. However, this is not supported by the fact that the very first run by mistake was an update without prior checking.
So the new debGet* routines are tremendously effective. The checking time is down to about 75% of the import time which by itself has been reduced by about 35%! Naturally, these values depend on the actual changes in the package dependencies but the performance is lightning fast now.
It is absolutely stunning what JasonW has achieved. Congratulations and many thanks, Jason!
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version