sce-update hi cpu usage

[jls]

Hi
If I'm not wrong sce-update is a long process and takes quite a lot of cpu. what do u guys think about adding a nice option in sceconfig?

[nitram]

To me adding 'nice' might be nice.

Running  sce-update -a  is CPU intensive and i've changed my habits accordingly. Import most SCEs dependent on xorg-* (keeps most SCEs small), only update system every 2 weeks unless something critical, re-import xorg-* without bothering update check (re-import almost as fast as an update check, almost twice as fast an an update check then re-import, almost guarantee >1 dependency requires update), reboot to activate, next session run  sce-update -a  and update remaining smaller SCEs fairly quick, less CPU cycles. The same is probably true for users utilizing large 'base' lists, almost guaranteed several updates in 'base' will be available in a 2 week window, probably faster to re-import every 2 weeks than check for updates then re-import.

Not taking away from the 'nice' request, as jls' query is directly related to  sce-update  intensity. Reviewed sce-update related scripts and wasn't clear to me. Running  sce-update -an  user wants everything updated fast, don't care about an updates list. As an SCE's deps are systematically update checked, does it still complete the long process even if a dep update is available?

If so, which i think is the case, can't a re-import immediately be flagged as soon as the first outdated dep is detected, break out of the check loop early and proceed to next SCE? If this isn't already default behaviour for the -n options, it might make a big difference in update check duration. For example, xorg-vesa has 147 dependencies. If the first dep checked was Xprogs and an update was available, the update check would take ~1/147 the amount of time, exit check loop early, flag xorg-vesa for re-import, process next SCE.

[Jason W]

I am running the command "nice -19 sce-update -an", and it is very nice indeed to my running system where I am wanting to watch video or use firefox or  both.  Would be simple to create a wrapper for sce-update and sce-import to use a nice value specified in /etc/sysconfig/sceconfig.  I also avoid running sce-update or sce-import when actually using my desktop.  Tomorrow I will upload a prototype, where a nice value can be specified in /etc/sysconfig/sceconfig. 

As for exiting and flagging an SCE for update on the first found package needing update, it would break the ability to see all that needs updating, but if that info is not really important, we can go that way too.

I will add the nice value tomorrow to test and we can go from there.

[Jason W]

The question that remains is do we really need to know all the packages or startup script changes involved in an SCE update?  I never needed to know them all, if an SCE needs updating due to changes in Debian/Ubuntu or our files it simply needs updating.  My vote is to halt the update check on the first found update need and move on the checking the next SCE just knowing the previous needs updating.  But I will listen to all opinions.

[nitram]

Thanks for suggesting nice jls and thanks for prototyping Jason - it's all so very, very nice

Many users may not care what was updated, just that it's updated efficiently.

This could be incorporated as a feature (-f fast check) so users have choice.

Draft --help or could incorporate into wiki:

Code: [Select]

sce-update -f  Fast SCE update. Flag SCE(s) for update upon encountering first
               outdated dependency. Fast check all SCEs using the -caf flags or
               auto-update all using the -af flags. Provides no DEBINX diff or
               outdated package list. Workaround for systems that keep
               /tce/import/debs/, run sce-debpurge immediately before
               and after a fast update to identify deprecated packages.

Sorry didn't think this was a new feature suggestion, but after thinking it through...

Edit: To clarify, sce-debpurge runs through quickly compared to sce-update.

[Jason W]

The -f flag in for sce-update in addition to a "nice" option in sceconfig that applies to sce-update and sce-import does sound very nice indeed.  Some may want to know exactly what is needed in a particular SCE update, so the details are not lost for those who want to know. 

Look for an RC of this in the coming days based on the suggestions in this thread.

[Jason W]

Nice option has been added in latest RC, will work on the -f for next cut.

[Jason W]

Added an -f option and tested it with xorg-all.sce, but it only cut about 3 seconds off the time, 1:04 versus about 1:07.  Most of the time with importupdatecheck is spent gathering package deps and md5sum data of the new packages, the part that runs through the md5sum files in /usr/local/sce/xorg-all to compare values is very fast, and that is the only part we can cut short on the first found mismatch.  So that function probably only takes a few seconds as it is simple comparison arithmetic. 

In this case, best to leave as is, but it was worth a try.

[nitram]

Hi Jason. Thanks for testing and reporting, obviously my logic failed! Playing with sce-update and importupdatecheck scripts, removed rotdash, added echos. It's apparent there's lots of heavy pre-processing before the quick md5sum comparison at the end. Replaced awk with mawk, no performance difference. Did some grep, sed and awk reading (fastest -> slowest), wondering it there's a tweak here or there. Will continue to play. Wish there was a faster update check method, just a heavy process, large database files, lots of dependencies, etc. Thanks again for your effort, much appreciated.

[Jason W]

Thanks jls for the idea of a nice option in sceconfig to sce-update/sce-import.

And thanks nitram for your testing different operations in sce-update and it's functions. 

Here is how I see it now, with the nice option, it becomes less relevant how fast sce-update/sce-import does it's thing.  With NICE=19 in sceconfig, I am able to use Firefox, Mplayer, etc with no interference in terms of cpu when an long sce-update/sce-import process is going.  And I have an older machine.  To me, the nice option solves the cpu demands of sce-import/sce-update and lets it run in the background with almost no noticable drain on the other apps one is using.

So no need to wait at all to run sce-update, it now should not interfere with a running session of one's favorite apps.  I admit, I only ran sce-update -an when I was away, but that is no longer needed, can do it now when I want.

[nitram]

With imports and updates this feature looks good, makes the system more responsiveness with other tasks. If majority agree, could default  sceconfig  be updated to indicate 'NICE=19' instead of blank. As all other sceconfig options specify a sane choice, most users don't need to add the file to backup. Also, new users first testing dCore will get a more responsive system from the get-go, providing a better first impression. Personally I don't see a downside, thanks.

[Jason W]

I agree, having a default NICE=19 in sceconfig would give a better impression, will add it in.

[nitram]

Greetings. The NICE feature functions well for high CPU tasks but causes unforeseen minor issues, mostly as the scripts have been renamed and a wrapper used.

* Running  sce-load --help  gives error, need to run  sce-load.sh --help  for error free output.
* Nice value outputed even when inappropriate, example  sce-load --help  outputs 'Using nice level 19'.
* The nice value outputs when running suppressed, example  sce-load -s SCE  .

Thanks.

[curaga]

There's no need to use a wrapper, a process can change its own nice value with renice. IIRC the script's own pid was in $$.

[Jason W]

Renice, didn't think of that, thanks. 

sce-load, sce-import, and sce-update are now wrapper free but with the same nice value function as previous.  Uploaded to RC and those issues should now be fixed.