Feature Request: configurable debian mirror

[cpoakes]

I am just getting started playing with dCore 5.0rc1, and am amazed at what I have been able to accomplish in a few hours. Looking forward to creating small custom Debian images for recycled Thin Clients.

As a regular Deb-a-holic, I have a squid proxy set up for storing debian packages on my NAS and I regularly use debian.osuosl.org.  I want to be able to configure to this Debian mirror, not ftp.us.debian.org.  I have already created a startup script to have sed apply changes to /usr/bin/getDebEnv.  (deb2sce always calls debGetEnv which always uses hard-wired mirror).  It would sure be nice if this could be configurable, i.e. respect the existing /tmp/debmirror.  I am sure there are plenty of other reasons to customize the mirror too...

Also, I expect to build and rebuild SCE files adding more deb packages to consolidate and reduce any redundancies.  Said proxy makes repeating the process faster, but is there a way to preserve the debs on my hard drive until I decide they are no longer needed?

Thanks for the project

[Jason W]

As for the consolidating sce's to eliminate redundancy, use the "import -f" option which will ensure no redudancies.   Just create a file with a list of desired packages, and the packages in the list along with any needed dependencies will made into one large sce with no duplications.

We will look into the subject of being able to choose a Debian mirror.

[cpoakes]

Thanks for the quick response. 

For the mirror, perhaps as /etc/sysconfig/debmirror?  Set to ftp.us.debian.org in the initramfs, but can be altered by the user and managed with filetools?  I realise debGetEnv is making a selection based on architecture, but since the initramfs IS architecure dependent, a fixed value should not be problematic.

Regarding the creation of mega and non-redundant SCEs, I am using "import -f".  It appears to download the DEB packages associated with the SCEs all over again and  not utilize the contents of the SCEs.  Though my proxy speeds things up, I would rather have the option to keep the DEBS around if import is not going to use the contents of the SCEs to build the mega-SCE.  While it would not be a good "default" behaviour to leave DEBs cluttering the filesystem, an option to do so would save time for build/test/modify/build/... iterations.

Cheers.

[Jason W]

Ok, I see what you mean.  I am looking at the option to use another Debian mirror, /opt/debmirror would make sense alongside /opt/tcemirror.  I will also look into the option of keeping debs and data tarballs, checking their md5sum against the Debian package list or our hosted md5sum files and downloading only if they have been updated upstream. 

[Jason W]

/opt/debmirror as a file that contains preferred Debian mirror has been discussed and accepted by the team.  Will be added to Git and part of RC2.

[Jason W]

I am thinking on the concept of keeping debs and related files, it does not seem to require too much code change but also perhaps introduces the need for a basic import configuration file so one's common options, like SECURE or KEEPDEB can be set so a combination of command line flags would not need to be issued on every import.  "import -f" of course would remain a flag, but the two and in the future perhaps others can be set in a config file for the sake of simplicity so forgetting to use a flag for keeping debs would not wipe out the directory of downloaded files that would be cleared by default without a flag or config file option.  The config file when left alone or modified the values within would be used unless overrode by a flag.

I use a single 500MB mydesktop.sce file made from a file list, and with this option and the set of saved debs and related files, say if I was interested in updates to openssl, I could do an update check on the large sce and when an update is available.  Then I could re-import the mydesktop file and it would only have to download the openssl package as well as any other available updates, which would be surely small compared to the entire set of debs that the sce is made of. 

I will arrive at the concept, put it into code, and present it to the team in the near future.

[cpoakes]

Glad the mirror is accepted and I like the possibility of persistence afforded to the /opt location.  As the product (import) matures, a configuration file seems the natural progression. 

Your 500MB desktop.sce is just the scenario I have in mind for keeping DEBs around.  I can certainly test desktop features with a bunch of independent SCEs and put off building the mega-SCE until things are settled.  The problem is I generally decide things are "settled" multiple times when I find omissions or just plain change my mind.  As with any development, a faster iterative cycle makes it less painful.

Thanks.  Look forward to the changes.