A Failure Mode for tce-load Re-download

[vitex]

tce-load is implemented using wget with the -c option.  The manpage for the GNU version of wget has the following caution:

... while using -c, any file that's bigger on the server than locally will be considered an incomplete download and only "(length(remote) - length(local))" bytes will be downloaded and tacked onto the end of the local file.  This behavior can be desirable in certain cases---for instance, you can use wget -c to download just the new portion that's been appended to a data collection or log file.

However, if the file is bigger on the server because it's been changed, as opposed to just appended to, you'll end up with a garbled file.  Wget has no way of verifying that the local file is really a valid prefix of the remote file....

This means that failure will result if an attempt is made to re-download a new version of an extension whose size has increased:

Code: [Select]

# Ensure e3.tcz is not in the download directory.
$ rm -f $(cat /opt/.tce_dir)/optional/e3.tcz*

# Create a simulated short previous version of e3.tcz.
$ echo === e3.tcz === >$(cat /opt/.tce_dir)/optional/e3.tcz

# Re-download e3.tcz.
$ tce-load -w e3.tcz
Downloading: e3.tcz
Connecting to distro.ibiblio.org (152.19.134.43:80)
e3.tcz               100% |*******************************| 16384   0:00:00 ETA
e3.tcz: FAILED
md5sum: WARNING: 1 of 1 computed checksums did NOT match
Error on e3.tcz

# Show that old and new versions were merged.
$ wc -c $(cat /opt/.tce_dir)/optional/e3.tcz
16384 /tmp/tce/optional/e3.tcz

$ head -1 $(cat /opt/.tce_dir)/optional/e3.tcz
=== e3.tcz ===

When working with extensions, it seems that it is only safe to use wget with the -c option when one knows that a download has been interrupted, which should be rare situation.  The usage of the -c option of wget in tce-audit, tce-fetch, tce-load, and tce-update should be reconsidered.