3.x repo observations

[Kingdomcome]

Ive rewritten my repo mirroring script to use zsync in hopes of conserving bandwidth.  Since doing so I have found some problems with the 3.x repo:

1) Some extensions are missing their .zsync file

Code: [Select]

alsa-modules-2.6.33.3-tinycore64.tcz
alsa-modules-2.6.33.3-tinycore.tcz
base-dev.tcz
binutils.tcz
bluetooth-2.6.33.3-tinycore64.tcz
bluetooth-2.6.33.3-tinycore.tcz
cpufreq-2.6.33.3-tinycore64.tcz
cpufreq-2.6.33.3-tinycore.tcz
doxygen-doc.tcz
filesystems-2.6.33.3-tinycore64.tcz
filesystems-2.6.33.3-tinycore.tcz
firewire-2.6.33.3-tinycore64.tcz
firewire-2.6.33.3-tinycore.tcz
firmware-iwlwifi.tcz
firmware-libertas.tcz
firmware.tcz
firmware-wimax.tcz
gcc_libs.tcz
gmp.tcz
graphics-2.6.33.3-tinycore64.tcz
graphics-2.6.33.3-tinycore.tcz
gs.tcz
gtk2_prefs.tcz
hwmon-2.6.33.3-tinycore64.tcz
hwmon-2.6.33.3-tinycore.tcz
i2c-2.6.33.3-tinycore64.tcz
i2c-2.6.33.3-tinycore.tcz
icu-dev.tcz
icu-doc.tcz
input-joystick-2.6.33.3-tinycore64.tcz
input-joystick-2.6.33.3-tinycore.tcz
input-tablet-touchscreen-2.6.33.3-tinycore64.tcz
input-tablet-touchscreen-2.6.33.3-tinycore.tcz
ipv6-2.6.33.3-tinycore64.tcz
ipv6-2.6.33.3-tinycore.tcz
irda-2.6.33.3-tinycore64.tcz
irda-2.6.33.3-tinycore.tcz
kvm-2.6.33.3-tinycore64.tcz
kvm-2.6.33.3-tinycore.tcz
leds-2.6.33.3-tinycore64.tcz
leds-2.6.33.3-tinycore.tcz
libglademm.tcz
libpulseaudio.tcz
libXtst.tcz
mpfr.tcz
mtd-2.6.33.3-tinycore64.tcz
mtd-2.6.33.3-tinycore.tcz
net-bridging-2.6.33.3-tinycore64.tcz
net-bridging-2.6.33.3-tinycore.tcz
netfilter-2.6.33.3-tinycore64.tcz
netfilter-2.6.33.3-tinycore.tcz
pavucontrol.tcz
pulseaudio-dev.tcz
pulseaudio.tcz
raid-dm-2.6.33.3-tinycore64.tcz
raid-dm-2.6.33.3-tinycore.tcz
scsi-2.6.33.3-tinycore64.tcz
scsi-2.6.33.3-tinycore.tcz
silgraphite-dev.tcz
silgraphite.tcz
usb-serial-2.6.33.3-tinycore64.tcz
usb-serial-2.6.33.3-tinycore.tcz
v4l-dvb-2.6.33.3-tinycore64.tcz
v4l-dvb-2.6.33.3-tinycore.tcz
wimax-2.6.33.3-tinycore64.tcz
wimax-2.6.33.3-tinycore.tcz
wireless-2.6.33.3-tinycore64.tcz
wireless-2.6.33.3-tinycore.tcz
Xfbdev.tcz
Xvesa.tcz
2)  Some extensions fail to download through zsync although wget downloads fine (Unknown cause, but very reproducible)

Code: [Select]

automake.tcz
bison.tcz
bluez.tcz
cloog.tcz
coreutils.tcz
e2fsprogs_apps.tcz
file.tcz
findutils.tcz
gcc.tcz
grep.tcz
gst-plugins-good.tcz
help2man.tcz
hplip.tcz
icu.tcz
libcanberra-dev.tcz
libcanberra.tcz
libmtp-dev.tcz
libmtp.tcz
m4.tcz
patch.tcz
ppl.tcz
pyqt.tcz
sed.tcz
sip.tcz
3)  A stray tczl has snuck in

Code: [Select]

help2man.tczl

[curaga]

Thanks, I've uploaded zsync files for the kernel and firmware extensions. Though if you already used rsync, it should be no change.

On those that won't download, seems that they have out-of-sync zsync files (tested patch.tcz).

[Kingdomcome]

Previously I was using gnu wget with the mirroring option.  This is my first experience with a syncing app.  Although it may take a little longer to complete, I believe that the possible bandwidth savings will be worth it, especially since the script runs on a dedicated server box and during the night.  Thanks for the quick fixes.

[Jason W]

I am in the process of getting my 3.x mirror in place, and after that it will be very easy for me to fix and upload any broken zsync files.   Maybe by tomorrow I can fix that second list that apparently has bad zsync files.

[Kingdomcome]

Jason,

  It appears that the problems I saw with the second list have disappeared.  Im not sure if anything was changed on ibiblio or not. but a resync I performed manually after Curaga's post proved totally successful.

[Jason W]

I am still having trouble with the second set.  Tonight I will have my mirror, so corrections will be easy when needed.

[Jason W]

I may be using the wrong test command.  Will look more into my result later.

[Jason W]

I am seeing that some work and some don't.  Will fix soon as I can.

EDIT - Confirmed, remaking .zsync files fixes the issue with the extensions I tested.  Will remake them.

[Juanito]

On those that won't download, seems that they have out-of-sync zsync files (tested patch.tcz).

That I would be me, I updated quite a few extensions in the 3.x repo - patch being one of them...

[Jason W]

Mirror is complete, I will remake the zsync files tonight. 

[Kingdomcome]

Just as a follow up, Ive written a little integrity check script that checks the 2.x and 3.x repos on ibiblio.  Here are the results:

Code: [Select]

Fri May 21 18:52:08 EDT 2010
Missing infos in 2.x repo

Missing lists in 2.x repo
pigz.tcz.list

Missing md5s in 2.x repo

Missing zsyncs in 2.x repo
blueman.tcz.zsync
cutecom.tcz.zsync
doxygen-doc.tcz.zsync
doxygen-gui.tcz.zsync
GConf-doc.tcz.zsync
gnome-vfsmm.tcz.zsync
icu-40.tcz.zsync
icu-44-dev.tcz.zsync
icu-44-doc.tcz.zsync
icu-44.tcz.zsync
libatomic_ops.tcz.zsync
libglademm.tcz.zsync
libgnomecanvasmm.tcz.zsync
libgnomemm.tcz.zsync
obex-data-server.tcz.zsync
pigz.tcz.zsync
pyrex.tcz.zsync
silgraphite-dev.tcz.zsync
silgraphite.tcz.zsync
star-cups-driver.tcz.zsync

Out of sync zsyncs in 2.x repo
bluez.tcz
dbus.tcz
doxygen.tcz
hplip.tcz
libmtp-dev.tcz
libmtp.tcz
pyqt.tcz
sip.tcz


Missing infos in 3.x repo

Missing lists in 3.x repo

Missing md5s in 3.x repo

Missing zsyncs in 3.x repo
base-dev.tcz.zsync
doxygen-doc.tcz.zsync
gcc_libs.tcz.zsync
GConf-doc.tcz.zsync
gmp.tcz.zsync
gs.tcz.zsync
gtk2_prefs.tcz.zsync
icu-dev.tcz.zsync
icu-doc.tcz.zsync
libglademm.tcz.zsync
libpulseaudio.tcz.zsync
libXtst.tcz.zsync
mpfr.tcz.zsync
pavucontrol.tcz.zsync
pulseaudio-dev.tcz.zsync
silgraphite-dev.tcz.zsync
silgraphite.tcz.zsync
star-cups-driver.tcz.zsync
Xfbdev.tcz.zsync
Xvesa.tcz.zsync

Out of sync zsyncs in 3.x repo
automake.tcz
binutils.tcz
bison.tcz
bluez.tcz
cloog.tcz
coreutils.tcz
doxygen.tcz
e2fsprogs_apps.tcz
findutils.tcz
gcc.tcz
grep.tcz
gst-plugins-good.tcz
help2man.tcz
hplip.tcz
icu.tcz
libcanberra-dev.tcz
libcanberra.tcz
libmtp-dev.tcz
libmtp.tcz
m4.tcz
patch.tcz
ppl.tcz
pyqt.tcz
sed.tcz
sip.tcz

[Jason W]

Thanks.  I will follow up and make corrections.

[Jason W]

Fixed those reported, thanks.