zsync has trouble updating provides.db when I use my local mirror

[GNUser]

Hi Rich. I use your wonderful  provides.sh  that's part of the base system. I've noticed that  tce/provides.db  can be updated normally when I'm using the official mirror, but not when I'm using my local mirror.

I made a small edit to  provides.sh  (removed the  -q  flag from the zsync command) so that it's more verbose. I also changed my real domain to "mydomain.net" for privacy (I'll send you my real domain in a PM):

Code: [Select]

$ sh -x /usr/bin/provides.sh bin/go
+ . /etc/init.d/tc-functions
+ '[' -f /proc/cmdline ]
+ CMDLINE=
+ cat /proc/cmdline
+ CMDLINE=' BOOT_IMAGE=/boot/vmlinuz64 tce=sda1 opt=sda1 tz=EST+5EDT,M3.2.0/2,M11.1.0/2 syslog lang=en_US.UTF-8 user=bruno loglevel=0 host=blaptop multivt text wayland noembed'
+ echo -e '\r\033[K'
'
+ echo -e '\033[1;31m'
+ RED=''
+ echo -e '\033[1;32m'
+ GREEN=''
+ echo -e '\033[1;33m'
+ YELLOW=''
+ echo -e '\033[1;34m'
+ BLUE=''
+ echo -e '\033[1;35m'
+ MAGENTA=''
+ echo -e '\033[1;36m'
+ CYAN=''
+ echo -e '\033[1;37m'
+ WHITE=''
+ echo -e '\033[0;39m'
+ NORMAL=''
+ useBusybox
+ alias 'ar=busybox ar'
+ alias 'awk=busybox awk'
+ alias 'cat=busybox cat'
+ alias 'clear=busybox clear'
+ alias 'cp=busybox cp'
+ alias 'cpio=busybox cpio'
+ alias 'cut=busybox cut'
+ alias 'date=busybox date'
+ alias 'dc=busybox dc'
+ alias 'df=busybox df'
+ alias 'du=busybox du'
+ alias 'depmod=busybox depmod'
+ alias 'expr=busybox expr'
+ alias 'fdisk=busybox fdisk'
+ alias 'fold=busybox fold'
+ alias 'grep=busybox grep'
+ alias 'gunzip=busybox gunzip'
+ alias 'hostname=busybox hostname'
+ alias 'kill=busybox kill'
+ alias 'killall=busybox killall'
+ alias 'less=busybox less'
+ alias 'ls=busybox ls'
+ alias 'md5sum=busybox md5sum'
+ alias 'ps=busybox ps'
+ alias 'sed=busybox sed'
+ alias 'sort=busybox sort'
+ alias 'swapoff=busybox swapoff'
+ alias 'swapon=busybox swapon'
+ alias 'sync=busybox sync'
+ alias 'tar=busybox tar'
+ alias 'timeout=busybox timeout'
+ alias 'wc=busybox wc'
+ alias 'wget=busybox wget'
+ alias 'mount=busybox.suid mount'
+ alias 'umount=busybox.suid umount'
+ VERSION='Version 0.3 Nov 25, 2024'
+ MMINoverride=
+ '[' -e /usr/local/bin/awk ]
+ TCEDIR=/etc/sysconfig/tcedir
+ DB=provides.db
+ LIST=/etc/sysconfig/tcedir/provides.db
+ Exact=0
+ NZ=1
+ FullPaths=0
+ MMIN=+15
+ TARGET=
+ TARGET=bin/go
+ SanitizeTarget
+ TARGET=bin/go
+ TARGET='bin\/go'
+ TARGET='bin\/go'
+ TARGET='bin\/go'
+ TARGET='bin\/go'
+ TARGET='bin\/go'
+ TARGET='bin\/go'
+ TARGET='bin\/go'
+ TARGET='bin\/go'
+ continue
+ StripTarget -nz
+ local '__SW=-nz'
+ TARGET='bin\/go'
+ TARGET='bin\/go'
+ TARGET='bin\/go'
+ '[' 'bin\/go' '==' -nz ]
+ StripTarget -nz
+ local '__SW=-nz'
+ TARGET='bin\/go'
+ TARGET='bin\/go'
+ TARGET='bin\/go'
+ '[' 'bin\/go' '==' -nz ]
+ StripTarget -f
+ local '__SW=-f'
+ TARGET='bin\/go'
+ TARGET='bin\/go'
+ TARGET='bin\/go'
+ '[' 'bin\/go' '==' -f ]
+ StripTarget -f
+ local '__SW=-f'
+ TARGET='bin\/go'
+ TARGET='bin\/go'
+ TARGET='bin\/go'
+ '[' 'bin\/go' '==' -f ]
+ echo 'bin\/go'
+ TARGET='bin\/go'
+ '[' 1 -eq 1 ]
+ UpdateProvidesDB
+ IsSignedInteger
+ local '__NUM='
+ '[' 0 -lt 2 ]
+ echo
+ return
+ MMINoverride=
+ '[' -n  ]
+ '[' -f /etc/sysconfig/tcedir/provides.db ]
+ find /etc/sysconfig/tcedir/provides.db -mmin +15
+ '[' -z /etc/sysconfig/tcedir/provides.db ]
+ /bin/ping -A -W 1 -c 2 8.8.8.8
+ getMirror
+ BUILD=
+ '[' -z  ]
+ getBuild
+ uname -m
+ BUILD=x86_64
+ '[' -f /lib/ld-linux-x86-64.so.2 ]
+ echo x86_64
+ BUILD=x86_64
+ read MIRROR
+ MIRRORPROTO=http
+ getMajorVer
+ getFullVer
+ grep '^VERSION_ID=' /etc/os-release
+ verid='VERSION_ID=17.0'
+ echo 17.0
+ fullver=17.0
+ echo 17
+ MIRROR=http://mydomain.net/tinycorelinux/17.x/x86_64/tcz
+ '[' http '!=' http ]
+ '[' http '==' https ]
+ '[' http '==' http ]
+ which zsync
+ '[' -n /usr/bin/zsync ]
+ USEZSYNC=yes
+ cd /etc/sysconfig/tcedir
+ '[' -n yes ]
+ zsync -i /etc/sysconfig/tcedir/provides.db http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.zsync
#################### 100.0% 0.0 kBps DONE   

reading seed file /etc/sysconfig/tcedir/provides.db: ***************************Read /etc/sysconfig/tcedir/provides.db. Target 98.1% complete.     
Read provides.db. Target 98.1% complete.      ******************
Read provides.db.part. Target 98.1% complete.      ******************
downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db:
###################- 98.1%^C
It seems  zsync  gets stuck in a loop at 98.1%. Server log shows it serving  provides.db  over and over again. Does the above suggest anything to you? I'm kind of stuck because I know next to nothing about zsync.

[GNUser]

By using  curl  I can confirm that  provides.db.gz  is present on my server and that the server can handle partial content (note the 206 http response):

Code: [Select]

$ curl -v --range 0-1000 http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.gz
*   Trying 76.117.245.38:80...
* Connected to mydomain.net (76.117.245.38) port 80
> GET /tinycorelinux/17.x/x86_64/tcz/provides.db.gz HTTP/1.1
> Host: mydomain.net
> Range: bytes=0-1000
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 206 Partial Content
< Accept-Ranges: bytes
< Content-Length: 1001
< Content-Range: bytes 0-1000/3253446
< Content-Type: application/x-gzip
< Last-Modified: Tue, 19 May 2026 08:15:43 GMT
< Date: Wed, 20 May 2026 15:06:27 GMT
<
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
* Failure writing output to destination
* Closing connection

[GNUser]

I tried running  lighttpd  as well as  busybox httpd  on my server rather than my homebrewed  go-httpd  . I was happy to discover that this problem (provides.sh hanging at the zsync step when provides.db is due for a refresh) occurs regardless of which httpd software I use.

So I think the problem is likely one of two things:

1. One or more files in my local mirror relevant to  provides.db  is/are missing/defective/misplaced

or

2. A minimalistic http server needs some extra configuration in order to successfully deal with a zsync client

Two quick questions:

1. On the official repo, are all files related to  provides.db  located in the tinycorelinux/[TCLversion]/[architecture]/tcz directory?

2. Is there any special configuration being used on official TCL repo's http server related to zsync?

[GNUser]

If I run  nginx  on the server, problem goes away. This eliminates hypothesis #1 in my previous post.

Problem must have something to do with this:

- A minimalistic http server needs some extra configuration in order to successfully deal with a zsync client

Maybe this has something to do with MIME types?

[Rich]

Hi GNUser

... 1. On the official repo, are all files related to  provides.db  located in the tinycorelinux/[TCLversion]/[architecture]/tcz directory?

Yes they are. And their sizes match your local repo.

[Rich]

Hi GNUser
Running busybox and GNU zsync at the command line against provides.db:

Code: [Select]

tc@E310:~/Scripting/Provides$ /usr/bin/zsync -i "provides.db" "http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.zsync"
#################### 100.0% 0.0 kBps DONE   

Read provides.db. Target 100.0% complete.      *****************
verifying download...checksum matches OK
used 33488896 local, fetched 0
tc@E310:~/Scripting/Provides$ /usr/local/bin/zsync -i "provides.db" "http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.zsync"
#################### 100.0% 0.0 kBps DONE   

Read provides.db. Target 100.0% complete.      *****************
verifying download...checksum matches OK
used 33488896 local, fetched 0
tc@E310:~/Scripting/Provides$
Running busybox and GNU zsync at the command line against provides.db.gz:

Code: [Select]

tc@E310:~/Scripting/Provides$ /usr/bin/zsync -i "provides.db.gz" "http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.zsync"
#################### 100.0% 0.0 kBps DONE   

Read provides.db.gz. Target 100.0% complete.      *******************
verifying download...checksum matches OK
used 33488896 local, fetched 0
tc@E310:~/Scripting/Provides$ /usr/local/bin/zsync -i "provides.db.gz" "http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.zsync"
#################### 100.0% 0.0 kBps DONE   

Read provides.db.gz. Target 100.0% complete.      *******************
verifying download...checksum matches OK
used 33488896 local, fetched 0
tc@E310:~/Scripting/Provides$
When I copied my TC10 provides.db in place of yours, this happened:

Code: [Select]

tc@E310:~/Scripting/Provides$ cp /etc/sysconfig/tcedir/provides.db .
cp: overwrite './provides.db'? y
tc@E310:~/Scripting/Provides$ /usr/bin/zsync -i "provides.db" "http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.zsync"
#################### 100.0% 0.0 kBps DONE   

Read provides.db. Target 8.3% complete.      *****
downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.gz:
#------------------- 8.3% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.gz:
#------------------- 8.3% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.gz:
#------------------- 8.3% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.gz:
#------------------- 8.3% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.gz:
#------------------- 8.3% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.gz:
#------------------- 8.3% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.gz:
#------------------- 8.3% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.gz:
#------------------- 8.3% 0.0 kBps         

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.gz:
#------------------- 8.3% 0.0 kBps

downloading from http://mydomain.net/tinycorelinux/17.x/x86_64/tcz/provides.db.gz:^CI had to Ctrl-C to stop it.

[GNUser]

Hi Rich. Interesting results. Does this suggest a solution to you?
I'm going to try using GNU zsync to see what happens.

EDIT: No luck with GNU zsync. If it's not  nginx  running on the server, zsync client enters an infinite loop. How frustrating. This is the only problem I've encountered since switching from nginx to something more lightweight.

[Rich]

Hi GNUser
When I tried running the script on my TC10 system against your repo, the
CPU usage shot up and the script never exited (endless loop).

When I ran zsync from the command line against the provides.db
that I downloaded from your repo, it was happy.

When I ran zsync from the command line against the provides.db
from my TC10 system, it entered an endless loop of partial downloads.

My conclusion is, the provides.db you are checking does not match
the provides.db.zsync file.

Make sure your  /etc/sysconfig/tcedir/provides.db  is the same as your repo copy.

Make sure the  provides.db.zsync  you are referencing is coming from your repo, and
not someplace else.

... This is the only problem I've encountered since switching from nginx to something more lightweight.

Maybe a stray leftover config file or setting causing mischief?

[GNUser]

Make sure your  /etc/sysconfig/tcedir/provides.db  is the same as your repo copy.

Isn't it zsync's job to check that--and update client's  provides.db  if it doesn't match repo copy? It sounds like the only time things are working well in your tests is when zsync has nothing to update because your  provides.db  is already current.

Make sure the  provides.db.zsync  you are referencing is coming from your repo, and
not someplace else.

/opt/tcemirror contains one line, which is the correct url for my repo. Shouldn't that be sufficient to ensure that the  zsync  command in  provides.sh  is grabbing the correct provides.db.zsync?

P.S. The most perplexing thing of all is that if local mirror is served by  nginx  on the server (vs. a more minimalistic httpd), all problems go away without changing any files in the local mirror and without changing any files on my laptop/zsync client.

[Rich]

Hi GNUser

Make sure your  /etc/sysconfig/tcedir/provides.db  is the same as your repo copy.

Isn't it zsync's job to check that--and update client's  provides.db  if it doesn't match repo copy? ...

My knowledge of how zsync works ranges from slim to none.

When running an update on the repo, this message does show up:

Code: [Select]

No URL given, so I am including a relative URL in the .zsync file - you must keep the file being served and the .zsync in the same public directory.

That seems to suggest there is no mechanism to check if the zsync file matches
the file being checked.

...

Make sure the  provides.db.zsync  you are referencing is coming from your repo, and
not someplace else.

/opt/tcemirror contains one line, which is the correct url for my repo. Shouldn't that be sufficient to ensure that the  zsync  command in  provides.sh  is grabbing the correct provides.db.zsync?

It should. But since I can't see your setup, so it was something to check.

[GNUser]

My knowledge of how zsync works ranges from slim to none.

Haha, so it's a "blind leading the blind" situation

zsync is like rsync, but for a single file. It downloads just a delta to bring a local file up to date with a remote file. I've just told you 100% of what I know about zsync.

I think the point of using zsync with  provides.db  is to avoid re-downloading the entire provides.db file, which for TCL standards is large (at around 30 MB).

This problem is not a big deal. I don't think it's important enough for me to go back to nginx or continue troubleshooting (I'm not optimistic about finding a solution).

I'll resort to a workaround: I'm going to use a modified version of  provides.sh  that simply re-downloads the entire  provides.db  when  provides.db  is deemed to be too old.