[Solved] zsync has trouble updating provides.db when I use my local mirror

[GNUser]

Hi Rich. I will investigate that. Thank you.

[mocore]

Hi GNUser
Actually, it was this statement that caught my eye:

Currently it must not use Transfer-Encoding: chunked (yes, I know that makes zsync non-compliant).

I thought maybe one of the simpler servers night be violating that requirement.

for the perpetually curious
"Understanding HTTP’s Transfer-Encoding: chunked" @ https://gist.github.com/nepsilon/3c5e2c1c5c2fece59a809217a66b8ea3

the body has each of its chunks separated with X\r\n, where X is the size of the chunk in hexadecimal.

[GNUser]

Maybe this has something to do with MIME types?

Yes, the problem did end up having to do with MIME types.

If I change this in my  go-httpd  server's source code:

Code: [Select]

http.ServeFile(w, r, path)
to this:

Code: [Select]

if strings.HasSuffix(path, ".db") || strings.HasSuffix(path, ".gz") || strings.HasSuffix(path, ".tcz") {
w.Header().Set("Content-Type", "application/octet-stream")
}
http.ServeFile(w, r, path)
...then all zsync-related problems go away.

Thread is solved.

[Rich]

Hi GNUser
So marked.

[GNUser]

I spoke too soon--problem happened again, so my "fix" in Reply #17 is no good

It seems this is a meta-bug that happens when the zsync client talks to some http servers (e.g., lighttpd, busybox httpd, golang server using net/http in the standard library).

I'm not optimistic I'm going to find a proper fix, but I did find two workarounds:

First and simplest workaround is to delete /usr/bin/zsync from the TCL machine. This causes both  provides.sh  and  tce-update  to fallback to using wget. The disadvantage is some wasted bandwidth.

Second solution is to use zsync2 (available at https://github.com/go-deltasync/zsync2). Note that the executable is called gozsync. For our purposes, it is a drop-in replacement for /usr/bin/zsync. Put gozsync in a directory in your PATH that appears before /usr/bin, for example, /home/tc/.local/bin, then:

Code: [Select]

$ cd /home/tc/.local/bin
$ ln -s gozsync zsyncNow gozsync/zsync2 takes precedence over the base system's zsync:

Code: [Select]

$ which -a zsync
/home/tc/.local/bin/zsync
/usr/bin/zsync
I did many tests and so far it seems zsync2/gozsync has no trouble talking to non-nginx http servers.

[GNUser]

I got in touch with Colin Phipps (zsync developer), who thinks my problem may be due to a bug in zsync 0.6 series. He's looking into it. I'll keep you all posted.

[Rich]

Hi GNUser

...

Code: [Select]

if strings.HasSuffix(path, ".db") || strings.HasSuffix(path, ".gz") || strings.HasSuffix(path, ".tcz") {
w.Header().Set("Content-Type", "application/octet-stream")
}
http.ServeFile(w, r, path) ...

When dealing with provides.db.gz, is HasSuffix  ".gz"  or  ".db.gz"?

[GNUser]

Hi Rich.

When dealing with provides.db.gz, is HasSuffix  ".gz"  or  ".db.gz"?

It works either way. See the specs at https://pkg.go.dev/strings#HasSuffix

Also, here is a simple program to test it, which I called test.go:

Code: [Select]

package main

import (
"fmt"
"strings"
)

func main() {
fmt.Println(strings.HasSuffix("provides.db.gz", ".gz"))
fmt.Println(strings.HasSuffix("provides.db.gz", ".db.gz"))
}
This is what happens when it is run:

Code: [Select]

$ go run test.go
true
true
Colin, the zsync developer, got back to me. He thinks this patch fixes the bug I reported: https://github.com/cph6/zsync/commit/4f044701

I'm going to test his patch tonight.

[Rich]

Hi GNUser

... It works either way. ...

Wow, I didn't see that coming.

... He thinks this patch fixes the bug I reported: https://github.com/cph6/zsync/commit/4f044701 ...

Looks promising.

[GNUser]

Hi Rich and Juanito. I compiled latest git of zsync (0.6 branch), which includes the patch. Problem is fixed

It would be nice if we could include this fixed zsync into the base system.

Juanito, do you want me to submit zsync for x86_64? I'm guessing it's too late for TCL17 unless you are planning a 17.1 release. Or it can go into TCL18.

    [Edit]: Made changes requested in reply #26.  Rich

[Juanito]

Yes, please submit - it can go in 17.1

[GNUser]

Hi Juanito. I submitted zsync executable for x86_64 base system plus an updated zsync extension (containing zsyncmake).

Hi Rich. Can you please fix my Reply #24? I said "I'm guessing it's too late for TCL16...16.1...TCL17" but what I meant was "I'm guessing it's too late for TCL17...17.1...TCL18." Thanks.

[Rich]

Hi GNUser

... Hi Rich. Can you please fix my Reply #24? ...

Done.