[Solved] secondary repository? script to create repository metadata?

[GNUser]

I have some custom extensions that I'd like to add to my local TCL repository so that multiple machines can have access to them. 

Back in my days of being a Devuan user, I'd create a small local repository containing only the custom packages, then would append a line to /etc/apt/sources.list. Would a similar strategy work in TCL? Alas, additional lines in /opt/tcemirror seem to have no effect (i.e., only the first line is parsed).

Also, if custom extensions were added to a local repository mirror or to local secondary repository, either way it would be burdensome to manually update/create all the repository metadata files:

Code: [Select]

info.lst
info.lst.gz
md5.db.gz
provides.db
provides.db.gz
provides.db.zsync
sizelist
sizelist.gz
tags.db.gz

So I'd like to ask two quick questions to my TCL friends, please:

1. Is there some way for  tce-load  to look in two different repositories (e.g., official repo and secondary local repo)?

2. Is there a script that parses  .tcz*  files in a directory and automatically creates/updates the repository metadata files listed above?

[Sashank999]

1) I think you can edit the mirrors.tcz extension's files to add your repository
Then tce-load the edited mirror.tcz and then select your mirror from the Apps Program.
2)I think we can create a script for that but I don't have enough knowledge about that.
Maybe this works :

Code: [Select]

#!/bin/bash
package=$1
gunzip info.lst.gz
echo "$package.info" >> info.lst
sort info.lst
gzip info.lst
md5sum $package >>  md5.db.gz
sort md5.db.gz
I don't know how the other files are structured 

[Rich]

Hi GNUser

... 1. Is there some way for  tce-load  to look in two different repositories (e.g., official repo and secondary local repo)? ...

It's clumsy, but you could script it. Create a script called  tce-load  in  ~/.local/bin.  Have that script call  /usr/bin/tce-load  twice
changing the contents of  /opt/tcemirror  in between calls. Since  ~/.local/bin  precedes  /usr/bin  in $PATH  running  tce-load
will default to your script.

A better way might be to do it from your local server. For instance, when  tce-load  requests  info.lst  from your server:
1. wget TinycoreRepo/Version/Arch/tcz/info.lst -O RemoteInfo.lst
2. cat RemoteInfo.lst LocalInfo.lst | sort > info.lst
3. Return the combined sorted info files to  tce-load.

When  tce-load  requests an extension:
1. grep extension.tcz LocalInfo.lst
2. If not found, fetch it from the Tinycore repo.

[Greg Erskine]

This might seem obvious but it took a while for the penny to drop for me. The tce* tools work great but they assume the repositories are really up-to-date mirrors of each other, hence the term "mirror" everywhere. duh.

If not, if you have an "modified" extension in your private repository with the same name as one in the official repository /opt/tcemirror has to be set correctly if you do an update for example. Once an extension is downloaded there is no indication from where it originally came from. I think it would be nice if the extension information file had the download URL in it.

So generally if /opt/tcemirror is pointing to the official repository and your try to use tce* info, search tools etc on a private repository extension you may get nothing or if it is a duplicate name you will get info about the wrong version of the extension. This situation is OK for the developers but confusing for users.

A couple of suggestions:

1. Warn the user if the "default" repository is not set in /opt/tcemirror
2. During a boot, reset /opt/tcemirror. It is probably also stored in mydata as well.
3. Add a prefix to the name of extensions in the private repository if possible.

There are scripts to generate the extra information in repositories.

[curaga]

Only one repo supported right now, yes. The server scripts aren't public currently.

[GNUser]

Thank you all for the helpful tips. With all your input I should easily be able to solve question/issue #1. I will try to solve it in my server. Short of that, I'll create a wrapper script for  tce-load  as Rich suggested.

The server scripts aren't public currently.

Well, that's awkward. I thought the TCL scripts were all open source (GPL). I'm sorry for asking.

If it's not ultra-top-secret, can you guys at least tell me which of the metadata files are the critical ones for "tce-load -wi foo.tcz" and "tce-update" to work? I wouldn't be using any other features (i.e., not using the Apps GUI, not browsing the .info files or extension contents via "tce" or "tce-ab"). If I know which files are required so that an extension can be installed and upgraded when appropriate, I can create a script myself.

[andyj]

This is going to sound obvious (for a reason), but if you unpack the files in question and learn what is in them I wouldn't think it would be a great challenge to create ones like them using your data.

[GNUser]

I thought asking for a list of files (for bare minimum functionality) wouldn't be a burden to someone who knows the answer, but fine. I'll just figure this out on my own.

Thanks guys!

[GNUser]

which of the metadata files are the critical ones for "tce-load -wi foo.tcz" and "tce-update" to work?

To answer my own question:
- I found that no metadata files are required for "tce-load -wi foo" to work--the only requirement is that  foo.tcz  and  foo.tcz.md5.txt  exist in the repository
- For  tce-update  to work, the repository must have a  .tcz.md5.txt  file for each extension as well as a  md5.db.gz  metadata file

So for minimum functionality (i.e., for "tce-load -wi foo" and 'tce-update" to work), the only required metadata file turns out to be md5.db.gz

[Rich]

Hi GNUser

...  can you guys at least tell me which of the metadata files are the critical ones for "tce-load -wi foo.tcz" and "tce-update" to work? ...

To the best of my knowledge they are used by the tools (tce-*, Apps, etc.) as follows:

1. info.lst.gz        Tells the tools what extensions are available.
2. md5.db.gz          Used by the tools to check for updates.
3. provides.db.gz     Used by the tools to find which extensions provide a specific file name.
4. provides.db.zsync  I think it's used to keep  provides.db  up to date. Don't know how it works.
5. sizelist.gz        Used by the tools to compute Extension+Dependencies size (Size tab in Apps).
6. tags.db.gz         Used by the tools to search for extensions by associated tags.

I wouldn't be using any other features (i.e., not using the Apps GUI, not browsing the .info files or extension contents via "tce" or "tce-ab"). ...

If you let your server handle it as I described and your custom extensions have all the required files, all features should work.

1. When one of the above files is requested, fetch it from the mirror and combine it with your local version before returning it.
2. When an extension is requested, return your custom version if available, otherwise fetch it from the mirror.
3. Since the tools handle file requests for dependency resolution:
    A. Stock dependencies don't need to be local. They would automatically be fetched from the mirror.
    B. You can override a stock extension by giving your custom extension the same name.
    C. Like B, you can create a custom dependency that overrides a stock one.

One other file that exists on the mirror (for each extension) is  EXTENSION.tcz.tree.

[Sashank999]

According to google, a .zsync file is used by a zsync program which is a partial/differential downloader.
If the local PC already has a provides.db.gz then zsync (in GUI Apps) compares it with the repo one and then downloads only new and modified parts of the repo's provides.db.gz and then appends those modified parts into the PC's provides.db.gz in the blocks's locations.
I think this is done to save the internet data of the User and also to save time instead of downloading whole provides.db.gz again.

[GNUser]

It's clumsy, but you could script it. Create a script called  tce-load  in  ~/.local/bin.  Have that script call  /usr/bin/tce-load  twice
changing the contents of  /opt/tcemirror  in between calls.

Thank you for this, Rich. It's probably the simplest solution. Here's a wrapper script that calls  /usr/bin/tce-load  a second time (using custom repo) only if the first call (using official repo) exits with an error:

Code: [Select]

#!/bin/sh

# user variables:
custom_repo="http://tiny-is-better.com/tinycorelinux-custom/"
main_repo="http://repo.tinycorelinux.net/"

echo "$main_repo" >/opt/tcemirror
if /usr/bin/tce-load "$@"; then
exit 0
else
echo "Trying custom repo now..."
echo "$custom_repo" >/opt/tcemirror
/usr/bin/tce-load "$@"
fi

[GNUser]

Wow, the information in Reply #9 is really handy. Thank you, Rich.

This thread can be marked as solved. Reply #11 (based on Rich's idea) solves first question in OP. Reply #8 makes it trivial to solve second question in OP.

[Rich]

Hi GNUser

... This thread can be marked as solved. ...

Done.

[andyj]

If you ever do want to create these files, my quick look came up with this:

info.lst.gz - a simple list of files

Code: [Select]

ls -1 *.tcz | sort -f | gzip > info.lst.gz
md5.db.gz - all the md5sums

Code: [Select]

cat *.tcz.md5.txt | sort -f -k2 | gzip > md5.db.gz
tags.db.gz - all the tags from the info files

Code: [Select]

grep "^Tags:" *.tcz.info | sed -E 's/\.info:Tags:[[:space:]]+/\t/' | sort -f -k2 | gzip > tags.db.gz might need more than one tab?

sizelist.gz - all the sizes of the .tcz files in bytes

Code: [Select]

ls --full-time *.tcz | awk '{print $9 " " $5}' | sort -f | gzip > sizelist.gz
provides.db.gz - all the files in each .tcz

Code: [Select]

for a in $(ls -1 *.tcz| sort -f); do echo $a >> provides.db; cat $a.list >> provides.db; echo >> provides.db; done; gzip provides.db
Since I don't have access to the secret ones, these are a clean-room implementation which you are free to use