Tiny Core Linux
dCore Import Debian Packages to Mountable SCE extensions => dCore X86 => Topic started by: Jason W on December 13, 2013, 03:46:23 PM
-
I have put in the ability to add extra repos to choose packages from. As it involves some major changes I have put the changes in a testing dCore.gz in the release candidates section.
http://tinycorelinux.net/5.x/x86/release_candidates/dCore.gz
To use this option, create a directory in /opt called debextra. Then add the mirror and repo in a file in that directory, one entry per file. /opt/debextra/repo1 might contain:
http://www.deb-multimedia.org main
Another file may contain the non-free backports, I think the main is also needed for the backports of that repo:
http://www.deb-multimedia.org wheezy-backports non-free
Or the wheezy-updates of one of the main debian repos:
http://ftp-mirror.internap.com/pub/debian wheezy-updates main
And so on. The package list files from the mirrors are parsed in the alphabetic or numeric order their files appear in that directory and the first match is chosen. And all extra repos have priority over the main one or the custom builds in the case of the same package name. The first entry is the mirror, the second is the distribution, and the third is the repo (main, non-free, contrib). In the absence of a distribution listed, wheezy is used. The ones I recommend outside of wheezy are the wheezy-backports and wheezy-proposed-updates, and wheezy-updates. Testing and unstable has a newer glibc so is not recommended.
Please test and report feedback.
-
hi jason w,
great enhancement!!!
i guess there is a little typo in the sources-line
http://www.deb-multimedia.org main
perhaps it should be
http://www.deb-multimedia.org wheezy main non-free
additionally we probably need the deb-multimedia-keyring.
nevertheless i have a problem to connect despite i use the deb-multimedia-keyring:
tc@box:~$ importsce -s minitube
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 8796160 local, fetched 0
Using the wheezy main repo of mirror http://www.deb-multimedia.org
wget: server returned error: HTTP/1.1 404 Not Found
bunzip2: invalid magic
Mirror http://www.deb-multimedia.org is unavailable
Using the wheezy-updates main repo of mirror http://ftp-mirror.internap.com/pub/debian
Using the wheezy main repo of mirror: http://ftp.us.debian.org/debian
minitube is not available as a package. Exiting...
the other two repos work fine, so far as i see.
thank you for that new great tool.
-
Though I am trying to follow the familiar Debian format of the sources.lst, it is not exactly the same. The entry
http://www.deb-multimedia.org main
is the same as using
http://www.deb-multimedia.org wheezy main
as wheezy is used in the absence of that field. I can remove that safeguard, though. However, main and non-free are two separate repos on a mirror. To include them both on one line looks simple, but would increase the complexity of the code to support this considerably. Main and non-free need to be listed on two separate mirror files.
Right now we require a repo that follows the Debian standard, as those that are not will have the same package listed in the same Packages.bz2 file, one for i386 and one for x86_64. And our package routine assumes one entry per Packages file, but there may be a way to work that in.
You are using the deb-multimedia keyring? I have been using the multimedia repo with no keyring. What is your exact entry in your deb-multimedia mirror file?
-
hi jascon w,
yes, i use the deb-multimedia-keyring.
tc@box:~$ df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 900M 84M 816M 10% /
tmpfs 500M 0 500M 0% /dev/shm
/dev/sdb2 3.5G 2.1G 1.4G 61% /mnt/sdb2
/dev/loop0 14M 14M 0 100% /tmp/tcloop/original-modules-3.8.13-tinycore
/dev/loop1 325M 325M 0 100% /tmp/tcloop/mydesktop
/dev/loop2 9.7M 9.7M 0 100% /tmp/tcloop/deb-multimedia-keyring
tc@box:/opt/debextra$ cat repo1
http://www.deb-multimedia.org wheezy main
tc@box:/opt/debextra$ cat repo2
http://www.deb-multimedia.org wheezy non-free
tc@box:/opt/debextra$ cat repo3
http://www.deb-multimedia.org wheezy-backports non-free
tc@box:/opt/debextra$ cat repo4
http://ftp-mirror.internap.com/pub/debian wheezy-updates main
tc@box:~$ importsce -s
Enter starting characters of package sought: minitube
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 8796160 local, fetched 0
Using the wheezy main repo of mirror http://www.deb-multimedia.org
Using the wheezy non-free repo of mirror http://www.deb-multimedia.org
wget: server returned error: HTTP/1.1 404 Not Found
bunzip2: invalid magic
Mirror http://www.deb-multimedia.org is unavailable
Using the wheezy-updates main repo of mirror http://ftp-mirror.internap.com/pub/debian
Using the wheezy main repo of mirror: http://ftp.us.debian.org/debian
minitube is not available as a package. Exiting...
thank you for your help.
-
hi jason w,
http://www.deb-multimedia.org wheezy-backports non-free
doesn't exit,
i guess you mean:
http://ftp.debian.org/debian/ wheezy-backports main ?
tc@box:~$ importsce -s minitube
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 8796160 local, fetched 0
Using the wheezy main repo of mirror http://www.deb-multimedia.org
Using the wheezy non-free repo of mirror http://www.deb-multimedia.org
Using the wheezy-backports main repo of mirror http://ftp.debian.org/debian/
Using the wheezy-updates main repo of mirror http://ftp-mirror.internap.com/pub/debian
Using the wheezy main repo of mirror: http://ftp.us.debian.org/debian
minitube is not available as a package. Exiting...
tc@box:/opt/debextra$ cat repo_wheezy_backports
http://ftp.debian.org/debian/ wheezy-backports main
tc@box:/opt/debextra$ cat repo_multimedia_main
http://www.deb-multimedia.org wheezy main
tc@box:/opt/debextra$ cat repo_multimedia_nonfree
http://www.deb-multimedia.org wheezy non-free
tc@box:/opt/debextra$ cat repo_wheezy_backports
http://ftp.debian.org/debian/ wheezy-backports main
tc@box:/opt/debextra$ cat repo_wheezy_updates
http://ftp-mirror.internap.com/pub/debian wheezy-updates main
-
The backports non-free doesn't exist after all, though it is just an example I will change it to something that does.
-
hi jason w,
now repos with names like
tc@box:/opt/debextra/repo1 numbered everything work fine for me.
repos with an own name-system like
tc@box:/opt/debextra/repo_wheezy_backports
work not so fine for the downloading-process of the packages,
because importsce is looking for the name convention as described above:
when importsce wants to fetch the packages, it is looking for repo1 or comparable.
-
There is one repo that does that, and says to put ./ ./ where the wheezy and main would go. But then there is the issue mentioned of having multiple arch's in the same Packages file, which at the moment is not supported. It should only be a matter of honing the awk script that searches the file, and if it can be done in a way that does not too much negatively affect performance then maybe we can go that route.
-
Oh, I see what you are saying. I did not know there is a limitation on the naming of the files in /opt aside from the usual unix standard - no spaces, no slashes, etc. I will look into it.
EDIT: I have no issue with a filename like below:
/opt/debextra/debian_multimedia_main.
-
hi jason w,
perhaps importupdatecheck needs also to be aware about this enhancement of the additional repos?
thank you for your help and commitment.
-
Ni netnomad, One day I may can work it in. It would require that the mirror files in /opt/debextra be the same as when the extension was imported though.
The bad thing about creating tools is that either development has to be held back to not break existing tools, or the tools get broken. For now I will focus on getting the ability to use other repos in place and polished, then perhaps importupdatecheck can be updated.
I am finishing being able to use Package files that have multiple arch's in them, which will support the use of other Debian mirrors that use one Package file for more than one arch. A bug or two to fix then I will upload that to be tested.
-
hi jason w,
that's fine, you make a great job!
thank you for every step in further development.
-
Ok, support for Packages.bz2 files that have multiple arch's in the same file has been added, along with some bugfixes. Please test, updated dCore.gz and iso are in the release candidates section:
http://repo.tinycorelinux.net/5.x/x86/release_candidates/
-
Reposted with a bugfix.
-
hi jason w,
tc@box:/opt/debextra$ cat repo1
http://www.deb-multimedia.org main
tc@box:/opt/debextra$ cat repo2
http://ftp.debian.org/debian/ wheezy-backports main
tc@box:/opt/debextra$ cat repo3
http://ftp-mirror.internap.com/pub/debian wheezy-updates main non-free
tc@box:/opt/debextra$ importsce -s minitube
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 8798208 local, fetched 0
Using the wheezy main repo of mirror http://www.deb-multimedia.org
Using the wheezy-backports main repo of mirror http://ftp.debian.org/debian/
Using the wheezy-updates main repo of mirror http://ftp-mirror.internap.com/pub/debian
Using the wheezy main repo of mirror: http://ftp.us.debian.org/debian
minitube is not available as a package. Exiting...
is the non-free ignored in repo3?
the rest seems to work fine for me :)
even importupdatecheck seems to work with the additional repos:
tc@box:~$ cd .TCE/sce && importupdatecheck mydesktop
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 8798208 local, fetched 0
Using the wheezy main repo of mirror http://www.deb-multimedia.org
Using the wheezy-backports main repo of mirror http://ftp.debian.org/debian/
Using the wheezy-updates main repo of mirror http://ftp-mirror.internap.com/pub/debian
Using the wheezy main repo of mirror: http://ftp.us.debian.org/debian
Checking for updated deb packages, please be patient..
Checking for updated dCore startup scripts, please be patient..
libvlc5
libvlccore5
vlc-data
libtag1-vanilla
libswscale2
libtag1c2a
libwbclient0
libsmbclient
libpostproc52
libdca0
vlc
vlc-nox
libjson0
libpulse0
libavformat53
libxvidcore4
libmp3lame0
libavcodec53
libavutil51
tzdata
libusb-1.0-0
fluxbox
openssh-server
openssh-client
ssh
tzdata
The above have package updates available in Debian, import mydesktop again to get them.
No dCore custom startup scripts inside mydesktop.sce have updates available.
the update-requests result from the updates in the additional repos.
-
There can only be one entry after the mirror and distro, main or non-free. Both can't be entered.
I will take a look at importupdatecheck next once things are working well in import, the update routine should not be working but I can use the same logic that is used in import to find the packages and md5sum.
-
Posted an update to dCore.gz and an updated iso dCore-5.13.12.19.iso to fix a bug in the md5 checking routine during import. Please download and test:
http://tinycorelinux.net/5.x/x86/release_candidates/
-
hi jason w,
i tested the last cut for a couple of occasions and everything works smooth and fine.
one little proposal that could be useful after an importupdatecheck:
could you clean up the results or the tmp-files... in the moment when i change entries in my repos under /opt/debextra the same result occurs...
i guess some tmp-files are left somewhere...
do you think it is very much effort to show in which repos updates are found?
with this information it's easier to decide whether an update is reasonable.
thank you for your help and your development.
-
Ok, I will add a cleanup for the /tmp/debinx.* files as they are re-downloaded every import session, that detail slipped by. And I will do the same for importupdatecheck.
I will update importupdatecheck and let it list which mirror an update is found.
-
sounds good :)
-
Mirror listing on updated packages during importupdatecheck now in place.
Also a bugfix in deb2sce in dCore.gz and the iso. Please re-download and test.
-
Just re-uploaded importupdatecheck with a bug fix in the md5 comparing routine.
-
hi jason w,
after my importupdatecheck i get a very long reply with multiple repo-remarks...
that's the end of the reply:
...
xserver-common Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main
xserver-xorg Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main
xserver-xorg-core Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main
wbar dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt
xorg Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main Debian wheezy main
Xprogs dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt
Xtc dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt
fltk-1.1.10 dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt dCore custom prebuilt
The above have package updates available with mirrors listed to the right, import mydesktop again to get them.
No dCore custom startup scripts inside mydesktop.sce have updates available.
tc@box:~$
-
I don't see that, and never have. What is the output of the command
cat /tmp/.importpkgtype
That file is supposed to be deleted if existing and the start and also at the end. Are you sure you have the latest updated importupdatecheck?
-
An observation - if you add mirrors in /opt/debextra, and those mirrors have their own version of a package that is available in wheezy main, and the package was originally imported from wheezy, then that package will show up as an available update. The extra mirrors take precedence over the main one as that is the desired behavior both in importing and update checking.
So to be accurate during an update check you need to keep the same mirror files that you import the sce with. But if you don't, or add mirrors that contain different versions of packages within the extension before an update check, then those packages will be listed as having available updates though the package in the repo you imported it from may have not been updated.
Example, ssh originally imported from wheezy and then the backport mirror was added before the check:
tc@box:~$ importupdatecheck ssh
#################### 100.0% 0.0 kBps DONE
verifying download...checksum matches OK
used 8798208 local, fetched 0
Using the wheezy-backports main repo of mirror http://deb-multimedia.org
Using the wheezy-backports main repo of mirror http://ftp.us.debian.org/debian
Using the wheezy main repo of mirror http://www.deb-multimedia.org
Using the ./ ./ repo of mirror http://www.pvv.ntnu.no/~knuta/xmms/squeeze
Using the wheezy main repo of mirror: http://ftp.us.debian.org/debian
Checking for updated deb packages, please be patient..
Checking for updated dCore startup scripts, please be patient..
openssh-server http://ftp.us.debian.org/debian wheezy-backports main
openssh-client http://ftp.us.debian.org/debian wheezy-backports main
ssh http://ftp.us.debian.org/debian wheezy-backports main
The above have package updates available with mirrors listed to the right, import ssh again to get them.
No dCore custom startup scripts inside ssh.sce have updates available.
tc@box:~$
-
Posted an updated dCore.gz and iso with a cosmetic change in debGetEnv, please test.
http://repo.tinycorelinux.net/5.x/x86/release_candidates/
-
works good for me :)
-
Reposted with ability to fetch Packages.gz as well as Packages.bz2, some mirrors only have the .gz.
-
Reposted dCore.gz and related iso to release_candidates directory, this time with a removing of an unneeded md5sum file creation that was used in the previous "secure" import mode. The md5 checking of existing debs is now done within deb2sce during each import session.
Please help test, and I hope to make this cut a formal release in the coming days for both x86 and armv7 ports. I have been using and testing armv7 now that my hardware is in place, and all is working as expected.
-
hi jason w,
looks pretty good, all work smooth and fine for me.
thank you for your work, happy hacking :)