Sources and Build Scripts x86/x64

[nick65go]

@CentralWare: for ArchLinux, first you go to https://archlinux.org/packages/
in the field "Keyword" just type "mc", for Midnight Commander, to filter the results.
then in the field "Name" just click on it ("mc") and it is done.
if you want the source package then now click (at top left)  on "Source files" and you arrived at its PKGBUILD:
https://gitlab.archlinux.org/archlinux/packaging/packages/mc/-/blob/main/PKGBUILD?ref_type=heads

[yvs]

+1 for something like aur PKGBUILD, it's comfy and familiar

[CentralWare]

@CentralWare: for ArchLinux...

Arch is already complete. I have a copy here (62GB!!!) on one of the file servers of the sources and community (5,000+ builds) and for just build scripts...  this puppy is huge!

Alpine's APKBUILD may come in handy for those packages which weren't on Arch, or failed using Arch's scripts, or failed with our own build scripts, etc. (There's too much uncertain navigating to make it worthwhile in an automated fashion - which is our endgame. It's also very likely why Alpine doesn't share their GIT openly like Arch does (must be accessed via their web interface) BUT...  I'm grateful none the less!)  I haven't tried to do a git clone of Alpine yet (it's being scanned right now) but to have those available without having to traverse their interface could be very worthwhile, too.

Update: Alpine is flagged to be used "as needed" - their tested script list is a lot smaller than I had imagined. It's broken down into three categories, MAIN, COMMUNITY and TESTING where it looks like more than half of their extensions are in testing.  (Midnight Commander, though, is part of their main releases!!!!)  Don't get me wrong -- it's still very valuable and I'm grateful to have access to it, but "testing" tells me possibly "unknown status." so they're seen as "notes" more than scripts.

@yvs: PKGBUILD itself is a BASH system which we'll likely avoid specifics pertaining to bash, but the command structure will possibly be similar. The goal here is to avoid focused commands/functions/etc. that are likely to fail under different shells and to use third party apps (like grep, awk, etc.) as little as possible without causing bloat or lag by scripting such. For example, Bash allows arrays of text to be included in a command such as ${item1, item2, item3} whereas this fails in Ash. The way we have to write it, then, is in a fashion where it'll work in both; in this case it may mean having three separate commands, which takes longer to write but should not take longer to compute - or have very little extra in computing time. I want the end product to function in most any environment on most any 'nix foundation.

LMAO...  Alpine's APKBUILD scripts are almost identical to Arch's. In fact, if it weren't for changed names here and there, I'd say they were the same source.
The only flaw between Arch and Alpine is that neither of them have version management.

For Example: filename-2.3.4.extension may have a dependency of OpenSSH Version 1.1.0 through 1.1.1-J whereas OpenSSH 1.1.1-M and newer won't work for the release. Both Alpine and Arch don't look as though they take versioning into account, which means some things are bound to break...  which is fine...  it's why this project was launched!  We'll likely use Tiny Core's version history to create ball-park entries for this (ie: To build MC version 1.2.3.4 which was found, let's say, in TCL v10.x we can check out the dependencies found in that release and see what those versions were at that time.  It's not ideal...  but it should work for now and can be automated!)

[yvs]

The goal here is to avoid focused commands/functions/etc. that are likely to fail under different shells and to use third party apps (like grep, awk, etc.) as little as possible without causing bloat or lag by scripting such.

in a totally declarative way? Kinda tradeoff between predictability and flexibility. There could be solutions between, something like OBS services.

I want the end product to function in most any environment on most any 'nix foundation.

Usually it's enough to have a working solution for a target distro. If that's about the universal solution... idk, I've not seen any popular ones (maybe because the cost of tradeoffs on a way to achieve that).

For Example: filename-2.3.4.extension may have a dependency of OpenSSH Version 1.1.0 through 1.1.1-J whereas OpenSSH 1.1.1-M and newer won't work for the release.

PKGBUILD #4.1
Version restrictions can be specified with comparison operators; if multiple restrictions are needed, the dependency can be repeated for each

[CentralWare]

PKGBUILD #4.1
Version restrictions can be specified with comparison operators; if multiple restrictions are needed, the dependency can be repeated for each

Based on the limited number of build scripts I've looked at so far with Arch and Alpine, I haven't come across anything (yet) that looks restrictive, but that's not saying they won't show up on the trials this week! I'm also looking forward to seeing how they handle drivers and other similar kernel-interactive packages!

Complete!  The local archives for both Arch and Alpine have finished!  Now we can start having some fun!
PKGBUILD...  for a program that's somewhat famous for building and compiling software...  it's rather elusive where the source code FOR PKGBUILD lives! I was trying to determine whether PKGBUILD was a script system or a binary...  still cannot say for certain, but I was able to find pkgbuild-assistant which was C source, so I'm going to just assume pkgbuild is also and leave well enough alone.  (Compiled binaries tells me there's different releases for each platform they're used on whereas scripts "tend" to be more universal... so we won't be borrowing anything directly from pkgbuild, BUT if I can find the source somewhere, I can look within to get command-line calls and internal logic without having to actually install Arch/Alpine just to have access to it.)

[Rich]

Hi CentralWare

... PKGBUILD...  for a program that's somewhat famous for building and compiling software...  it's rather elusive where the source code FOR PKGBUILD lives! I was trying to determine whether PKGBUILD was a script system or a binary...  still cannot say for certain, ...

Have you seen these:
https://wiki.archlinux.org/title/PKGBUILD
https://wiki.archlinux.org/title/Makepkg

I only briefly skimmed those Wiki links, but the impression I got
was that  PKGBUILD  serves as a guide for  makepkg.  Kind of
like  Makefile  for  make.

[Rich]

Hi CentralWare
Also found this  PKGBUILD  template on github:
https://gist.github.com/valeth/f94e42cd9ecf76034ef7

[CNK]

I don't know if PKGBUILD is better suited to making TC extensions, but I described using Pkgsrc in TC a year ago. It's a similar sort of thing for doing scripted builds from source code (which break too often for my taste when things change).

[CentralWare]

I don't know if PKGBUILD is better suited...

When it comes to change, I don't think there is such a thing as a "best way" or even a "better way" but here's my take on it:
PKGBUILD, PKGsrc, etc. all achieve their destinations based on assumptions that nothing has changed. For a good number of extensions, this likely holds true.
MY goal here is to take (approx) 6,000 individual extensions and run "logical" build scripts which eventually expand into AI (Automated Ingenuity - the other one doesn't really exist )

Example: (I'm using dropbear because it currently doesn't require secondary dependencies)
Let's say 01-01-2024 dropbear_1.2.3.src.tar.gz compiles perfectly using glibc_a.b.c and gcc_d.e.f, make_1.2.3, etc. - we then have a documented history of successes.
On 01-01-2025 our builder throws up a flag and tells us the newest stable release doesn't compile using newer ingredients as a foundation. We compare notes to find the sore spot(s):

Dropbear's build script (its filename) is sent to the Builder Network with a Research flag, meaning if/when it fails compiling attempt one of a dozen or more combinations to see if the NEW dropbear compiles with a slightly downgraded foundation, or if the OLDER Dropbear compiles with our newest foundation members.

• First, attempt to build the most recent successful, stable dropbear using the newer foundation items. Leave notes/findings for a developer to investigate once successful
• IF FAILED, downgrade each foundation item until we're successful again - leave notes for a developer to investigate

This is basically what the maintainer would be responsible for - updating scripts when change demands it of us; in this case, given a head start and notes as to what was already attempted and what the results were from those attempts.

PRO: With the automation hardware system we're building, a ton of maintainer hours is expected to be alleviated.  (This also costs $, so we're working on methods to cut this as well)
CON: There are still going to be apps which require hand-holding every so often to keep things compliant with our system. It's not yet "completely" hands free! It may never be.

PRO: The final stage of the project will be us creating a peer-to-peer-like volunteer network to assist with the cost of compiling (behind the scenes)
CON: ...assuming we can forcefully maintain security procedures as hacked extensions defeat us doing any of this in the first place.


@Rich: Yes, I have seen those pages/links and have already torn apart a hand-made template based on live source packages.  There are exported variables I need to find out how they get from PKG to build scripts or however it works within arch/alpine/etc. (Thank you, though - every angle is appreciated!)
Example: $CHOST, $CBUILD, $CFLAGS,  $CPPFLAGS and others are randomly spotted throughout some of their extensions, but PKG looks to be what creates these exports, so knowing where some of these defaults come from (and what their preferred defaults would be) would help me "see" what they were thinking when PKG was created.  I have no clue as to whether there are cross-compilings going on or whether all of these scripts are being executed on end-user machines only...  and being as I've never had reason to tinker with Arch/Alpine I've not had reason to break it apart before.

Here's a kicker: "A PKGBUILD is a Bash shell script containing the build information required by Arch Linux packages."  Terminology on G00gle is the same way...  discussing the shell script(s) AS if they WERE pkgbuilds, so I'm going to have to launch a virtual of Arch/Alpine and do some open-heart to figure it all out with front row seats.

"makepkg is a script to automate the building of packages. The requirements for using the script are a build-capable Unix platform and a PKGBUILD."
Again, they call makepkg a script... and they talk about "...and a PKGBUILD" which makes me assume these are PKGBUILDs to them.
If so, PKGBUILD <-- MakePKG <-- PacMan looks to be the method of the madness!

If anyone has arch/alpine installed, send in a copy of makepkg if you would! I haven't found a link to it yet and it doesn't spawn results on arch or alpine.
I take that back, I MAY have found something.  LOL - all 2,400 lines of it.

[yvs]

I don't know if PKGBUILD is better suited to making TC extensions, but I described using Pkgsrc in TC a year ago. It's a similar sort of thing for doing scripted builds from source code (which break too often for my taste when things change).

pkgsrc (from NetBSD) and ports (from FreeBSD) are good, but it's based on bunch of bsd make files, and mostly targeted those systems

Code: [Select]

% uname -sr
NetBSD 10.0
% ls -l /usr/pkgsrc/mk/**/*(.) | wc -l
     668
for pkgsrc there's also wip (work-in-progress) for non-system repositories.
And if I got it correctly some mix of concepts from pkgsrc and pkgbuild is used on Void linux with xbps

[nick65go]

@CentralWare: For PHASE1 of the project, you could gain speed using STATIC LINKED TOOLS. I mean for ash/bash, sed, grep, awk, diff, patch, make. Few reasons:

- the new scripts for TC16.x should run with independent tools, NOT depending on LIBC version of the TC root.fs. There can be new bugs in new versions, or changed arguments syntax, etc. Better to use SAME VERSION of tools for the same ARCH (ex: x86). Ex:Archlinux did not change their pack-tools for years, even with new kernel+ libc etc.

- You bypass (for the time being) the diff syntax for ASH vs. BASH; you can concentrate of other aspects of scripts.

- some tools are faster than other (ex:GAWK maybe 5x time vs. busybox-awk). You could compile (for Arch=x86) on a farm of AMD ZEN5 servers (with Arch=x86_64 base).

- the script COULD be (near) the same, invariant of libc / musl, by changing just CCFLAGS in/out of scripts.

PS: I suggest to not bite too much in one step, better version1 and test, then vers2 and test/measure. Because Pareto principle: you could gain almost 80% final result with just 20% effort. the rest is.. tuning and diminished profitability.

[nick65go]

You can chroot in any mini-root and have a live experience of how their packaging is working. Or to compare with your future TC scripts.

mini-root for Alpine Linux; go to https://www.alpinelinux.org/downloads/
and download https://dl-cdn.alpinelinux.org/alpine/v3.20/releases/x86_64/alpine-netboot-3.20.2-x86_64.tar.gz (is 3,409 KB).

mini-root for ArchLinux; from https://mirror.cmt.de/archlinux/iso/2024.08.01/ and download the archlinux-bootstrap-x86_64.tar.zst  (is 113,189 KB).

PS: It is amazing how Archlinux PACMAN download/upgrade 10+ packages (tar.zstd) in PARALLEL, if you have /etc/pacman.conf; section [options], ParalledDownloads =10 ; Sorry, I cheat a little because I have 12 threads APU and my pacman is from CachyOS (arch=x86-64-v3). But if your machine is modern, you could do it too, at least to download/compile; what metters is the final result to be correct, not how you did it.

[nick65go]

It seams that the "theory" is simple:
https://wiki.archlinux.org/title/Arch_packaging_standards#Makepkg_duties
When makepkg is used to build a package, it does the following automatically:
1. Checks if package dependencies and makedepends are installed
2. Downloads source files from servers
3. Checks the integrity of source files
4. Unpacks source files
5. Does any necessary patching
6. Builds the software and installs it in a fake root
6. Strips symbols from binaries
7. Strips debugging symbols from libraries
8. Compresses manual and/or info pages
9. Generates the package meta file which is included with each package
10. Compresses the fake root into the package file
11. Stores the package file in the configured destination directory (i.e. the current working directory by default)

FYI: https://reproducible-builds.org/
"Whilst anyone may inspect the source code [of free and open source software for malicious flaws], most software is distributed pre-compiled with no method to confirm whether they correspond...
The motivation behind the Reproducible Builds project is therefore to allow verification that no vulnerabilities or backdoors have been introduced during this compilation process".

https://reproducible-builds.org/who/projects/
ex: Alpine Linux, Arch Linux, Fedora, Debian, openSUSE;
wow, just a small bunch of Distro (max 37, out of 269 shameless clones) takes security as a concern; all others are just closed mouth lips.

[CentralWare]

"Simple" broken down

1. Checks if package dependencies and makedepends are installed - YES, we do similar
2. Downloads source files from servers - YES
3. Checks the integrity of source files - Assuming signatures, not authenticity*
4. Unpacks source files - YES
5. Does any necessary patching - It just runs existing/instructed/included patch files
6. Builds the software and installs it in a fake root - In an environment OTHER than TCL**
6. Strips symbols from binaries - Same as us
7. Strips debugging symbols from libraries - Same as us
...

*   There are some packages out there with numerous "maintainers" or authors...  it's sometimes hard to determine genuine sources of sources.
** TCL standard core doesn't necessarily play nice with chroot outside of the root/tc accounts.

The above also implies running yet another third party application (pacman>makepkg>pkgbuild) - something else that if changes are made by their authors/maintainers...  it's something we have to rebuild everything to compensate for.  This is why we're taking as much out of the equation as possible.

- some tools are faster than other (ex:GAWK maybe 5x time vs. busybox-awk). You could compile (for Arch=x86) on a farm of AMD ZEN5 servers (with Arch=x86_64 base).

Are these tools foolproof for compiling ARM and other processors?  Or are you suggesting buying a farm of AMD ZEN5 (rather specific??) just to suit x86/64?
I never said anything about cranking out an entire repository in a day.  Speed is nice; it's not vital.  Success is.
"The AMD Zen 5 release date is August 15, 2024."  Feel free to donate a few 9950X to the farm! I'll even buy the motherboards, RAM, etc.!

[nick65go]

@CentralWare: Hi, maybe I should clarify few things, even with the risk to repeat myself. I already provided pieces of this personal info, but maybe they are splited in many forum categories.

I am not a zeolot (stuborn fan) of any particular distos (linux kind or not). I like many of them (KolibryOS, WinXP-64, Win11), I even love few of them (tinycore, Alpine linux, Arch linux). But as in life/love nothing is perfect to my taste, yet I accept compromise because otherwise I should be upset almost of my time. So, I prefer a laptop machine and x86-64 APU (because my laziness and their afordable price -- for now, for me) plus I like compact size machine (no desktop+monitor in my small "house") and a silent one (no fan for CPU/GPU). For the time beeing I am not focus on ARM achitecture, sorry.
I use the computer for its software, not for its basic blocks (OS, kernel+drivers); like I used a car to move from A to B, it does not matter the color, or the engine type).

I provided just few ideas, sorry if they do not mach your preferences/goals. The long time surviving linux distros have tested and learned some eficient ways to do things for their fan/contributors. Some distros even try to please a larger (not-contributory) audience, such as Ubuntu. Each of them have good ideas, but not the same implemented allover.

As the software became more bloated, because powerful & (relativly) cheap hardware, maybe is time to re-think and colect the best ideas from each of those distros. Even today there is no consens for the type of package (NixOS, snap, flat-pack) or the compression type (ta.gz, tar.zstd). Because their target audience is different. Some distro want compatibility with older CPU (x86) when other focus on performance (CachyOS) of modern machines (ex: UEFI + x86-64-v4).

I am looking forward to test/use new tinycore software and to learn few more new things in my spare time. I accept that I can not cahange the world, but I could change "my world" as I see it from my pink coloured glasses