WelcomeWelcome | FAQFAQ | DownloadsDownloads | WikiWiki

Author Topic: Sources and Build Scripts x86/x64  (Read 2564 times)

Offline CentralWare

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 739
Sources and Build Scripts x86/x64
« on: August 13, 2024, 04:34:23 AM »
Good morning!

@Curaga, @Juanito, @Rich, @Paul_123 and other TCL extension maintainers:

I had to build a few TCL related extensions over the past couple weeks and I keep finding myself hunting for things that are buried X number of releases ago so I put together a few scripts to help clean out and consolidate v5.x through v15.x tcz/src content (currently x86 and x64 only, I haven't gotten to ARM yet) and have a set of lists which show a good number of extensions whom do not have build content on file.  This part of the project, gathering build notes, is PHASE 1 (of 5).  The ENTIRE project will be repeated for ARM once x86/64 starts PHASE 3.

For example, @Juanito is listed as the maintainer for "unzip" but the script I put together didn't find anything in tcz/src for it throughout v5~15.  (I'm not picking on @Juanito :) unzip just happened to be a random choice from the first run.)

These scripts I ran are far from extensive, perfect in any way and they do not de-dupe related extensions in any fashion as this was intended only as a starting point.
Two text/list files are enclosed in the attachment:
The file nosource.lst is the result of extensions which we did not find a source directory/build script/etc. in v5~v15
The file nohomes.lst is the result of /tcz/src/items that exist which did not find or exactly match up to extension filenames (such as tcz/src/xorg wouldn't see "Xorg-7.7" as a match.)

What I'm hoping to accomplish is to gather build notes for extensions that otherwise have none.  Hopefully these notes contain web links to source tarballs, any exports/flags (CC/CXX/etc), compile and runtime dependencies, anomalies, trouble shooting experiences, etc. pertaining to each given extension - anything we can gather and build upon for the years to come.  Granted, there's a good number of source packages that configure > make > install without an ounce of effort, but even so much as a link to sources (www, github, etc.) could save hours of hunting over time.  (Yes, some of these links are in filename.tcz.info - I haven't gotten that far, yet! :) )

These build notes will eventually be turned into full compilation scripts which handle version control, configuration, compilation, extension creation, dependency checking, etc. IN PHASE 2.

@Anyone with extension or compilation experience is more than welcome to help chip away at the list in any capacity they can offer!  If creating build notes/scripts, please use the naming convention of extension_filename.tcz.build so they line up with the actual TCZ files.

For PHASE 3, a group of networked hardware is going to be available online (ranging from numerous ArmV6, V7, V8, A64 devices and quite a few X86 and X64 formatted machines) where someone can download a script template, update it to manage a specific extension and submit the builder on actual, live equipment.  There's more than 3,200 extensions in TCL's 80x86/64 history and I'm guessing around 10% of those are driver/kernel related, but that still leaves a large number of packages that need attention!
* This will eventually include the kernel, toolchain, busybox and other core components.

Offline gadget42

  • Hero Member
  • *****
  • Posts: 712
Re: Sources and Build Scripts x86/x64
« Reply #1 on: August 14, 2024, 06:02:23 AM »
this will be very beneficial to Tiny Core Linux and will ultimately assist other projects who/whom silently copy/emulate/monitor/watch/etc TCL.

@CentralWare
please keep us posted on progress as well as communicating any potential ability to accept _donations_of_whatever_might_be_of_assistance.
The fluctuation theorem has long been known for a sudden switch of the Hamiltonian of a classical system Z54 . For a quantum system with a Hamiltonian changing from... https://forum.tinycorelinux.net/index.php/topic,25972.msg166580.html#msg166580

Offline andyj

  • Hero Member
  • *****
  • Posts: 1032
Re: Sources and Build Scripts x86/x64
« Reply #2 on: August 14, 2024, 10:40:25 AM »
About five years ago a few of us on the forum decided to do something similar. I developed a database for postgresql, some shell scripts to load the data, and some PHP scripts to present a restful interface. Unfortunately we couldn't agree on how it should be designed, as one of the team members thought the build system should be purely file based. This, and that I do back end web programming and I didn't have any desire to write a front end it basically died there. The scripts I have were developed for TCL 8, but they would all probably still work today. If this sounds like something you might be interested in I could spend a little time getting it going again. If this isn't the direction you have in mind you won't hurt my feelings if you say no.

Offline Paul_123

  • Administrator
  • Hero Member
  • *****
  • Posts: 1211
Re: Sources and Build Scripts x86/x64
« Reply #3 on: August 14, 2024, 01:30:36 PM »
For what its worth, mine are all individual scripts that I keep on my git server.  At one point I tried to use a few common scripts and configuration file for the specific package.   But I found for the packages I build, I could not get enough commonality in the script to make it worth my time.

The hardest part is the "vision"

BTW: The build information for some base packages are in the tool chain notes, as they get built very early in the toolchain process and/or are parts of the initrd.

Offline CentralWare

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 739
Re: Sources and Build Scripts x86/x64
« Reply #4 on: August 14, 2024, 09:20:18 PM »
If this sounds like something you might be interested in I could spend a little time getting it going again.
ALL directions are valid directions!  Take your LAMP scripts, for example...  they're more like a story to you when you read the story and you, personally, know how to get from chapter to chapter with the least amount of effort.  Someone else comes in and reads your story and they could be entirely lost.  I was able to read your scripts rather easily, and if memory serves, even commended you on your note-taking and thoroughness even when you struggled with MariaDB.  Yes, everything that CAN be offered is welcome!  Very few things could be considered valueless here...  if a build-note-file contained more of a wish list than anything but still had CC flags or a website that contained a GZ file whereas their GIT site had only a master download with no heads or tails to version management...  even the URL could save HOURS of wasted work!

The hardest part is the "vision"
BTW: The build information for some base packages are in the tool chain notes, as they get built very early in the toolchain process and/or are parts of the initrd.
I'm guessing your "vision" is what I noted above to AndyJ...  one person being able to "see" another person's thoughts, in writing...  or be able to "read" another's chicken-scratch to make heads or tails out of it.  However, I'm game!  I can't afford to bring the crew in and pay their salary based on this particular vision...  but down the road may be a different story!

@Everyone: There's no WRONG anything here!  Currently, there's no "standard" to go by, so there's no templates to follow (yet!) but that's ~ PHASE 4 when we reach out even to some of the application authors and see if they're willing to chip in...  you know, when they're testing out a new Beta or putting up a new release...  they had to compile it themselves!  "Get your notes together and make 'em available!" (or better yet, grab a template and fill it in!)  Even if they build on Debian, or Red Hat, or even Slack...  every piece of information offered could be one less step for us to figure out on our own...  and SO much less wasted PERSON time and PROCESSOR time.  At the end of the day, when this project is complete, technically it could easily be retrofitted for just about any 'nix out there.

@Paul_123: The toolchain notes are a little scattered as to the WHY portion of a few things (and missing links or /src content for the last half) but I'm guessing there's a reason for the specific order of things.  The links I can probably find most of via extension.tcz.info files.  Manually...  all by myself...  at three in the morning... instead of sleeping... (LOL!)
When you see the proposed "templates" you'll see why "all under one roof" really matters.  We want to be able to do everything necessary from the command line; from downloading the source "tarball" to stripping and packaging the finished product.  ZERO third party content...  no dependencies at all which are not required from the extension source.  (ie: No Perl unless Perl is needed to create the extension itself.  No PHP...  not even Bash...  I'm trying to put this project together using nothing more than the Common Core.)  "...but I created my build using Bash!?"  That's rarely ever a problem; converting Bash to (A)sh isn't all that big of a deal; I'm just trying to keep everything do-able via Busybox/Core supported functionality.

What you could help with in the months to come, IF you don't mind - especially with TOOLCHAIN, is to create a list of dependencies (DEP: kernel headers or DEP: glibc-x.xx one DEP per line would be ideal but if it's easier to copy/paste 20 into a single line...  I'll take it!) at each of your cd extension[/i] lines as the upcoming templates have two dependency lists, one for compilation, the second for runtime, so if extension-1.23.tar.xz needs ncurses(-dev) to compile, this is very helpful information to have up front.  YOU know why libstdc is that far down the list because you've already compiled it and found two or three things it needed before it was able to compile...  but "vision" isn't seeing it yet :)  (How many times have you compiled something just to have to rebuild numerous times as library-xyz is a prerequisite...  then app-abc turns out to be needed after that one (ESPECIALLY on apps like MariaDB where on a 2x40 CPU server can still take forever to compile just to get to 97% and hey! Another dependency!!!) DEP: above can cut so much of that hunting out of the equation. Especially when a good number of authors don't list their dependencies, co-deps, etc. on their websites, gits, and so on.

Toolchain Example: libstdc gets built well after its gcc parent.  I'm GUESSING there's an app/library or two that are built between GCC and LibStdC which makes it necessary to separate things in this fashion?  Which ones?  I don't know yet...  I haven't really dug that deep.  However, if I had a DEP: list, I'd know which BUILD TOOLS to load which we've already compiled --- OR, if a couple are missing, I'd know which ones NEED to be compiled, then I'd come back to the current one when they were done.  This way, if I was building mc for example, the system could toss out an error saying "Hey, dummy! I need curses and readline before I can do this...  sit tight and I'll be back when those are done..."

Back to the LibStdC example...  let's say there was a single dependency needed...  and it could have been built before EITHER of our intended libraries.  If its own DEP: list was already tended to, it could be placed higher up on the priority list and "out of the way" - and the build manager would be responsible for prioritizing extensions based on those DEPs.

@Paul_123: Another note after reading (and expanding) this thread, again, to be even more long-winded...  one of the toolchain goals is JUST the compilation and creation of the extensions (build tools) which MAKE the toolchain possible.  Any and all methods to create the initrd, for example, really should be separated into the core tools (which is another monster for the near future.)  For right now, the v14 toolchain note file is perfect as I can disregard everything else until after all of the tools compile perfectly as separate scripts.

NOTE: The system that is being built assumes NOTHING UP FRONT.  This project's first job is to kill off any and all running extensions to leave the system as fresh and clean as possible, so everything has to be declared.  Yes, compiletc is a cheat but since it exists, no harm...  no foul.  Someday, however, I'd like to see everything listed on its own without macro extensions.  Only the CORE remains (unless there are locked files) so we get a clean slate with EVERY build, allowing numerous builds with a guaranteed starting point and the DEP: lists are accurate for any operating system on (most) any platform with our BusyBox and inherent tools as the only (portable) dependencies.

As with TOOLCHAIN, the template's job is also to allow BUILD TOOLS (building extensions specifically to help create other ones, not necessarily creating TCZs for the repo.) Due to the fact that there are dozens of chain apps/libraries it may be necessary to "call it a day" 33% of the way through building the chain...  creating build extensions takes the place of /tools BUT similar to TCZ files, can be shut down and loaded fresh the next day with a dedicated partition to mount and a "guaranteed clean" TCL motto.  (copy2fs mentality compared to squash mounting will possibly be required... but it's too soon to tell.)

The final stage of this project (PHASE 5) involves creating a client/server back-bone (like peer2peer networking, but focused on compilations and testing as opposed to file sharing.)  We have about 40 Raspberry Pi units (thanks to a TCL member, we now have a RasPi-1 again!) and a dozen or two x86/64 systems which will be the starting point for this part of the project along with anyone else's machines out there who wish to participate by loaning us a thread or more similar to the VPS (Virtual Private Server) type of operation.  This also allows us to test extensions on SO many more physical platforms for debugging where otherwise we wouldn't be able to "see for ourselves" the outcome of a build - or a failed one, in particular.

Example: We have a TCL user who is trying to use an older AMD Athlon mid-tower and an even older x86 Celeron laptop to run an extension we haven't supported for a good number of years (boinc) and though the extension seems to work perfectly fine on my newer i686 workstation, it crashes on his.  PHASE 5 will allow the user to send me an "ID" from his test machine allowing me to single out his specific machine and remotely compile some of the extensions in question on his hardware, allowing me to gather hardware notes and build logs from his specific hardware.  PHASE 5 is intended to be completely user controlled and if everything works as the theory predicts, we won't even have to ponder firewalls or routers as it'll be client -> web server managed and most of the time, non-interactive unless we have a situation like his.

As always, thoughts and opinions are welcome! :)

Take care, Peeps!

any potential ability to accept _donations_of_whatever_might_be_of_assistance.
Okay...  how does this sound...
You place $1 USD in an envelope and mail it in. (other options would exist for non-US residents for supplies instead of dollars.)
You then find two friends who are willing to repeat the process and those two friends get two more friends each and so on!
YOU spend $1 and a few minutes of your time.  What happens afterward is...

$0.80 of each dollar will be used to furnish computer and/or IoT hardware to children ages 5 to 16 (on average) in an introduction to computers, 3D Printing, IoT Development, Robotics and a few other topics that are being considered in a project called "IND001 INTRO DESIGN AND FABRICATION" which is in trials right now (starts Monday) in our local school district between grades 1 and 10.  The school's current funding is dismal (which is expected in a trial run) and is only scheduled to run this 2024/25 school year if the public response isn't above terrific. Elementary kids have "labs" during summer months, everyone else during the school year.

$0.12 of each dollar will be used to hopefully entice a few bigger names and companies to stop by and throw a mini "seminar" for the kids throughout the year (mostly to keep their costs to a minimum, not for anyone to think they're going to make it rich! :) )

The remaining $0.08 is planned for maintenance, fuel/shipping, supplies, etc.

If this gets enough attention and we're able to repeat it for 2025/26 we'll be bringing in a camera crew, getting local media involved, etc. so that we can create a curriculum out of the entire year which we'd put out there for the world to share in.  Again, it's hoped to gain traction and if we're lucky, other schools across the planet will pick up on the program!

Offline gadget42

  • Hero Member
  • *****
  • Posts: 712
Re: Sources and Build Scripts x86/x64
« Reply #5 on: August 15, 2024, 07:11:01 AM »
reminds of Ken Starks and his many efforts.

for your convenience and respectful of your limited time, links included(although dated and not sure if Ken is still around anymore):

https://www.heliosinitiative.org/help.html
https://fossforce.com/2021/11/ken-starks-hangs-up-his-spurs-at-reglue/
https://yourswryly.blogspot.com/2020/01/broken.html
https://linuxlock.blogspot.com/
https://techaeris.com/2014/05/01/lucky-break-puts-technology-hands-disadvantaged-kids/
https://lwn.net/Articles/437057/

even more dated but from one of Ken's donors:
https://linuxlock.blogspot.com/2013/06/one-for-money.html#comment-6245728272361132352
https://thomasaknight.com/blog/15/

have never met Ken but very highly respect his efforts and tenacity
The fluctuation theorem has long been known for a sudden switch of the Hamiltonian of a classical system Z54 . For a quantum system with a Hamiltonian changing from... https://forum.tinycorelinux.net/index.php/topic,25972.msg166580.html#msg166580

Offline gadget42

  • Hero Member
  • *****
  • Posts: 712
Re: Sources and Build Scripts x86/x64
« Reply #6 on: August 15, 2024, 08:42:25 AM »
also, since we're talking about "schools" and subsequently "students" these by Nadia Asparouhova are definitely thought-provoking

https://nayafia.substack.com/p/protecting-our-attention
(found via her rss feed: https://nayafia.substack.com/feed)

referenced in above commentary:
https://arenamag.com/2024/07/31/playing-with-guns-and-phones/
The fluctuation theorem has long been known for a sudden switch of the Hamiltonian of a classical system Z54 . For a quantum system with a Hamiltonian changing from... https://forum.tinycorelinux.net/index.php/topic,25972.msg166580.html#msg166580

Offline Paul_123

  • Administrator
  • Hero Member
  • *****
  • Posts: 1211
Re: Sources and Build Scripts x86/x64
« Reply #7 on: August 15, 2024, 10:52:34 AM »
The toolchain order is based on the Linux From Scratch documentation.

Offline nick65go

  • Hero Member
  • *****
  • Posts: 818
Re: Sources and Build Scripts x86/x64
« Reply #8 on: August 15, 2024, 12:42:31 PM »
I am not a programmer, but I could recommend you to be inspired by well proven PKBUILD (package build) scripts from ArchLinux or AlpineLinux. Yes they have as back-end as a DATABASE (with optionally crypto-key for validation). This allow them to see many important FIELDS (URL, version, dependency packages, etc) something like TC info + dep files.

Plus in Archlinux web-site you can see (for any package) the list of files in the package (aka TCZ) but also the library dependency (*.so*). It is OK to have a new original vision, new template format; but is no harm to inspire from the "competitors".

In the end, neither Arch Linux. nor Alpine Linux, [or Debian, openSUSE, etc] develop their own software (exception their own front-end package MANAGER); instead they merely pack the upstream developed software AND apply their specific PATCH sometimes. Their main servers workload is about packaging with scripts.
« Last Edit: August 15, 2024, 12:44:05 PM by nick65go »

Offline CentralWare

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 739
Re: Sources and Build Scripts x86/x64
« Reply #9 on: August 16, 2024, 03:26:09 AM »
reminds of Ken Starks and his many efforts.
@Gadget42: Thanks for the reading material!!  I promise I'll look into it when time permits; though as of last night and the universe's ultimate gut-punch a parent could ever receive regarding the word cancer, it might be a minute. Virtually every medical journal out there states the words "...when caught and treated early..." yet these asinine physicians today all seem to be in a daze as if to say "...eh, I'll get around to it if and when I feel like it!"  My kid has more ambition and he's a Nintendo addict!  ("...ooowwwww! Dang, Dad, what is that stuff?!?!" the child screams as the window dressing is pulled to the side.  "It's called sunlight, my son. GET OUT AND GET SOME!"  Just obviously not too much these days!  Maybe these kids know something I don't!?)

@nick65go: PKBUILD has been added to my journal for investigation; thank you!  As just mentioned, it might be a minute as I'm about to go into Rambo + Terminator mode, ready to storm the medical community with one-liners starting with "...DO your damn job!" and ending with "I'll be back!" :)

@Paul_123: LFS documentation...  understood.  Thanks!

As for "school" stuff... my teen (above notes) starts school Monday and they have him in a class that focuses around a machine shop environment (lathe/mill/press/welders/etc.) and I laughed during orientation last night saying "...if you show promise with the concept of machining and with safety guidelines that "I" would approve, I'll give you a set of keys to the CNCs, 3D printers and laser rigs!" and his eyes just lit up.  My six year old then broke into the conversation with "What about me, Daddy?"

...

...

"Anyone hungry??"

Offline gadget42

  • Hero Member
  • *****
  • Posts: 712
Re: Sources and Build Scripts x86/x64
« Reply #10 on: August 16, 2024, 10:19:32 AM »
mentioning the "big C" reminded of:

https://jakeseliger.com/2024/08/04/starting-hospice-the-end/

via Bradley Taunt:

https://btxx.org/posts/perspective/

re: machining/machinist/CNC/etc, my niece's husband does that sort of thing for Blue Origin(seems good machinists are few and far between...go figure)
The fluctuation theorem has long been known for a sudden switch of the Hamiltonian of a classical system Z54 . For a quantum system with a Hamiltonian changing from... https://forum.tinycorelinux.net/index.php/topic,25972.msg166580.html#msg166580

Offline CentralWare

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 739
Re: Sources and Build Scripts x86/x64
« Reply #11 on: August 18, 2024, 12:16:02 AM »
It is OK to have a new original vision, new template format; but is no harm to inspire from the "competitors".

The beauty of being "me" is that I have no "competitors."
I see everyone as friends with similar interests.

LOL - not everyone carries that same mentality, but hey...
   Linus could have said "Closed Source!"
      Gates could have said "Free!"
         Jobs could have said "...apples AND oranges..."

Oh, what the world would be like today... :P

Offline CentralWare

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 739
Re: Sources and Build Scripts x86/x64
« Reply #12 on: August 18, 2024, 01:27:11 AM »
@Paul_123: There's a reason I keep an almost-military haircut...  it's too short to pull out when frustrated, and LFS led me RIGHT down that path! :)

GIT the initial build content for alfs...  check
Mount a RAID-5 based share to do all of the work under...  check
Move the build content into its new home and run "make" to configure it...  check
GET TO LINE 19 and CRASH...  check!

(come to find out, A/LFS doesn't LIKE being on a share, as it can't create links in the fashion it wants to for some reason)
One hour, nineteen minutes flushed down the loo...

Regardless, she's downloading all of the support sources as we speak, finally, and I'm curious to see what awaits!

Offline CentralWare

  • Retired Admins
  • Hero Member
  • *****
  • Posts: 739
Re: Sources and Build Scripts x86/x64
« Reply #13 on: August 18, 2024, 04:51:23 AM »
@nick65go:
Plus in Archlinux web-site you can see...

Arch looks very similar to what we're building (and very promising!) ours is just more elaborate (which it needs to be, considering everything it's supporting.)

Granted, the first app I usually pick on is Midnight Commander (I don't know why...  it's just been this way since Norton) and I was blown back when I couldn't find it on Arch/sources/packages :)
(It does seem to be in /community...  but still, she's LONG time tested...  should be in the home extensions!! :) )

Alpine, however, I haven't completely traced out their repository (haven't found sources, but found binaries) so that's a mission for another day.

@gadget42: Jake's situation is heart-wrenching... my logic screams, though "...what the :-X is he doing getting Bess prego again when he knows things are far from perfect??"  Regardless, our prayers go out to them.

Offline nick65go

  • Hero Member
  • *****
  • Posts: 818
Re: Sources and Build Scripts x86/x64
« Reply #14 on: August 18, 2024, 08:28:27 AM »
@CentralWare: For Alpine Linux, usually I go first to https://pkgs.alpinelinux.org/packages
, then filter by "package name" and optionaly "ARCH"
, so for MC you arrive at https://pkgs.alpinelinux.org/package/edge/main/x86_64/mc
; and then click on "Git repository" and arrive at source : https://git.alpinelinux.org/aports/tree/main/mc/APKBUILD
« Last Edit: August 18, 2024, 08:42:40 AM by nick65go »