Sources and Build Scripts x86/x64

[CentralWare]

For the time beeing I am not focus on ARM achitecture, sorry.

YOU do not have to concern yourself with ARM, nor does probably half of the planet. This is my undertaking... my challenge (aka: my problem! ) so on the first day I chose to pull out the first blank page of virtual paper to start on this novel, I had to consider all of the different materials that would be used in the making of this work of art - and in the end, with the numerous hands that will go into this project, it will in fact end up being a true work of art!  Granted, in the end very few people will be able to take a few steps back and gaze through said pink glasses (or rose, or what ever color one chooses!)  Optimism is part of a universal language I prefer to speak.

This project is not intended solely for Tiny Core Linux, but is more so intended for all 'nix brethren under processors and drivers we can support thus far.
This project is not intended as much for end-users as it is for distro staff (who in turn "feed" their end users...) and software authors, including brand new ones.

OH...  It's 'nix based only.  Sorry, everyone, I'm too old to clean up after and build new windows!
   Every window I know of that has ever been constructed, eventually, leaks!™

Bloated: That's not me...  that's more distro level.  We're the blacksmiths...  we just make the tools.

"...sorry if they do not match your preferences..." NO...  First, there's nothing to be sorry about.  (Asking someone to chuck $20k or so for a couple dozen "just released" processors/computers will flip my sarcasm switch regardless - especially when it's my bank that's financing it!)  You did say a "farm" of machines - which to me, starts around 20 or so and works its way up.  People do tend to be avid shoppers when they're spending someone else's money!

What triggered my interest in building a project like this was, for example, @Paul_123 a number of months ago had made a comment to someone where he spoke about his already overwhelming workload and he was being tasked with even more responsibility.  (You'll notice the "Retired" below "CentralWare" -- has something to do with it, but far from being the only piece.)  Paul is one of the many, many people who spend quite a few hours, hundreds even, tending to some of the mundane things that everyone else generally takes for granted as most of the time they're oblivious it even exists.

If I showed up with a few thousand already compiled binary ZIP files (and the build scripts for each were public domain, and the binaries had a paper trail of anything involved to help in their creation) @Paul_123 and the rest of the TCL staff and maintainers could spend a short while perfecting the ARC2TCZ script to their liking that would convert these ZIP files into something Tiny Core Linux can then use for their repository and automate the entire thing, reducing those wasted hours.  Ubuntu, Slack, Debian, etc. crews can come in later and do the same thing if they wished, and so on.  I don't care WHO uses it...  I just hope they help keep it alive after its infancy as I won't live on forever and I've seen first hand here at TCL what happens when an admin/moderator/host/maintainer vanishes, quits, dies...  potential chaos that can go on for years.

"Hey, what about five to ten years from now when a new type of processor is released?"
I've paved the way attempting to put some method of standardization together...  to bring a little bit of calm to the chaos of attempting to read the minds of thousands of software authors and incorporate "change" as though it were a friend - or at least a necessary evil.  There are patches which pertain to only a given distribution...  and there are patches that fix quirks with a given software application - we'll eventually grow to where we'll be able to support both. When there's a new processor to add, though, "Here you go, peeps!  You have the building blocks!  Make it so!"  LOL - I can't be expected to do everything! :p

Kernels and Cores (initrd) are a part of my personal quest for Tiny, but they're more of a side gig and testing grounds for this project (what I've lovingly named SimpleNIX; Keep It Simple S___ sounded too creepy when you abbreviate it to KissNix ) and each distro has their own recipe for both which are so painfully different I couldn't begin to see how I could accommodate an automation system (other than toolchain apps, possibly) to please that kind of an audience.

Notice: This project will not be able to comply to every distro's way of doing things but should be able to provide binary content packages where the distro's methods can be met with some basic scripting.

@nick65go: I wouldn't curse you with my home "computer" as it was built right before the COVID lock-down here in the States with the assumption a "lock-down" was going to take place and we needed a way to work from home as initially, the US Government didn't see us as being "First Responders" so we were forcefully closed like most everyone else.  Until sh*t started breaking of theirs.  If it wasn't for this rig and three others like it, we probably wouldn't have survived the lock-down.  If it weren't for these rigs, I wouldn't have been diagnosed with having COVID five separate times and am now dealing with long term effects of being a First Responder.  Having a smaller abode with a laptop, a tablet or what ever makes it worthwhile for you is actually envied...  I'm here with a family of five where four of the five are truly clinically addicted to digital...  whereas personally...  I'd rather walk away from it.

All this...  to avoid the COVID shut-down.  I'm not really certain it was worth it.
BUT...  take all of this and think of this project's goal...  to help people stop wasting precious, irreplaceable time...
THAT is what makes it...  worth it.

Okay, I'm taking a TCL break for a bit to get some real work done on this instead of rambling on a forum.  If you guys need anything, shoot me an email.  If you want to lend a hand, shoot me an email.  If you want to buy the next pot o'coffee, shoot me an email.  Okay... too much shooting going on...  take care!

[CentralWare]

UPDATE:

16,004 extensions found between Alpine, Arch and Tiny Core

• About a dozen or so will be Tiny Core module related (filename-KERNEL.tcz) (oops!)
• A good chunk of these will be secondary extensions (ie: filename, filename-dev, filename-doc, etc.)
• Many will be Related and Unrelated extensions (alsa, alsa-mixer, etc.) and some Relateds will be from the same source package.
• Some of these extensions will be duplicates due to naming (ie: xorg versus Xorg)

From A to Z we're almost at the end of the F's and she's only been running ~10 minutes

Due to the extent of the unknowns, we're not filtering builder content at all.  (ie: When searching for "alsa-mixer" it's reasonably precise as-is, when searching for "alsa" both alsa and alsa-mixer build content will be included in the "alsa" directory. This is perfectly fine this early on as the sorting logic would be astronomical.)

Due to the many, many filename naming conventions and versioning methods, we may have to get creative with the resulting repository.
Due to the number of naming issues which caused problems with our trimming methods (filename-ver.si.on.extension) where there are odd long-file-names-1.2.3 or versions which are not period based (such as dropbear-2024-85.tcz) there will be a number of extension directory names which will have to be manually cleaned up after and likely merged with others created by TCL's info.lst

I'd estimate at least a couple/few dozen will be empty references where Tiny Core will have a tcz/src/extension directory containing source code, but no builder info which may also not have an exact match on A/A.  These will have to be created using an empty template, but compared to the total...  this is nothing!

[CentralWare]

UPDATE:

• From A to Z on the second pass we're currently at py3-x* where the repository is being filled by Alpine, Arch and Tiny content where applicable.
• There are a number of "empty" extensions which I'm guessing are probably Tiny's macro extensions (such as compile-tc which is just .dep/.info files) and AA's that are similar.
• Tiny's extensions are being scoured from version 5.x to current, both PC and ARM/AARCH platforms. dCore doesn't fit the project "as is" and PowerPC is a maybe someday at the moment as we don't stock PPCs and unless someone wanted to donate a few minis, I'd have to pass that baton as we don't have the hardware to test or build on and I don't see myself investing in ~20 year old Macs for a crowd which may not even exist any longer.
• Tiny's kernel content is scrubbed (ie: anything-tinycore and anything-piCore extensions were removed.) I'm guessing there's some AA content similar to these.
• This pass should be complete sometime tomorrow (Friday), then run through the cleaner and afterward, we'll add empty extensions back into the scanner for one last run before flagging them as Lost Boys.

I'm estimating Monday for commencement of the new builder system, but may need to take 'er to work with me to pass it onto the rack servers.

The current part of the project is being called Search, Rescue and Save and GIT will have three directories named accordingly - this is the search phase, rescue would be the sorting process and save would be the new scripts that come from it all.  Search & Rescue will eventually become empty directories resulting in just Saved and Lost Boys directories where some actual extensions may be flagged as Lost Boys temporarily if they're problematic, deprecated, etc. and to be researched afterward when time permits after the bulk is done.

@Admins and @Builders: please donate a few minutes by offering up your personal choices for DEFAULT link/compile/etc. flags for the architectures of x86, x64, arm6, arm7 and aarch64 which are tried and tested for Core.  Granted, most will likely be the same as the next person's...  but this is a good time to bring up opinions such as "...but this comes in handy in instances such as..." as we're likely to come across these conditions as we rebuild a large chunk of the 'Nix world!

[Juanito]

CC="gcc -march=i486 -mtune=i686 -Os -pipe" CXX="g++ -march=i486 -mtune=i686 -Os -pipe -fno-exceptions -fno-rtti"
CC="gcc -mtune=generic -Os -pipe" CXX="g++ -mtune=generic -Os -pipe -fno-exceptions -fno-rtti"
CC="gcc -march=armv6zk -mtune=arm1176jzf-s -mfpu=vfp -Os -pipe" CXX="g++ -march=armv6zk -mtune=arm1176jzf-s -mfpu=vfp -Os -pipe -fno-exceptions -fno-rtti
CC="gcc -march=armv8-a+crc -mtune=cortex-a72 -Os -pipe" CXX="g++ -march=armv8-a+crc -mtune=cortex-a72 -Os -pipe -fno-exceptions -fno-rtti"

-flto and -DNDEBUG can also be used, -fno-exceptions and -fno-rtti sometimes fail.

autotools
CC="one of the above" CXX="one of the above" ./configure --prefix=/usr/local --localstatedir=/var --disable-static --libexecdir=/usr/local/lib/pkgname
"find . -name Makefile -type f -exec sed -i 's/-g -O2//g' {} \;" removes "-g -O2" sprinkled through the Makefiles, occasionally it is -O3

cmake
-DCMAKE_C_FLAGS_RELEASE="one of the above" -DCMAKE_CXX_FLAGS_RELEASE="one of the above" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_LIBDIR=lib [x86_64 only otherwise lib64 is used] -DCMAKE_INSTALL_PREFIX=/usr/local

meson
CC="one of the above" CXX="one of the above" ./configure --prefix=/usr/local --localstatedir=/var --disable-static --libexecdir=/usr/local/lib/pkgname --buildtype=plain

[Paul_123]

What he said. 

There are a hand full of packages that use no automated configure script, which requires manual editing config files, makefiles, etc.

[CNK]

As well as Juanito's GCC options, I've been trying -fno-asynchronous-unwind-tables sometimes since this thread. When I haven't it's because I forgot.

[CentralWare]

@Juanito: Thank you!  (I didn't even have an armv8 of my own yet; I must be getting old and lazy )
The "templates" for each supported platform will all have DEFAULT flags which will eventually be tweaked per application, so I figured I'd ask the Great Gurus of Core for theirs to launch with!

There are a hand full of packages that use no automated configure script, which requires manual editing config files, makefiles, etc.

@Paul_123: yes, but sometimes we're stuck with "...it is what it is..." OR if/when frustration or repetition makes somebody do something about it!   Thus the whole purpose of this project!
Not EVERY package we've ever come across will fit into this project as if it were MADE for it... but if we're lucky, maybe some of those authors may join in and make their own contribution!?

Final automated numbers:
14,832 "Searched" Extensions (one or more "anything" files found from Tiny, Alpine or Arch repositories for a given filename wildcard)  (See Attachment)
  1,250 Lost Boys (Most are likely due to tcz/src directories for a given app simply not existing on the repo from the looks of it)

The attachment is an extension list of non-lost-boys with file counts in the format of  [EXTENSION NAME] [ALPN] [ARCH] [TINY]
SOME of the Tiny sources look as though they may just be source tarballs and nothing more, a few I saw whizzing by looked to be just patches...  we have our work cut out for us!

[CentralWare]

As well as Juanito's GCC options, I've been trying -fno-asynchronous-unwind-tables sometimes since this thread. When I haven't it's because I forgot.

@CNK: Thank you; notation added

[CentralWare]

@Everyone: Good evening!

I'm trying to come up with a reliable, shell only method to determine CPU architecture and looking for suggestions to add checks/double-checks considering the kernel tends to sway results based on what it was compiled for.

For ARM processors, I just used the Revision codes to determine which board was being used (again, this might be swayed by piCore's kernel -- yet to be tested)
For Intel/AMD/IBM processors (ARE there any Cyrix processors still running out there today??) all I have right now are processor flags LM (16 bit), TM (32 bit) and RM (64 bit) but I'm not sure these are reliable determinations.
For PPC, I'm not sure yet how to go about hardware detection...  /proc/cpuinfo > grep Platform > grep Power* maybe?

From what I just tested, compiler detection is swayed by the kernel (cc -dumpmachine reflects the running kernel, not the hardware's capabilities - I have an x64 running 14x86 and CC says it's a 486 ) so this isn't any better than flag testing, which doesn't require external software.)

Note: 3rd party apps such as cc, lscpu, etc. thus far all seem to be using kernel's uname or cpuinfo results to display content, thus far we haven't found anything that digs deeper than the obvious; nor have I yet envisioned a WAY to dig deeper myself.

[CNK]

I'm trying to come up with a reliable, shell only method to determine CPU architecture and looking for suggestions to add checks/double-checks considering the kernel tends to sway results based on what it was compiled for.

For ARM processors, I just used the Revision codes to determine which board was being used (again, this might be swayed by piCore's kernel -- yet to be tested)

I haven't looked at the kernel code, but the origin of the revision code for the Pis is the information reported by the VPU firmware that's returned in response to a request from the CPU through the mailbox property interface. The Pi revision codes shouldn't change between kernels unless there's a bug. Although I think the kernel generates the text description shown in RPi OS (but not on the versions of PiCore I've tried).

I don't know any specifics for universal x86 or x86_64 CPU identification. DMI would help for newer hardware, but that might need dmidecode.

[yvs]

@Everyone: Good evening!
I'm trying to come up with a reliable, shell only method to determine CPU architecture and looking for suggestions to add checks/double-checks considering the kernel tends to sway results based on what it was compiled for.

If that's for shell conditional settings depended on repository arch, why not `uname -m` for four TCL15 archs? (something like: i686 -> build for x86 repository, x86_64 -> x86_64, armv7l -> armhf, ..)
If that's for code optimization (besides compiler defaults), I'd not use any (not so big performance boost, but really hard to find out when it doesn't work).

[CentralWare]

@yvs:
uname -m running TCL 14 x86 kernel on an Intel i7 responds with i686 - which is "correct enough."
uname -m running TCL 14 x86 kernel on an AMD X2 responds with i486 - which is untrue.
This is why we're looking for ways to check and double-check results in the function so we can be reasonably sure the returned value is at least "kinda' accurate."

The need for this information (and its accuracy) is for compiling - but maybe a bit more than one would assume.

If a machine is x86 and x64 compliant, it means we can reboot the machine between the two kernels as needed to work on software packages from both platforms.

If a machine is arm8 and complies to arm7 and arm6 the same applies - we can have it natively build apps in all three environments.  RasPi5 may be the exception; we'll see just how backward compatible those are when we get ready to launch the AI.  For the EC-9100 boards, I think we're good for arm7 but arm6 fails with a panic I haven't bothered investigating yet.

* We're not after speed or performance at this point...  we're after success rates.

@CNK: We created a grid of Revision codes dating back to Pi A/B/Zero processors to the current Pi5 and PiZero2 which is very lightly tested right now, but the tests we've run so far SEEM to be stable (ie: running piCore13xarm7 on a RasPi4 shows the CPU level (arm8) versus the kernel level (arm7) the way she's supposed to - so far! )  The problem arises when it's not a RasPi card we're dealing with.  In these cases, so far, I've had to resort to the cpuinfo -> model name, for example, showing *armv7* which can be unreliable when the next card can be seen as *Armv7* or when the vendor_id notes *Cortex A7* to hint to the hardware platform.

[yvs]

uname -m running TCL 14 x86 kernel on an Intel i7 responds with i686 - which is "correct enough."
uname -m running TCL 14 x86 kernel on an AMD X2 responds with i486 - which is untrue.

Supposing that there's no so many `uname -m` options, they can be grouped for x86, for x86_64, and so on in some table. Not?

It was maybe hundred years ago when I used it and if I can recollect it correctly... devel/cpuflags from NetBSD-pkgsrc for Linux used `uname -m` + "model name" string from proc/cpuinfo to choose CPU architecture based on small hardcoded tables.

[Rich]

Hi CentralWare
This is what I use when I set up a build script.

I have this function that I source from another file:

Code: [Select]

# ---------------------------------------------------------------------------------------- #
GetProcessorType()
{
PROCESSOR_TYPE=`uname -m`
echo "$PROCESSOR_TYPE detected."

case "$PROCESSOR_TYPE" in
        i686)
        CFLAGS="$FLTO -fuse-linker-plugin -march=i486 -mtune=i686 $OPTIMIZE $SYMBOLS $DEFINES -pipe -Wall -Wextra -fno-plt"
        CXXFLAGS="$FLTO -fuse-linker-plugin -march=i486 -mtune=i686 $OPTIMIZE $SYMBOLS -pipe -Wall -Wextra -fno-exceptions -fno-rtti"
        LDFLAGS="-Wl,-T/usr/local/lib/ldscripts/elf_i386.xbn"
        ;;

        x86_64)
        CFLAGS="$FLTO -fuse-linker-plugin -mtune=generic $OPTIMIZE $SYMBOLS $DEFINES -pipe -Wall -Wextra -fno-plt"
        CXXFLAGS="$FLTO -fuse-linker-plugin -mtune=generic $OPTIMIZE $SYMBOLS -pipe -Wall -Wextra -fno-exceptions -fno-rtti"
        LDFLAGS="-Wl,-T/usr/local/lib/ldscripts/elf_x86_64.xbn"
        ;;

        armv*)
        CFLAGS="-march=armv6zk -mtune=arm1176jzf-s -mfpu=vfp $OPTIMIZE $SYMBOLS $DEFINES -pipe -Wall -Wextra"
        CXXFLAGS="-march=armv6zk -mtune=arm1176jzf-s -mfpu=vfp $OPTIMIZE $SYMBOLS -pipe -Wall -Wextra -fno-exceptions -fno-rtti"
        LDFLAGS="-Wl,-O1"
        ;;

        aarch64)
        CFLAGS="-march=armv8-a+crc -mtune=cortex-a72 $OPTIMIZE $SYMBOLS $DEFINES -pipe -Wall -Wextra"
        CXXFLAGS="-march=armv8-a+crc -mtune=cortex-a72 $OPTIMIZE $SYMBOLS -pipe -Wall -Wextra -fno-exceptions -fno-rtti"
        LDFLAGS="-Wl,-O1"
        ;;

        *)
        echo "$PROCESSOR_TYPE: Unknown processor type. Please add an entry for it in this script."
        exit
        ;;
esac
}
# ---------------------------------------------------------------------------------------- #
Then the build script contains this:

Code: [Select]

# ---------- Set compiler options.
OPTIMIZE="-Os"
SYMBOLS="-g"
FLTO="-flto"
# Used to pass -D defines such as DEFINES="-DX_DISPLAY_MISSING" for example.
DEFINES=""

GDEBUG="No"
# Uncomment the next line to compile a version that can be run under gdb.
#GDEBUG="Debug"

if [ "$GDEBUG" == "Debug" ]
then
OPTIMIZE="-O0 -ggdb"
# -flto interferes with gdb.
FLTO=""
fi
# ---------- End compiler options.



GetProcessorType

export CFLAGS CXXFLAGS LDFLAGS


 ----- Snip -----


# Strip programs and libraries
if [ "$GDEBUG" == "No" ]
then
cd $CUR/"$PROGRAM"_all
sudo find . | xargs file | grep "executable" | grep ELF | grep "not stripped" | cut -f 1 -d : | xargs strip --strip-all 2> /dev/null
sudo find . | xargs file | grep "shared object" | grep ELF | grep "not stripped" | cut -f 1 -d : | xargs strip --strip-unneeded 2> /dev/null
cd $CUR
fi

[Juanito]

The problem on tinycore at least is not particularly choosing the cpu, but more ensuring that all 32bit compile for i486 or armv6, whereas 64bit doesn’t seem to be a problem.

Forcing i486 is becoming increasingly difficult, but armv6 only seems to become a problem when neon is involved.