Sources and Build Scripts x86/x64

[CentralWare]

LOL - you guys are awesome!
Okay, let me see if I can shed a little more light on the cause and effect of needing processor types...

Let's say I dedicate a slightly aged Intel i3 into this project.
"I" know it's an x64 and without spending any grey matter, and I'm reasonably certain it'll handle most if not all x86 functions as expected.
Once the machine is plugged into the associated network and TCL is installed with the AI foundation, the box becomes a part of the smog...  or cloud...  or what ever we want to call it!

Now, TCL fires up and goes to do a self examination.  It knows nothing about itself except for the kernel that booted and what that kernel sees of itself.  This is where CPU_TYPE() becomes important.
TCL logs into the master machine and is given a set of extensions it needs to compile based on what capabilities the machine has.
If it's an x64 machine, it does its hand-shake with the master machine and if it's running x64 at that moment and needs to build x86 apps, for example, it reboots and does so with an x86 kernel running.
(x86 apps would then be loaded as needed based on the build's specs; x86 and x64 having their own "optional" directories, per se'.)

If it were a 32 bit machine and we assumed incorrectly it had a 64 bit structure...  when it would go to reboot into a x64 kernel...  "Error! Error! Warning Will Robinson!"

THEN... once we're running the appropriate kernel, apps, etc. we'd put cpu associated flags into motion for compiling as shown in @Rich's post.

@yvs: I listed a couple anomalies with uname -m that came up with odd motherboards where I had hoped there were other methods we could think up which might fine-tune results, but yes, there are "tables" out there - but SO MANY processors over the years that a table would be huge.  (ie: CPU World)

@Juanito: I've learned that for a cleaner i486 I had to use older hardware (which takes forever to compile, but seems to come closer to the real thing, which is why I kept some old Atom motherboards on the rig - it's not PERFECT, but seemed to have fewer flaws.  The down-side is, my last 486-DX went out the door over a decade ago as we no longer had requests for it so "testing" apps is really no longer possible - at least without emulation.)  Rumor had it a couple years ago that Linus was considering dropping i486 support as it were...  I'm not sure what came of it as I didn't dig any deeper.

[CNK]

It seems that checking the Long Mode flag in the CPUID response ("lm" flag in /proc/cpuinfo) is the common way of telling 32/64bit x86.

[CentralWare]

@CNK:

For ARM processors, I just used the Revision codes to determine which board was being used (again, this might be swayed by piCore's kernel -- yet to be tested)
For Intel/AMD/IBM processors (ARE there any Cyrix processors still running out there today??) all I have right now are processor flags LM (16 bit), TM (32 bit) and RM (64 bit) but I'm not sure these are reliable determinations.
For PPC, I'm not sure yet how to go about hardware detection...  /proc/cpuinfo > grep Platform > grep Power* maybe?

We'll experiment accordingly, but it would be nice to get some people with older hardware to join in on the testing (later) as the hardware we're dedicating to this project will all be running headless save for a few "controllers" which will be tied into a KVM, but we'll call those 99% headless.  I don't foresee adding i586 or older boards to the rack, so we're good there.

[CNK]

"LM (16 bit)" doesn't make sense to me - you're saying that the LM flag indicates a 16bit CPU? Maybe I'm completely wrong then, but see the Wikipedia link.

[CentralWare]

@CNK: I may have typed the flags in the wrong order when I posted.
I think what I did was type in the flag acronyms without paying attention to their "order" and probably added 16/32/64 after the fact.

Long Mode I vaguely remember it being released as an AMD64 64-bit flag, so your assessment makes sense.  (IF memory serves, ia64 was Intel's - but with oddities)
TM (and I believe there was a TM2) I'm thinking was Thermal Monitor(ing) which may have been found to be a pre-32bit; I'll have to check.
Real Mode would be 16 bit dating back to the 8088

Good catch!

[CentralWare]

Looking for inspiration!   Two topics...

The first topic is a generalized file extractor based on file suffixes.

Code: [Select]

extract() {
    FNAME=$1
    case $FNAME in
        *.tar.bz2 | *.bz2 | *.tbz | *.tz2 | *.tb2 | *.tbz2 ) tar -vjxf $FNAME || exit 1;;
        *.tar.gz | *.tgz | *.taz )                           tar -vzxf $FNAME || exit 1;;
        *.tar.lz )                                           tar -vxf --lzip $FNAME || exit 1;;
        *.tar.lzma | *.tlz | *.lzma )                        tar -vxf --lzma $FNAME || exit 1;;
        *.lzop | *.tar.lzo )                                 tar -vxf --lzop $FNAME || exit 1;;
        *.tar.xz | *.txz | *.xz )                            tar -vxf $FNAME || exit 1;;
        *.tar.Z | *.tZ | *.taZ )                             tar -vxf --lzw $FNAME || exit 1;;
        *.tar.zst | *.tzst )                                 tar -vxf --zstd $FNAME || exit 1;;
        *.zip )                                              unzip $FNAME || exit 1;;
        *) echo "${RED}ERROR! ${YELLOW} Unknown Compression For ${FNAME}${NORMAL}"; exit 1 ;;
    esac
}
The goal here is to extract source tarballs which can be packaged in a number of flavors; please feel free to add to this list (along with a command to extract)

The second topic...  I'm looking for a clean method in sh/ash to replace a string ( REPLACE_ME! ) with the content from a file or variable.
Example: we have a script file named file.sh and a number of lines into this file we have a flag we want to replace in the middle of the script:

Code: [Select]

#!/bin/sh

## COMMENT 1
## COMMENT 2

REPLACE_ME!

## COMMENT 3
## COMMENT 4
sed couldn't handle it since $REPLACE could contain one of many unescaped things to break the stream
Splitting the file into two using awk -F"REPLACE_ME!" didn't pan out well, either.  '{print $1}' worked, $2 didn't for some reason.

I ended up using wc to count how many lines the file has, grep -n to find which line number REPLACE_ME! lives on and head/tail to do the job

input.sh

Code: [Select]

string="This is a long
string of text which can
be on multiple lines"

...leaves me with

Code: [Select]

#!/bin/sh

## COMMENT 1
## COMMENT 2

string="This is a long string of text which can be on multiple lines"

## COMMENT 3
## COMMENT 4
...unless I encapsulate "${REPLACE}" within double quotes, and then it preserves LF.  (Oops!)
It just doesn't feel as though it's the most efficient way - again, within sh/busybox confines.

Thoughts/suggestions welcome!

[patrikg]

Does not tar read magic. and use corresponding option to extract the tar file.
I newer use nothing more than xvf to extract files with lots of different compression types.
Maybe there difference between busybox and gnutar??

[CentralWare]

@patrikg: One would assume, yes, but more times than not -xf (busybox) doesn't know how to manage bz2, for example, unless it's trigger is specified, so to play it safe I have to throw all of the triggers/switches in the mix.  It can still DO the job...  just not without being told.  I'm guessing that's why it specifies the engines without hinting it knows any better:

Code: [Select]

        -Z      (De)compress using compress
        -z      (De)compress using gzip
        -J      (De)compress using xz
        -j      (De)compress using bzip2
        --lzma  (De)compress using lzma
        -a      (De)compress based on extension

It's okay though...  this project is surrounded by trying to keep everything under one roof where TCL's cut of busybox is the only real dependency and thus far, the project is coming along "swimmingly!"  (Which translates to...  if the project fails, the computer will likely find itself in a large body of water...   Swimmingly! )

[Rich]

Hi CentralWare

... The second topic...  I'm looking for a clean method in sh/ash to replace a string ( REPLACE_ME! ) with the content from a file or variable. ...

Is this what you are looking for:

Code: [Select]

#!/bin/sh

# Newline variable
NL="
"

needle="This is a long
string of text which can
be on multiple lines"

newstring="I ended up using wc to count how many lines the file has, grep -n to
 find which line number REPLACE_ME! lives on and head/tail to do the job"

haystack=""

# Clear out contents of file.
> ReplaceMe.txt

for N in $(seq 1 2)
do
echo "## COMMENT $N" >> ReplaceMe.txt
done

echo "$needle" >> ReplaceMe.txt

for N in $(seq 3 4)
do
echo "## COMMENT $N" >> ReplaceMe.txt
done

echo "$needle" >> ReplaceMe.txt

for N in $(seq 5 6)
do
echo "## COMMENT $N" >> ReplaceMe.txt
done

echo "$NL""This is ReplaceMe.txt"
cat ReplaceMe.txt

# Read the file into a variable.
haystack="$(cat ReplaceMe.txt)"

# Global replace of needle by newstring and save to file.
echo "${haystack//$needle/$newstring}" > ReplaceMe.txt

echo "$NL""This is modified ReplaceMe.txt"
cat ReplaceMe.txt
This is the result:

Code: [Select]

tc@E310:~/ReplaceMe$ ./ReplaceMe.sh

This is ReplaceMe.txt
## COMMENT 1
## COMMENT 2
This is a long
string of text which can
be on multiple lines
## COMMENT 3
## COMMENT 4
This is a long
string of text which can
be on multiple lines
## COMMENT 5
## COMMENT 6

This is modified ReplaceMe.txt
## COMMENT 1
## COMMENT 2
I ended up using wc to count how many lines the file has, grep -n to
 find which line number REPLACE_ME! lives on and head/tail to do the job
## COMMENT 3
## COMMENT 4
I ended up using wc to count how many lines the file has, grep -n to
 find which line number REPLACE_ME! lives on and head/tail to do the job
## COMMENT 5
## COMMENT 6
tc@E310:~/ReplaceMe$

[CentralWare]

@Rich: Sorry, guess my description is in need of a little expression.

Envision a filename.tcz.info where we're trying to replace the field content for

Code: [Select]

Title: [TITLE]
Author: [AUTHOR]
Date: [DATE]
...SED was having a field day and choking on almost every test due to non-escaped characters existing in either the TAG NAME (such as the "[" in [TITLE]) or anything not escaped in the replacement string.

For example:
sed -i "s/[TITLE]/$STRING/" filename.tcz.info
SED cries about the s command not being closed, which first happens because of the [, and then $STRING should it contain anything outside of the typical AlphaNumeric.

I created a function which uses head/tail/wc to find the first occurrence of [TITLE] and pastes head+$STRING+tail together skipping the [TITLE] line entirely.
Not pretty, but worked "well enough for the moment" if $STRING contained the entire line of content, not just the value.

In the end, I had to create a tiny encode() function which basically escapes "everything out of the ordinary" and SO FAR, (a-z we're at "D") it seems to be doing 'eh okay.

Code: [Select]

encode() {
    echo "${@}" | sed 's/[^a-zA-Z 0-9]/\\&/g'
}
so now we're escaping /[TITLE] up front and sending $STRING through encode()
sed -i "s/\[TITLE]/$(encode $STRING)/" filename.tcz.info
Reasonably functional... not necessarily perfect for the cause yet. Bare in mind, encode() is being sent anywhere from a word to a half page of content.

[Rich]

Hi CentralWare
How about something like this:

Code: [Select]

#!/bin/sh

TITLE="[TITLE]"
Title="MyExtension.tcz"
AUTHOR="[AUTHOR]"
Author="CentralWare"
DATE="[DATE]"
Date="$(date +"%D")"


# Newline variable
NL="
"

# Clear out contents of files.
> ReplaceMe.txt
> ReplaceMe.tmp


echo "Title: $TITLE
Author: [AUTHOR]
Date: [DATE]

Comment: What a great extension.
Change-log: First version.
Current: $Date" > ReplaceMe.txt


while read -r Line
do
case $Line in
*"$TITLE"*) echo "${Line/"$TITLE"/"$Title"}" >> ReplaceMe.tmp ;;
*"$AUTHOR"*) echo "${Line/"$AUTHOR"/"$Author"}" >> ReplaceMe.tmp ;;
*"$DATE"*) echo "${Line/"$DATE"/"$Date"}" >> ReplaceMe.tmp ;;

*)
echo "$Line" >> ReplaceMe.tmp
;;
esac
done < ReplaceMe.txt

echo "$NL""This is ReplaceMe.txt"
cat ReplaceMe.txt


echo "$NL$NL""This is ReplaceMe.tmp"
cat ReplaceMe.tmp
This is the result:

Code: [Select]

tc@E310:~/ReplaceMe$ ./ReplaceMe.sh

This is ReplaceMe.txt
Title:          [TITLE]
Author:         [AUTHOR]
Date:           [DATE]

Comment:        What a great extension.
Change-log:     First version.
Current:        09/05/24


This is ReplaceMe.tmp
Title:          MyExtension.tcz
Author:         CentralWare
Date:           09/05/24

Comment:        What a great extension.
Change-log:     First version.
Current:        09/05/24
tc@E310:~/ReplaceMe$

[CentralWare]

@Rich: Interesting while read > case concept.  From A to Z I'm at "T" at the moment so I'll likely have to wait until Friday evening to tinker with implementing something like it.

Issues:
We're repeating this process with 6,000+ files...
Each file has been in the hands of different humans...
Each humans likes to do things differents...
OMG what an brain-strain!

Here's an example source file: https://git.alpinelinux.org/aports/tree/main/mc/APKBUILD
We're populating a template file with a slightly similar layout to filename.tcz.info using a search/replace tag system as noted earlier using the content from pages such as these.
If you'll notice in the link above, the "source=" line is split into a number of rows AND doesn't contain a format of source="filename::weblink" like you're "supposed to do" per Alpine.

LOL - so many What If's to tend to in order to help prevent the same what-ifs later down the road

[mocore]

"LM (16 bit)" doesn't make sense to me - you're saying that the LM flag indicates a 16bit CPU? Maybe I'm completely wrong then, but see the Wikipedia link.

is any of this https://guix.gnu.org/blog/2024/building-packages-targeting-psabis//
*generally* relevant  ? wrt to hw / flags ect

... https://gitlab.com/x86-psABIs/x86-64-ABI ?
.. https://hpc.guix.info/blog/2022/01/tuning-packages-for-a-cpu-micro-architecture/

[mocore]

i happened to (just) read this

  I doubt there is a single distribution keeping separate repositories for every processor. 

which remind me of  https://hpc.guix.info/blog/2022/01/tuning-packages-for-a-cpu-micro-architecture/

also (four encouragement;-)
some similar sentiment wrt build-script is mentioned @ https://forum.tinycorelinux.net/index.php/topic,963.msg5548.html#msg5548