Extensions build system?

[Key]

The Tiny Core Linux growing with amazing speed and i think it'll be very hard to maintain extensions in near future. I've create a prototype (see attach) of extensions build system, similar to BSD-ports system. For example, if you want to build tce/tcz extension for iodine package, simpy type

Code: [Select]

cd net/iodine
makeand you'll get iodine.tce/tcz with .list, .info and .md5.txt files in net/iodine/tcp directory.

Every port contains of three files: COMMENTS (comment field of info file), FILES (list of needed source files with md5 sums) and Makefile. For example, iodine's Makefile:

Code: [Select]

# TCL Extension Build file

DISTNAME= iodine
# empty or l (libs) or m (modules)
DISTTYPE=
COMMENT= IP over DNS tunneling tool.
VERSION= 0.5.1
CATEGORY= net

SRCDIR= ${DISTNAME}-${VERSION}

BUILDDEPS= libpcap.tcel
RUNDEPS=

AUTHOR= Bjorn Andersson
HOMEPAGE= http://code.kryo.se/iodine/
LICENSE= Custom

DOWNLOADSITE= ${HOMEPAGE}

EXTENSIONBY= Key

include ../../tc-base.mk

build:
cd ${BUILDDIR} && \
${SUDO} make -j3

package:
cd ${BUILDDIR} && \
${SUDO} make DESTDIR=${DSTDIR} install

strip:
What still not done:
- md5 checks
- dependencies for build and dep files
- autoload dependencies for build
- mirrors and alternative download sites
- ...

All what I want to know is "Do Tiny Core Linux need such build system?". If yes, I'll continue develop it in free time and 'll ask for some help in developing .

P.S.: Sorry for my *bad* English.

[Jason W]

Your English is very good.    I personally think there is a place for such a system of extension building. In the beginning, a ports system was in talked about for building extensions especially before we had hosting for extensions themselves.  Now that we have our own toolchain and compile extension it could be done easily.  In my opinion it would definitely help with keeping extensions up to date and easy to rebuild when needed. There has been interest mentioned in this at least once before recently:

http://forum.tinycorelinux.net/index.php?topic=963.0

As well as others who have expressed to me an interest in such a system to ease the extension building process.

The sources and patches would be kept in the source area under the extensions name.  The port would fetch from that location and build the package.  The only thing that may be desired as  a manual operation is the seperation of development files as well as pruning man pages and such.  But the build itself could be done in ports style, and the rest could be scripted or manually.  At least the build would be accounted for. 

I think using this kind of method would ease extension making and not complicate it.  With a template one could download and simply plug in a few variables like configure options, extension making would be a snap.  Of course, the downloading and all could be optional as it would be simple to place the sources in the same directory as the port file.  At least until a simple variable could be set so one only has to enter "xmms-1.2.11.tar.gz" in a field and it would get fetched.

I will be happy to help with this kind of approach.  Take a look at CRUX pkgfiles and see how simple a port can be if you think this would add too much complexity.  It can be very simple and would create a maintainable extension area in the long run. 

[tobiaus]

Every port contains of three files: COMMENTS (comment field of info file), FILES (list of needed source files with md5 sums) and Makefile.

sounds very good!

[Key]

Ok, time for next approach. I've made some changes:
- autodetection of bzip/gzip source archive
- autodetection if ./configure script  presents
- ports at first search for sources in distfiles directories and then tries to download from vendor site
- make clean now works better
- ready packages now stored in packages/tce and packages/tcz directories

You can download new version from: http://tinycorelinux.timeold.ru/tce-make.tgz

Creating makefile for ngrep has taken for me less than 2 mins

Here it is:

Code: [Select]

# TCL Extension Build file

DISTNAME= ngrep
# empty or l (libs) or m (modules)
DISTTYPE=
COMMENT= network traffic grep tool
VERSION= 1.4.5
CATEGORY= net

SRCDIR= ${DISTNAME}-1.45

BUILDDEPS= libpcap.tcel
RUNDEPS=

AUTHOR= Jordan Ritter
HOMEPAGE= http://ngrep.sourceforge.net/
LICENSE= GPL

DOWNLOADSITE= http://switch.dl.sourceforge.net/sourceforge/ngrep/

EXTENSIONBY= Key

# add here configure options and make flags
CONFOPTIONS=
MAKEFLAGS=

include ../../tc-base.mk

# add here custom post-build instructions
build:

# add here custom strip instructions
strip:
What you think about it?

Main problem now is autoloading dependencies. How can i determine that, for example, libpcap.tcel (or libpcap.tczl) is loaded?

[tobiaus]

have a look at tce-load.sh and tce-fetch, (included in the tc base) you should be able to just call those, as far as i know nothing happens if the extension is already loaded.

the ideal i know is not to download anything that's already running, i don't know if/how they do that, but they appear to know.

[Jason W]

"tce-load extension.tcel wget"  will load the extension if it is not loaded already.  I am not on TC now so I can't double check.  I am at work and haven't looked into the tarball yet, but I would like for the configuring and building to be in one field, like in Arch Linux or Crux.  Say here would be the build funciton:

cd "$DISTNAME"-"$VERSION"
patch -p1 < "$SRCDIR"/"$SOURCE2"
./configure --prefix=/usr/local --without-option
make
make install DESTDIR="$PKG"

Or like this

cd "$DISTNAME"-"$VERSION"
make prefix=/usr/local
make prefix=/usr/local install

The build process varies considerably from one package to another, and I like the simplicity of manually calling ./configure and make or whatever commands build the package.

[Key]

Just try current vesrion Patching doing automatically, if any patches in patch/ directory exists, and configure/make/make install doing automatically too if configure and/or Makefile exists. That's make creating new packages faster. You can pass options to configure/make through variables CONFOPTIONS/MAKEFLAGS. Info, List and md5 files crating automated too. But, if automatic build is a problem, I can reverse to the first version (see code in the first post of the topic), where you do configure/make/make install manually. But I think, that at least patching should be done in automatic way.

[Jason W]

The issue I see with automating the patch command is that some patches are done with patch -p0 and others patch -p1.   I do prefer simply having the untarring of source done automatic,  ./configure --prefix=/usr/local && make && make install in the build field which can be edited along with calling upon patch in that function as patch commands vary.  I will look at what you have already done in the tarball before I comment further.

I don't want folks to be in a situation where they refrain from submitting extensions because of feeling they have to conform to a build procedure they don't understand.  I do like the idea of making the extension building process easier but aside from submitters providing basic build instructions to satisfy GPL requirements the rest is going to be optional.  With the basic build instructions for a package, one could make it into a port at a later time.

But I would like to try out some various ways of scripting the extension build process for folks who would like to use that approach.  There are some examples in the extension source directory for reference.  With packages that take several attempts to build, you just tweak the script, restart the build with one command, then come back later to see the end result.

[Key]

Ok, final approach. See http://tinycorelinux.timeold.ru/tc-build.tgz

Now, you simply build package with:

Code: [Select]

cd net/ngrep
sudo ../../build.sh
After it, you'll get in ../../packages/tce and ../../packages/tcz TCE/TCZ packages.
The package definition file looks like:

Code: [Select]

# TCL Extension Build file

DISTNAME="ngrep"
DISTTYPE=""
COMMENT="network traffic grep tool"
VERSION="1.4.5"
CATEGORY="net"

SRCDIR="${DISTNAME}-1.45"

BUILDDEPS="libpcap.tcel"
RUNDEPS=""

AUTHOR="Jordan Ritter"
HOMEPAGE="http://ngrep.sourceforge.net/"
LICENSE="GPL"

DOWNLOADSITE="http://switch.dl.sourceforge.net/sourceforge/ngrep/"

EXTENSIONBY="Key"

# patch, configure, make and install here
build()
{
cd ${SRCDIR}
./configure --prefix=${PREFIX}
make -j3
make DESTDIR=${DSTDIR} install
}

If all is ok, and this is almost what you like to see , I'll also add autobuild procedure, that will be look like:

Code: [Select]

build()
{
    autobuild
}

This is only prototype. Awaiting for your opinions. 

[tobiaus]

I don't want folks to be in a situation where they refrain from submitting extensions because of feeling they have to conform to a build procedure they don't understand.

i agree. this should be an optional compile tool, for people that want to make extensions that are easier to maintain, or (what i like it for) extensions that are easier to build. one person could create a build package like this, another could make an extension. moc comes to mind...

[Jason W]

I have looked over your stuff, and it seems like a good starting point for an easy to use template to ease extension building.  Here is a routine that would fetch dependencies if they are listed in the BUILDDEPENDS line:

for I in `echo "$BUILDDEPS"`; do
su - tc -c "tce-load "$I" wget install"

done

With some tweaking this can work pretty well.  Having build.sh in and extension that is a script like /usr/local/bin/tcbuild, just enter the directory (net/nmap) and issue the command "tcbuild".  I like that all the building and such happens in the port directory.  That way the port directory could be on permamant storage for memory savings.

I think that package seperation and pruning would probably best be left to be done manually.  However, the package could be installed into a directory called pkg, then seperate it into a directory called dev, create the icon, menu, etc, then a command arguement could then create the extensions from those two directories. 

[Key]

Final version: http://tinycorelinux.timeold.ru/tc-build.tgz

ChangeLog:
+ autoloading tce dependencies
+ checking md5s sum of source files
+ autocreating dep files

I think, this now is more than working prototype, but useful script for extension creating

Any suggestions or feature requests?

Btw, may be we should create a cvs/svn/git tree for it?  Storing build scripts in it will be very convinient.

[Jason W]

As for storing the build scripts if they are sent in via the extension gmail account I can upload them to the source directory for each given extension.

[tobiaus]

may be we should create a cvs/svn/git tree for it?  Storing build scripts in it will be very convinient.

whatever the system, it shouldn't be any farther farther from .tcz in complexity than it has to be. you don't need anything outside the 10mb base to install a package, cvs/svn/git just seems like way too much trouble and formality, there's nothing "tiny" about it. (there's nothing tiny about ooo either, but you don't actually need it to manage packages in tc.)