TC build scripts? (/building low ram LFS )

[mb0]

I'm trying to build a linux system with reduced memory requirements, ultimately for an arm system, but using x86 so far..

all i really need is:
sqlite
node.js[1]
nginx[1]
python
oss

plus usb, extX and wifi (or just ethernet at a push).

The MC setup i've been using so far works fine with ~75MB of ram, but the arm board i'm looking at (mini 2440) only has 64MB (and swap will be too slow), and i'd probably need some more headroom for a larger sqlite DB.

Would it be correct that TC's non-persistance causes it to use more ram? Thoughts on memory-saving of eg. uLibC? I.E. any doubts saving the required ram is possible? /difficult?

So, not knowing how else to proceed, i've been looking at linux-from-scratch.[2]

What i think I would find really useful however, is a good build script as a starting point, that I could tweak. Are the TC/MC build scripts available /where??

Any other related advice in this area (or re: qemu) very welcome (tutorials, build scripts, communities.. w/e).

Thanks.

.

1: could just use node for these, but AFAIK nginx only uses ~4MB or so. NB: single-digit clients.

2: i've been through to build the basic system (mostly copy+paste).. but the lfs user also shows on my host system (debian on virtualbox) and the vhd doesn't boot without sitting beside the debian one. Trying to get around BLFS, ALFS (so i can revise it and push it to multiple machines) and CLFS (!). head..

[gerald_clark]

Set up a persistent tce directory. ( See the wiki ).

[Rich]

Hi mb0

Would it be correct that TC's non-persistance causes it to use more ram?

Yes. If you plan on having a lot of stuff in your /home directory, make it persistent.

[bmarkus]

As a site note, I have a communication gateway written in Python using sqlite database. It runs smoothly on ARM and MIPS based routers (ASUS WL-500 and RouterBoard) equipped with 32Mbyte RAM with OpenWrt.

[mb0]

I already have a persistant home+opt. To go persistant with /tce, i add tce={path} bootcode.. then what? I guess I have to replace using onboot.lst with something? Any rough ideas how much memory we are talking about saving here?? This would make the whole thing persistant and negate the need for .filetool.lst etc ??

Also, what about the build scripts (+ rest) ??

@bmarkus: how big is your database?.. i have wondered if a router-linux might be a better starting-point than TC.. am interested in 'remote' updating too.. think this would be a better approach?

[bmarkus]

how big is your database?

My database is about 4 Mbyte as I'm purging adn vacuuming some data after 48 hours which are less important.

i have wondered if a router-linux might be a better starting-point than TC.. am interested in 'remote' updating too.. think this would be a better approach?

It depends on. TC is not ported to ARM or MIPS yet. You can do it yourself of course, but if your main goal is to develop an applications it is better to use already existing routers which are available for others easily.

[curaga]

OT, but node.js is both slow and a RAM hog:
http://blog.monkey-project.com/2012/03/13/duda-web-services-framework-for-monkey/
http://edsiper.linuxchile.cl/blog/2011/11/08/monkey-nodejs-when-where/

It really has no business running on something with 64mb RAM, no matter how light the base distro. Even Apache would be lighter, unless all of your server-side logic is in Javascript and requires node.

Python is also not that light, but you probably know that.


OT2: Mini 2440 looks to be quite bad for the price of 100$, 400MHz ARM9 + 64mb ram + 1gb flash, given that you can get an ARM netbook for less with better specs.

If you post your requrements, perhaps we can point towards a lighter server

[bmarkus]

I have no problem running a multitasking Python 2.6 program with three processes in 32Mbyte RAM including sqlite database. I have never tried, but lighthttpd is available in OpenWrt repo.

[curaga]

@bmarkus: Please post how much RAM they take, out of curiosity. Both according to top and "pmap -d $PID | tail -n1"

[mb0]

@curaga: absolutely not OT- appreciated, it can be pretty hard to find RAM stats aside from testing. I am using socket.io, that is why node.js, for event-based push-to-clients websocket stuff, not just vanilla web serving (thats what nginx is for). I understand the latest nginx supports websockets.. so that might be somewhat possible to use.. but socket.io has heartbeats and comprehensive fallbacks. Tornado/Twisted could be used.. but being python i was assuming it would be using even more memory?? NB that typical use will be low single-digit users- so whilst latency and initial ram use is important, supporting 20K concurrent connections- and how memory use scales with users- is not so much.
Yes, i know python isn't so light.. i'm not aware of any easy way of making it lighter (eg pypy uses way more memory) without rewriting in C(/++) or something, which is very much a last resort. (not really knowing C).

@bmarkus: have you found a point at which the sqlite database becomes so big as to be a problem performance wise? How many rows makes up your 4MB?

I would be using either OS (router eg openwrt, or MC) as a 'starting point' and hacking stuff out, etc.

As for the board, it is for development.. ARM9 are cheaper to produce (for a non-megacorp) than anything else that could do the job AFAICT, so if you know of a cheaper arm9 system (4-500mhz) with similar 'support' then i'm all ears. I'm not aware of any routers with these specs (+usb, 64MB ram) for less, either. The nokia N770 comes close, but its only a 200mhz arm9.

E: qemu is a possibility (which may be able to rule in/out arm9 as a viable platform).. although i've struggled to get to grips with it so far, being unfamiliar with both it, and cross compiling. Perhaps if i can find an arm9 image from which to start from??

[bmarkus]

@bmarkus: have you found a point at which the sqlite database becomes so big as to be a problem performance wise? How many rows makes up your 4MB?

My actual database is 1.9M, its size depends on network traffic. There are several tables, largest has 8,780 rows. It is vacuumed in every 24 hours. Mainly it is used for logging traffic data on a slow radio channel and to lookup data with simple quiries, it is not a transaction system.

I found sqlite unusable in terms of speed with abt. 100Mbyte database even for very simple operation on a dual core AMD 64-bit ATHLON 4800 CentOS system.

All depends on the usage, but there are no miracles. Maybe you can keep some tables in RAM to speed it up.

[curaga]

RPI Maybe not available though, and arm11.

DX has such netbooks, like this one:
http://www.dealextreme.com/p/7-lcd-android-2-2-umpc-netbook-w-camera-wi-fi-wm8650-800mhz-4gb-rj45-sd-102067?item=6

$88, 800Mhz arm9, 256mb RAM, 100Mbps ethernet, 3x usb. Even has webcam and built-in UPS, heh.

[bmarkus]

VIA APC:

Article says 512M RAM, 2GB flash, 4x USB

I like it. Much better than the Pi.

http://www.reghardware.com/2012/05/22/chip_maker_via_outs_49_dollar_raspberry_pi_alike/

[mb0]

I saw the via, and it is neat, but arm11.

Thanks for the heads up curaga on the deal-extreme stuff. I've seen it before, but assumed nothing would be arm9. This is my preferred device so far:
http://www.dealextreme.com/p/7-0-resistive-android-2-2-tablet-pc-w-camera-tf-via-wm8650-800mhz-2gb-106407?item=8

There are 2 questions i need to deal w/:

1) is it easy to adjust the processor frequency for this device/cpu? Or otherwise (properly) in software. The chip i'm considering is 400mhz IIRC, so it would need to be running comparably.
2) getting linux on it. Obviously this is a pretty big one... i know it supports android so should work w/ the linux kernel, but i don't have experience porting linux to new devices.. so i would hope someone would have done it before.. tbh I wouldn't even know what to google for this device. Or would it not be a major issue?

FWIW i'm looking to support up to 10k rows.. fairly straightforward queries, although it would need to be reasonably speedy. Which seems to sound possible (hopefully) from what you have said, bmarkus.

as for node.js alternatives for my use??

(OT; there are some pretty crazy awesome tablets if you can put up w/ the shitty display.. 1.2ghz A8, mali GPU, 512M ram, capacitive touch, ICS, microSD.. $108)

[solorin]

>as for node.js alternatives for my use??

http://mongrel2.org/
http://distro.ibiblio.org/tinycorelinux/4.x/x86/tcz/mongrel2.tcz.info

http://jabberwocky.eu/2011/02/02/html5-pubsub-and-browser-push/

http://luvit.io/
something closer to node.js but with luajit
not stable yet i think.

if you end up using Core & nodejs. you might want to look into updating nodejs.tcz
node development moves fast, and the last one i packaged is already out of date.

cheerio,
solorin