WelcomeWelcome | FAQFAQ | DownloadsDownloads | WikiWiki

Author Topic: TC build scripts? (/building low ram LFS )  (Read 5595 times)

Offline mb0

  • Jr. Member
  • **
  • Posts: 86
TC build scripts? (/building low ram LFS )
« on: May 22, 2012, 06:58:02 AM »
I'm trying to build a linux system with reduced memory requirements, ultimately for an arm system, but using x86 so far..

all i really need is:
sqlite
node.js[1]
nginx[1]
python
oss

plus usb, extX and wifi (or just ethernet at a push).

The MC setup i've been using so far works fine with ~75MB of ram, but the arm board i'm looking at (mini 2440) only has 64MB (and swap will be too slow), and i'd probably need some more headroom for a larger sqlite DB.

Would it be correct that TC's non-persistance causes it to use more ram? Thoughts on memory-saving of eg. uLibC? I.E. any doubts saving the required ram is possible? /difficult?

So, not knowing how else to proceed, i've been looking at linux-from-scratch.[2]

What i think I would find really useful however, is a good build script as a starting point, that I could tweak. Are the TC/MC build scripts available /where??

Any other related advice in this area (or re: qemu) very welcome (tutorials, build scripts, communities.. w/e).

Thanks.

.

1: could just use node for these, but AFAIK nginx only uses ~4MB or so. NB: single-digit clients.

2: i've been through to build the basic system (mostly copy+paste).. but the lfs user also shows on my host system (debian on virtualbox) and the vhd doesn't boot without sitting beside the debian one. Trying to get around BLFS, ALFS (so i can revise it and push it to multiple machines) and CLFS (!). head..

Offline gerald_clark

  • TinyCore Moderator
  • Hero Member
  • *****
  • Posts: 4254
Re: TC build scripts? (/building low ram LFS )
« Reply #1 on: May 22, 2012, 07:24:16 AM »
Set up a persistent tce directory. ( See the wiki ).

Online Rich

  • TinyCore Moderator
  • Hero Member
  • *****
  • Posts: 5960
Re: TC build scripts? (/building low ram LFS )
« Reply #2 on: May 22, 2012, 07:49:48 AM »
Hi mb0
Quote
Would it be correct that TC's non-persistance causes it to use more ram?
Yes. If you plan on having a lot of stuff in your /home directory, make it persistent.

Offline bmarkus

  • Administrator
  • Hero Member
  • *****
  • Posts: 6982
    • My Community Forum
Re: TC build scripts? (/building low ram LFS )
« Reply #3 on: May 22, 2012, 08:17:44 AM »
As a site note, I have a communication gateway written in Python using sqlite database. It runs smoothly on ARM and MIPS based routers (ASUS WL-500 and RouterBoard) equipped with 32Mbyte RAM with OpenWrt.
« Last Edit: May 22, 2012, 08:19:29 AM by bmarkus »
Béla
Ham Radio callsign: HA5DI

"Amateur Radio: The First Technology-Based Social Network."

Offline mb0

  • Jr. Member
  • **
  • Posts: 86
Re: TC build scripts? (/building low ram LFS )
« Reply #4 on: May 22, 2012, 08:22:23 AM »
I already have a persistant home+opt. To go persistant with /tce, i add tce={path} bootcode.. then what? I guess I have to replace using onboot.lst with something? Any rough ideas how much memory we are talking about saving here?? This would make the whole thing persistant and negate the need for .filetool.lst etc ??

Also, what about the build scripts (+ rest) ??

@bmarkus: how big is your database?.. i have wondered if a router-linux might be a better starting-point than TC.. am interested in 'remote' updating too.. think this would be a better approach?
« Last Edit: May 22, 2012, 08:27:02 AM by mb0 »

Offline bmarkus

  • Administrator
  • Hero Member
  • *****
  • Posts: 6982
    • My Community Forum
Re: TC build scripts? (/building low ram LFS )
« Reply #5 on: May 22, 2012, 08:29:16 AM »
Quote
how big is your database?

My database is about 4 Mbyte as I'm purging adn vacuuming some data after 48 hours which are less important.

Quote
i have wondered if a router-linux might be a better starting-point than TC.. am interested in 'remote' updating too.. think this would be a better approach?

It depends on. TC is not ported to ARM or MIPS yet. You can do it yourself of course, but if your main goal is to develop an applications it is better to use already existing routers which are available for others easily.
« Last Edit: May 22, 2012, 08:32:31 AM by bmarkus »
Béla
Ham Radio callsign: HA5DI

"Amateur Radio: The First Technology-Based Social Network."

Offline curaga

  • Administrator
  • Hero Member
  • *****
  • Posts: 9873
Re: TC build scripts? (/building low ram LFS )
« Reply #6 on: May 22, 2012, 08:32:57 AM »
OT, but node.js is both slow and a RAM hog:
http://blog.monkey-project.com/2012/03/13/duda-web-services-framework-for-monkey/
http://edsiper.linuxchile.cl/blog/2011/11/08/monkey-nodejs-when-where/

It really has no business running on something with 64mb RAM, no matter how light the base distro. Even Apache would be lighter, unless all of your server-side logic is in Javascript and requires node.

Python is also not that light, but you probably know that.


OT2: Mini 2440 looks to be quite bad for the price of 100$, 400MHz ARM9 + 64mb ram + 1gb flash, given that you can get an ARM netbook for less with better specs.

If you post your requrements, perhaps we can point towards a lighter server ;)
The only barriers that can stop you are the ones you create yourself.

Offline bmarkus

  • Administrator
  • Hero Member
  • *****
  • Posts: 6982
    • My Community Forum
Re: TC build scripts? (/building low ram LFS )
« Reply #7 on: May 22, 2012, 08:36:36 AM »
I have no problem running a multitasking Python 2.6 program with three processes in 32Mbyte RAM including sqlite database. I have never tried, but lighthttpd is available in OpenWrt repo.
Béla
Ham Radio callsign: HA5DI

"Amateur Radio: The First Technology-Based Social Network."

Offline curaga

  • Administrator
  • Hero Member
  • *****
  • Posts: 9873
Re: TC build scripts? (/building low ram LFS )
« Reply #8 on: May 22, 2012, 08:39:34 AM »
@bmarkus: Please post how much RAM they take, out of curiosity. Both according to top and "pmap -d $PID | tail -n1"
The only barriers that can stop you are the ones you create yourself.

Offline mb0

  • Jr. Member
  • **
  • Posts: 86
Re: TC build scripts? (/building low ram LFS )
« Reply #9 on: May 22, 2012, 08:58:49 AM »
@curaga: absolutely not OT- appreciated, it can be pretty hard to find RAM stats aside from testing. I am using socket.io, that is why node.js, for event-based push-to-clients websocket stuff, not just vanilla web serving (thats what nginx is for). I understand the latest nginx supports websockets.. so that might be somewhat possible to use.. but socket.io has heartbeats and comprehensive fallbacks. Tornado/Twisted could be used.. but being python i was assuming it would be using even more memory?? NB that typical use will be low single-digit users- so whilst latency and initial ram use is important, supporting 20K concurrent connections- and how memory use scales with users- is not so much.
Yes, i know python isn't so light.. i'm not aware of any easy way of making it lighter (eg pypy uses way more memory) without rewriting in C(/++) or something, which is very much a last resort. (not really knowing C).

@bmarkus: have you found a point at which the sqlite database becomes so big as to be a problem performance wise? How many rows makes up your 4MB?

I would be using either OS (router eg openwrt, or MC) as a 'starting point' and hacking stuff out, etc.

As for the board, it is for development.. ARM9 are cheaper to produce (for a non-megacorp) than anything else that could do the job AFAICT, so if you know of a cheaper arm9 system (4-500mhz) with similar 'support' then i'm all ears. I'm not aware of any routers with these specs (+usb, 64MB ram) for less, either. The nokia N770 comes close, but its only a 200mhz arm9.

E: qemu is a possibility (which may be able to rule in/out arm9 as a viable platform).. although i've struggled to get to grips with it so far, being unfamiliar with both it, and cross compiling. Perhaps if i can find an arm9 image from which to start from??
« Last Edit: May 22, 2012, 09:11:44 AM by mb0 »

Offline bmarkus

  • Administrator
  • Hero Member
  • *****
  • Posts: 6982
    • My Community Forum
Re: TC build scripts? (/building low ram LFS )
« Reply #10 on: May 22, 2012, 09:24:45 AM »

@bmarkus: have you found a point at which the sqlite database becomes so big as to be a problem performance wise? How many rows makes up your 4MB?


My actual database is 1.9M, its size depends on network traffic. There are several tables, largest has 8,780 rows. It is vacuumed in every 24 hours. Mainly it is used for logging traffic data on a slow radio channel and to lookup data with simple quiries, it is not a transaction system.

I found sqlite unusable in terms of speed with abt. 100Mbyte database even for very simple operation on a dual core AMD 64-bit ATHLON 4800 CentOS system.

All depends on the usage, but there are no miracles. Maybe you can keep some tables in RAM to speed it up.
« Last Edit: May 22, 2012, 09:26:52 AM by bmarkus »
Béla
Ham Radio callsign: HA5DI

"Amateur Radio: The First Technology-Based Social Network."

Offline curaga

  • Administrator
  • Hero Member
  • *****
  • Posts: 9873
Re: TC build scripts? (/building low ram LFS )
« Reply #11 on: May 22, 2012, 09:31:29 AM »
RPI ;) Maybe not available though, and arm11.

DX has such netbooks, like this one:
http://www.dealextreme.com/p/7-lcd-android-2-2-umpc-netbook-w-camera-wi-fi-wm8650-800mhz-4gb-rj45-sd-102067?item=6

$88, 800Mhz arm9, 256mb RAM, 100Mbps ethernet, 3x usb. Even has webcam and built-in UPS, heh.
The only barriers that can stop you are the ones you create yourself.

Offline bmarkus

  • Administrator
  • Hero Member
  • *****
  • Posts: 6982
    • My Community Forum
Re: TC build scripts? (/building low ram LFS )
« Reply #12 on: May 22, 2012, 09:45:12 AM »
VIA APC:

Article says 512M RAM, 2GB flash, 4x USB :)

I like it. Much better than the Pi.

http://www.reghardware.com/2012/05/22/chip_maker_via_outs_49_dollar_raspberry_pi_alike/
Béla
Ham Radio callsign: HA5DI

"Amateur Radio: The First Technology-Based Social Network."

Offline mb0

  • Jr. Member
  • **
  • Posts: 86
Re: TC build scripts? (/building low ram LFS )
« Reply #13 on: May 22, 2012, 03:03:54 PM »
I saw the via, and it is neat, but arm11.

Thanks for the heads up curaga on the deal-extreme stuff. I've seen it before, but assumed nothing would be arm9. This is my preferred device so far:
http://www.dealextreme.com/p/7-0-resistive-android-2-2-tablet-pc-w-camera-tf-via-wm8650-800mhz-2gb-106407?item=8

There are 2 questions i need to deal w/:

1) is it easy to adjust the processor frequency for this device/cpu? Or otherwise (properly) in software. The chip i'm considering is 400mhz IIRC, so it would need to be running comparably.
2) getting linux on it. Obviously this is a pretty big one... i know it supports android so should work w/ the linux kernel, but i don't have experience porting linux to new devices.. so i would hope someone would have done it before.. tbh I wouldn't even know what to google for this device. Or would it not be a major issue?

FWIW i'm looking to support up to 10k rows.. fairly straightforward queries, although it would need to be reasonably speedy. Which seems to sound possible (hopefully) from what you have said, bmarkus.

as for node.js alternatives for my use??

(OT; there are some pretty crazy awesome tablets if you can put up w/ the shitty display.. 1.2ghz A8, mali GPU, 512M ram, capacitive touch, ICS, microSD.. $108)

Offline solorin

  • Full Member
  • ***
  • Posts: 184
Re: TC build scripts? (/building low ram LFS )
« Reply #14 on: May 22, 2012, 10:36:25 PM »
>as for node.js alternatives for my use??

http://mongrel2.org/
http://distro.ibiblio.org/tinycorelinux/4.x/x86/tcz/mongrel2.tcz.info

http://jabberwocky.eu/2011/02/02/html5-pubsub-and-browser-push/

http://luvit.io/
something closer to node.js but with luajit
not stable yet i think.

if you end up using Core & nodejs. you might want to look into updating nodejs.tcz
node development moves fast, and the last one i packaged is already out of date.

cheerio,
solorin
« Last Edit: May 29, 2012, 12:20:30 AM by solorin »
. . . if you don't know, now you know. . .
        ----- R.I.P. Biggie Smalls -----