Tiny Core Linux
General TC => General TC Talk => Topic started by: mb0 on May 22, 2012, 09:58:02 AM
-
I'm trying to build a linux system with reduced memory requirements, ultimately for an arm system, but using x86 so far..
all i really need is:
sqlite
node.js[1]
nginx[1]
python
oss
plus usb, extX and wifi (or just ethernet at a push).
The MC setup i've been using so far works fine with ~75MB of ram, but the arm board i'm looking at (mini 2440) only has 64MB (and swap will be too slow), and i'd probably need some more headroom for a larger sqlite DB.
Would it be correct that TC's non-persistance causes it to use more ram? Thoughts on memory-saving of eg. uLibC? I.E. any doubts saving the required ram is possible? /difficult?
So, not knowing how else to proceed, i've been looking at linux-from-scratch.[2]
What i think I would find really useful however, is a good build script as a starting point, that I could tweak. Are the TC/MC build scripts available /where??
Any other related advice in this area (or re: qemu) very welcome (tutorials, build scripts, communities.. w/e).
Thanks.
.
1: could just use node for these, but AFAIK nginx only uses ~4MB or so. NB: single-digit clients.
2: i've been through to build the basic system (mostly copy+paste).. but the lfs user also shows on my host system (debian on virtualbox) and the vhd doesn't boot without sitting beside the debian one. Trying to get around BLFS, ALFS (so i can revise it and push it to multiple machines) and CLFS (!). head..
-
Set up a persistent tce directory. ( See the wiki ).
-
Hi mb0
Would it be correct that TC's non-persistance causes it to use more ram?
Yes. If you plan on having a lot of stuff in your /home directory, make it persistent.
-
As a site note, I have a communication gateway written in Python using sqlite database. It runs smoothly on ARM and MIPS based routers (ASUS WL-500 and RouterBoard) equipped with 32Mbyte RAM with OpenWrt.
-
I already have a persistant home+opt. To go persistant with /tce, i add tce={path} bootcode.. then what? I guess I have to replace using onboot.lst with something? Any rough ideas how much memory we are talking about saving here?? This would make the whole thing persistant and negate the need for .filetool.lst etc ??
Also, what about the build scripts (+ rest) ??
@bmarkus: how big is your database?.. i have wondered if a router-linux might be a better starting-point than TC.. am interested in 'remote' updating too.. think this would be a better approach?
-
how big is your database?
My database is about 4 Mbyte as I'm purging adn vacuuming some data after 48 hours which are less important.
i have wondered if a router-linux might be a better starting-point than TC.. am interested in 'remote' updating too.. think this would be a better approach?
It depends on. TC is not ported to ARM or MIPS yet. You can do it yourself of course, but if your main goal is to develop an applications it is better to use already existing routers which are available for others easily.
-
OT, but node.js is both slow and a RAM hog:
http://blog.monkey-project.com/2012/03/13/duda-web-services-framework-for-monkey/
http://edsiper.linuxchile.cl/blog/2011/11/08/monkey-nodejs-when-where/
It really has no business running on something with 64mb RAM, no matter how light the base distro. Even Apache would be lighter, unless all of your server-side logic is in Javascript and requires node.
Python is also not that light, but you probably know that.
OT2: Mini 2440 looks to be quite bad for the price of 100$, 400MHz ARM9 + 64mb ram + 1gb flash, given that you can get an ARM netbook for less with better specs.
If you post your requrements, perhaps we can point towards a lighter server ;)
-
I have no problem running a multitasking Python 2.6 program with three processes in 32Mbyte RAM including sqlite database. I have never tried, but lighthttpd is available in OpenWrt repo.
-
@bmarkus: Please post how much RAM they take, out of curiosity. Both according to top and "pmap -d $PID | tail -n1"
-
@curaga: absolutely not OT- appreciated, it can be pretty hard to find RAM stats aside from testing. I am using socket.io, that is why node.js, for event-based push-to-clients websocket stuff, not just vanilla web serving (thats what nginx is for). I understand the latest nginx supports websockets.. so that might be somewhat possible to use.. but socket.io has heartbeats and comprehensive fallbacks. Tornado/Twisted could be used.. but being python i was assuming it would be using even more memory?? NB that typical use will be low single-digit users- so whilst latency and initial ram use is important, supporting 20K concurrent connections- and how memory use scales with users- is not so much.
Yes, i know python isn't so light.. i'm not aware of any easy way of making it lighter (eg pypy uses way more memory) without rewriting in C(/++) or something, which is very much a last resort. (not really knowing C).
@bmarkus: have you found a point at which the sqlite database becomes so big as to be a problem performance wise? How many rows makes up your 4MB?
I would be using either OS (router eg openwrt, or MC) as a 'starting point' and hacking stuff out, etc.
As for the board, it is for development.. ARM9 are cheaper to produce (for a non-megacorp) than anything else that could do the job AFAICT, so if you know of a cheaper arm9 system (4-500mhz) with similar 'support' then i'm all ears. I'm not aware of any routers with these specs (+usb, 64MB ram) for less, either. The nokia N770 comes close, but its only a 200mhz arm9.
E: qemu is a possibility (which may be able to rule in/out arm9 as a viable platform).. although i've struggled to get to grips with it so far, being unfamiliar with both it, and cross compiling. Perhaps if i can find an arm9 image from which to start from??
-
@bmarkus: have you found a point at which the sqlite database becomes so big as to be a problem performance wise? How many rows makes up your 4MB?
My actual database is 1.9M, its size depends on network traffic. There are several tables, largest has 8,780 rows. It is vacuumed in every 24 hours. Mainly it is used for logging traffic data on a slow radio channel and to lookup data with simple quiries, it is not a transaction system.
I found sqlite unusable in terms of speed with abt. 100Mbyte database even for very simple operation on a dual core AMD 64-bit ATHLON 4800 CentOS system.
All depends on the usage, but there are no miracles. Maybe you can keep some tables in RAM to speed it up.
-
RPI ;) Maybe not available though, and arm11.
DX has such netbooks, like this one:
http://www.dealextreme.com/p/7-lcd-android-2-2-umpc-netbook-w-camera-wi-fi-wm8650-800mhz-4gb-rj45-sd-102067?item=6
$88, 800Mhz arm9, 256mb RAM, 100Mbps ethernet, 3x usb. Even has webcam and built-in UPS, heh.
-
VIA APC:
Article says 512M RAM, 2GB flash, 4x USB :)
I like it. Much better than the Pi.
http://www.reghardware.com/2012/05/22/chip_maker_via_outs_49_dollar_raspberry_pi_alike/
-
I saw the via, and it is neat, but arm11.
Thanks for the heads up curaga on the deal-extreme stuff. I've seen it before, but assumed nothing would be arm9. This is my preferred device so far:
http://www.dealextreme.com/p/7-0-resistive-android-2-2-tablet-pc-w-camera-tf-via-wm8650-800mhz-2gb-106407?item=8
There are 2 questions i need to deal w/:
1) is it easy to adjust the processor frequency for this device/cpu? Or otherwise (properly) in software. The chip i'm considering is 400mhz IIRC, so it would need to be running comparably.
2) getting linux on it. Obviously this is a pretty big one... i know it supports android so should work w/ the linux kernel, but i don't have experience porting linux to new devices.. so i would hope someone would have done it before.. tbh I wouldn't even know what to google for this device. Or would it not be a major issue?
FWIW i'm looking to support up to 10k rows.. fairly straightforward queries, although it would need to be reasonably speedy. Which seems to sound possible (hopefully) from what you have said, bmarkus.
as for node.js alternatives for my use??
(OT; there are some pretty crazy awesome tablets if you can put up w/ the shitty display.. 1.2ghz A8, mali GPU, 512M ram, capacitive touch, ICS, microSD.. $108)
-
>as for node.js alternatives for my use??
http://mongrel2.org/
http://distro.ibiblio.org/tinycorelinux/4.x/x86/tcz/mongrel2.tcz.info
http://jabberwocky.eu/2011/02/02/html5-pubsub-and-browser-push/
http://luvit.io/
something closer to node.js but with luajit
not stable yet i think.
if you end up using Core & nodejs. you might want to look into updating nodejs.tcz
node development moves fast, and the last one i packaged is already out of date.
cheerio,
solorin
-
If you search for "wm8505 linux" you should find the kernel community and some ready distros. But given it's a consumer device built for android, it may requre some effort to get it running something else.
The netbooks usually are easier than the tablets, but YMMV.
On the CPU freq, if it's not possible to adjust directly, you can use cpulimit to make sure your app only uses X% of the cpu. Should be almost equivalent.
-
@bmarkus: Please post how much RAM they take, out of curiosity. Both according to top and "pmap -d $PID | tail -n1"
There are no pmap in the system. Free and top output attached (look fordixprsmain.py processes)
-
That is interesting with the APC from via. What is odd
is when they are discussing the APC, the guys never actually mention
connecting to the android market place.
Kind of odd that they don't mention it. (must just be implied)
::)
V
-
@cast-fish: usually (perhaps always?) android market can be 'hacked' on even if no out-of-box support.
I snapped up one of these with a broken touchscreen on ebay. There are linux disk images which just need to be copied onto the SD card and then it runs w/o even affecting the android install.. so even easier/better than i hoped, all up and running.
When looking at these tablets it is clear there is heavy bending of the truth re: specs -- 'HD' screens that are 800x480, 'dual core' tablets that are single core CPU+ single core GPU, and so on. So, is there a benchmark that i can run which will tell me the performance of the cpu (ie to guess @ its mhz).. as there appear to be some 350mhz versions of these tablets. ??
-
Just read /proc/cpuinfo since you have linux on it.
-
ok.. it says 797.97 'BogoMIPS' ? (and confirms that it is an arm9 which i wasn't doubting)
-
Usually on ARMs the bogomips correspond directly to Mhz, so you should have a 800MHz one.