Chasing a memory leak ... ?

[philip]

I'm running TC 2.0rc2 on a little thin client box with just a few extensions, all in the TCE style. The machine has no hard disk (and no fan--hooray) but there is a USB stick always connected.

The 'free' command tells me how much of my 182 MB RAM is busy at any given time. Every two minutes, about 50KB of memory moves from "free" to "used". That's an average figure, with actual amounts varying widely from 16K to 112K, but the overall effect is to starve my system of free memory after it has been running for some time.

The machine is doing nothing except reporting on its free memory every 2 minutes.

Does anyone have tools or methods that could help me isolate and fix this problem? Can anyone else reproduce my observations? Here's my memory-reporting script for anybody who wants to try. I call it 'memlog2'.

Code: [Select]

#!/bin/sh
PREV=`free | grep Mem | sed -e 's/[ ]*/ /g' | cut -d' ' -f4`
while [ 1 ]
do
sleep 120
USED=`free | grep Mem | sed -e 's/[ ]*/ /g' | cut -d' ' -f4`
CHG=$(( $USED - $PREV ))
echo "`date +%T`: used ${USED}K, a change of ${CHG}K from last note"
PREV=$USED
done

[roberts]

Does watching top help?

[jpeters]

I noticed a bit of a memory leak with firefox loaded that ended as soon as I unloaded it.  I have mine installed to HD, so it was easy to test out.  Even that was less than 2 megs that stabilized over time.  Might just be some daemon running in the background

[philip]

Does watching top help?

Unfortunately, no, because the change is so gradual that the numbers top shows look steady. Googling introduced me to the command pmap that TC makes available through the package procps.tcel, but that overshoots my needs: now instead of having too little info about memory usage, I have more detail than I can cope with. I'll continue to tinker with this, but any other suggestions will be warmly welcomed. Thanks for your interest.

[mcewanw]

Here's my memory-reporting script for anybody who wants to try. I call it 'memlog2'.

I've also noticed similar effects when running most browsers but haven't investigated it further as yet. I tried your script and it works fine on tc. For comparison purposes I thought I'd also run it on a Puppy linux setup (I have two computers in front of me on the same network). However, for some reason I haven't investigated your script didn't report the relevant info when run on Puppy so I quickly modified it to the following, which worked fine with both distributions. I haven't had time to check any results as yet but hope to report back later.

Code: [Select]

#!/bin/sh
PREV=$(echo $(free | grep Mem) | cut -d' ' -f3)
while [ 1 ]
do
sleep 120
USED=$(echo $(free | grep Mem) | cut -d' ' -f3)
CHG=$(( $USED - $PREV ))
echo "`date +%T`: used ${USED}K, a change of ${CHG}K from last note"
PREV=$USED
done

[mcewanw]

Results of running memlog2 on two systems:

1. TCver2.0rc2.0 with 256 Mbyte RAM total
2. Puppy Linux ver 2.17.1 with 256 Mbyte RAM total

Note that both machines have a pretty standard set of apps (browser, abiword, gnumeric etc) but for this test I ran memlog2 immediately after booting in a single terminal.

With TC I was using PPR/TCZ for all extension loading.

I think it is clear that no memory seems to be leaking in either case.

[EDIT: I thought TC was using more RAM than Puppy overall but new figures below show it isn't; my original test was in error...]

I note that with TC, tmpfs (mounted on /dev/shm) occupies half of the total physical RAM (this is default tmpfs behaviour I believe) but none of it is being used (isn't that effectively a waste? For PPR/TCZ mode, couldn't the reserved tmpfs amount be reduced and thus save heaps of RAM?).  [EDIT: TC user Curaga has cleared up my misconception about this in a later post below]

Puppy on the other hand reserved much less RAM for tmpfs, which it clearly uses.

Anyway, here are the results from running memlog2 on both machines simultaneously as described:

Code: [Select]

1. [TC2.0rc2.0]
root@box:/mnt/hda6# ./memlog2
18:06:48: used 133572K, a change of -128K from last note
18:08:48: used 133812K, a change of 240K from last note
18:10:48: used 133812K, a change of 0K from last note
18:12:48: used 133812K, a change of 0K from last note
18:14:48: used 133812K, a change of 0K from last note
18:16:48: used 133812K, a change of 0K from last note
18:18:48: used 133812K, a change of 0K from last note

from df -h: [TC2.0rc2] tmpfs size total = 127 MBytes (used = 0 MBytes)
----------------------------------------------------------------------

2. [Puppy ver 2.17.1]
# ./memlog2
03:09:59: used 180680K, a change of -172K from last note
03:11:59: used 180888K, a change of 208K from last note
03:13:59: used 180708K, a change of -180K from last note
03:15:59: used 180948K, a change of 240K from last note
03:17:59: used 180768K, a change of -180K from last note
03:19:59: used 180888K, a change of 120K from last note
03:21:59: used 180948K, a change of 60K from last note
03:23:59: used 180768K, a change of -180K from last note

from df -h: [Puppy] tmpfs size total = 52.6 MBytes (used = 51.6 MBytes)


[curaga]

- tmpfs doesn't use ram at all if there's nothing
- the default free "used" column also counts cache and buffers, which can be freed if an app needs the ram. They are in ram to make things faster, for example recently read files.

Philip, do you also see the effect when running in text mode? If not, it should help cut down on the possible offenders.

[mcewanw]

- tmpfs doesn't use ram at all if there's nothing
- the default free "used" column also counts cache and buffers, which can be freed if an app needs the ram.

I was hoping that was the case. But if it is (and I don't doubt your word at the personal level), why is it that my system starts using swap once the "used" column indicates that all the RAM is used up (yet df still suggests that tmpfs occupies 127 MB of RAM some of which should surely be "freed"). I can't help feeling that RAM would in fact be freed up if tmpfs wasn't set to occupy so much RAM, but I admit that I could indeed be completely wrong about that! (but what else could explain a Puppy Linux system occupying less RAM than TC? - it would surely occupy just as much cache and buffers in order to speed things up..?)

Where is tmpfs set up (somewhere, a script perhaps, in tinycore.gz I presume or is it arranged by the kernel itself somehow)? - I would like to experiment to try and clear my mind about how tmpfs works in practice (I've read a lot of the theory but remain unsatisfied with my level of understanding...)

[curaga]

It's mounted because it's in /etc/fstab, and the default is to set the max size to half ram.

If you wish to test, just umount it, measuring free before and after. Here is what I saw:

bash-3.2$ free
             total       used       free     shared    buffers     cached
Mem:       2058072     902624    1155448          0      36808     401588
-/+ buffers/cache:     464228    1593844
Swap:       971924          0     971924
bash-3.2$ sudo umount /dev/shm
bash-3.2$ free
             total       used       free     shared    buffers     cached
Mem:       2058072     903244    1154828          0      36812     402144
-/+ buffers/cache:     464288    1593784
Swap:       971924          0     971924

Puppy loads things to ram differently, would be my guess. Since they use unionfs, I think they load the compressed base to ram, and mount it, without extracting things. TC extracts the base to ram every boot. Even the uncompressed TC base takes less space than compressed puppy, but if you have installed TCE type extensions that are also extracted, a TC system might take more ram. edit: using tcz extensions on mass storage, I didn't see that part of your post.

[mcewanw]

Puppy loads things to ram differently, would be my guess. Since they use unionfs, I think they load the compressed base to ram, and mount it, without extracting things. TC extracts the base to ram every boot. Even the uncompressed TC base takes less space than compressed puppy

I'm convinced!

Unfortunately, I think I have also made a mistake in my above results for Puppy. On checking out the version of memlog2 I was running I seem to have accidentally used:

Code: [Select]

USED=$(echo $(free | grep Mem) | cut -d' ' -f4)when I should have cut out column -f3 and not -f4 (the version of memlog2 I used with TC was fine though).

I'll run the test again on Puppy Linux and modify my earlier post accordingly once I have a spare moment or two [DONE].

Thanks for the info re: tempfs and /etc/fstab. I should have known that, but my mind had gone a bit numb for some reason (occupied with too many other things I think).

[philip]

My system's memory usage stabilizes nicely when I unload the extension xonclock. Too bad: it was pretty.

[jpeters]

My system's memory usage stabilizes nicely when I unload the extension xonclock. Too bad: it was pretty.

I wouldn't get too hung up on the numbers.  For example, I loaded xonclock without a browser, and it was completely stable.  It does use almost 13 megs, however, and is a daemon.  You might try without your memory leak script running, and see if that helps.   

edit: Here's a sample using links browser and xonclock.  It appears that links browser is causing the "leak".  It's interesting that when I closed the browser, I got most of it back ...so it might not be a leak.

edit: before writing off links, I got exactly the same memory usage another time with just the apps launcher and help menu loaded. 

Code: [Select]

LINKS BROSWER & XONCLOCK

00:11:30: used 405212K, a change of 0K from last note
00:13:30: used 405336K, a change of 124K from last note
00:15:30: used 405336K, a change of 0K from last note
00:17:30: used 405460K, a change of 124K from last note
00:19:30: used 405460K, a change of 0K from last note
00:21:30: used 405584K, a change of 124K from last note
....
03:31:31: used 412428K, a change of 140K from last note
03:33:31: used 412412K, a change of -16K from last note
03:35:31: used 412552K, a change of 140K from last note
03:37:31: used 412536K, a change of -16K from last note
03:39:31: used 412676K, a change of 140K from last note
03:41:31: used 412660K, a change of -16K from last note
03:43:31: used 412800K, a change of 140K from last note
REMOVE LINKS BROWSER
03:45:31: used 407180K, a change of -5620K from last note
03:47:31: used 406016K, a change of -1164K from last note
03:49:31: used 406016K, a change of 0K from last note
03:51:31: used 406016K, a change of 0K from last note
03:53:31: used 406016K, a change of 0K from last note
03:55:31: used 406016K, a change of 0K from last note
03:57:31: used 406016K, a change of 0K from last note
.....


[jpeters]

Quote from wiki:  http://en.wikipedia.org/wiki/Memory_leak

To put it another way, a memory leak arises from a particular kind of programming error, and without access to the program code, someone seeing symptoms can only guess that there might be a memory leak. It would be better to use terms such as "constantly increasing memory use" where no such inside knowledge exists.