[Solved] Possible Tiny Core 14 Memory Issue?

[xipcore]

Tiny Core 14 Memory Issue?

Issue:
After two-five days of uptime, the OOM killer kicks in and starts killing processes.  I’ve tested this on multiple boxes, and each time it does not appear that the system is ever actually out of memory (based on the OOM killer output, available memory in the buff/cache, process usage, and vmware’s memory monitoring). 

The buff/cache memory does not appear to be releasing automatically when needed.

Has anyone else experienced buff/cache or OOM issues on TCv14, or am I missing something obvious?

Background:
I recently stood up several new Tiny Core 14 VMs in VMWare running on ESXi 7.  The only three non-system processes running are the BESClient, BESRelay, and vmtoolsd. 

I have been using Tiny Core 12 for years in the same configuration without any OOM issues.  The systems all have 1.5GB of RAM. 

On a separate VM, I put a job in place that runs ‘cache-clear’ every day at midnight.  Manually clearing the cache this way has prevented the OOM killer from being invoked. 

In another test, I gave one of the systems 3GB of RAM and the same thing happened.

I have seen the BESClient, BESRelay, vmtoolsd, and even crond invoke OOM. 

These processes are not using any additional memory on TCv14 compared to TCv12.  Additionally the v12 and v14 systems are on the same ESXi host.

Here is what the free memory typically looks like (on both v12 and v14):

      Total   used   free   shared   buff/cache   available
Mem:   1.5Gi   88Mi   76mi   14Mi   1.3Gi      1.2Gi
Swap:   2.3Gi   8.0Mi   2.3Gi

This is the top process usage less than five min before the OOM hit:

Code: [Select]

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      1446  0.4  2.2 258876 34672 ?        Sl   Oct10  38:04 /opt/BESRelay/bin/BESRelay
root      1502  2.0  3.1 121448 49040 ?        Sl   Oct10 159:18 /opt/BESClient/bin/BESClient
root      1369  0.0  0.3  34904  5100 ?        Sl   Oct10   5:27 /usr/local/bin/vmtoolsd
root     16457  0.0  0.1   2928  1724 ?        R    23:15   0:00 ps aux --sort -size,-rss
root       163  0.0  0.1   2816  2376 ?        Ss   Oct10   0:00 /sbin/udevd --daemon
root      1263  0.0  0.0   2732  1196 ?        S    Oct10   0:00 /sbin/udevd –daemon
Example of vmtoolsd invoking OOM killer below.  It appears the free memory drops to 42MB which invokes the OOM killer, even though there is plenty of memory listed as ‘available’ and plenty of swap.  It’s as though the buff/cache won’t release any of the ‘buff/cache’ memory?:

Code: [Select]

Oct 15 23:18:32 HOSTNAME kern.warn kernel: vmtoolsd invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=0, oom_score_adj=0
Oct 15 23:18:32 HOSTNAME kern.warn kernel: CPU: 0 PID: 1369 Comm: vmtoolsd Not tainted 6.1.2-tinycore #612
Oct 15 23:18:32 HOSTNAME kern.warn kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Oct 15 23:18:32 HOSTNAME kern.warn kernel: Call Trace:
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc0778e5e
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc0778e7a
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc07701ec
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  ? 0xc07804c2
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  ? 0xc0760bab
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc020c5d7
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  ? 0xc020c472
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc020cafe
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc023cdea
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc024ce03
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc024da5d
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  ? 0xc02b3a0e
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc0221636
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  ? 0xc02b3a38
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc02b3a38
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  ? 0xc02b20b1
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  ? 0xc02b3a0e
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc0264eb8
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  ? 0xc026359e
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  ? 0xc02b3a0e
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc02651e0
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc02654b0
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc026640d
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  ? 0xc03d6eaa
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc02664f1
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc0258b44
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc02595dd
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc077904c
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  0xc078197d
Oct 15 23:18:32 HOSTNAME kern.warn kernel: EIP: 0xb7c846ac
Oct 15 23:18:32 HOSTNAME kern.warn kernel: Code: 05 e8 18 d4 01 00 83 c4 2c 5b 5e 5f 5d c3 55 b8 21 00 00 00 e8 85 cb 04 00 81 c2 59 f9 0a 00 89 e5 53 8b 4d 0c 8b 5d 08 cd 80 <3d> 00 f0 ff ff 76 0e 8b 92 b0 fe ff ff f7 d8 65 89 02 83 c8 ff 5b
Oct 15 23:18:32 HOSTNAME kern.warn kernel: EAX: ffffffda EBX: bf81ac7c ECX: 00000000 EDX: b7d33ff4
Oct 15 23:18:32 HOSTNAME kern.warn kernel: ESI: b791e5d0 EDI: bf81a97c EBP: bf81a918 ESP: bf81a914
Oct 15 23:18:32 HOSTNAME kern.warn kernel: DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000292
Oct 15 23:18:32 HOSTNAME kern.warn kernel: Mem-Info:
Oct 15 23:18:32 HOSTNAME kern.warn kernel: active_anon:9652 inactive_anon:5274 isolated_anon:0
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  active_file:9568 inactive_file:198069 isolated_file:0
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  unevictable:0 dirty:15 writeback:0
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  slab_reclaimable:4605 slab_unreclaimable:2994
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  mapped:8705 shmem:1387 pagetables:102
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  sec_pagetables:0 bounce:0
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  kernel_misc_reclaimable:0
Oct 15 23:18:32 HOSTNAME kern.warn kernel:  free:149620 free_pcp:687 free_cma:0
Oct 15 23:18:32 HOSTNAME kern.warn kernel: Node 0 active_anon:38608kB inactive_anon:21096kB active_file:38272kB inactive_file:792276kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:34820kB dirty:60kB writeback:0kB shmem:5548kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 20480kB writeback_tmp:0kB kernel_stack:920kB pagetables:408kB sec_pagetables:0kB all_unreclaimable? yes
Oct 15 23:18:32 HOSTNAME kern.warn kernel: DMA free:3588kB boost:0kB min:236kB low:292kB high:348kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:1804kB unevictable:0kB writepending:0kB present:15992kB managed:5496kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct 15 23:18:32 HOSTNAME kern.warn kernel: lowmem_reserve[]: 0 839 1495 1495
Oct 15 23:18:32 HOSTNAME kern.warn kernel: Normal free:42552kB boost:0kB min:42704kB low:53380kB high:64056kB reserved_highatomic:0KB active_anon:256kB inactive_anon:0kB active_file:376kB inactive_file:782284kB unevictable:0kB writepending:56kB present:884728kB managed:871556kB mlocked:0kB bounce:0kB free_pcp:2748kB local_pcp:2748kB free_cma:0kB
Oct 15 23:18:32 HOSTNAME kern.warn kernel: lowmem_reserve[]: 0 0 5247 5247
Oct 15 23:18:32 HOSTNAME kern.warn kernel: DMA: 9*4kB (ME) 10*8kB (UME) 3*16kB (E) 1*32kB (E) 1*64kB (E) 2*128kB (E) 0*256kB 0*512kB 1*1024kB (E) 1*2048kB (E) 0*4096kB = 3588kB
Oct 15 23:18:32 HOSTNAME kern.warn kernel: Normal: 1572*4kB (UME) 1261*8kB (UME) 554*16kB (UME) 143*32kB (UME) 57*64kB (UME) 45*128kB (UME) 5*256kB (UME) 2*512kB (ME) 1*1024kB (M) 0*2048kB 0*4096kB = 42552kB
Oct 15 23:18:32 HOSTNAME kern.warn kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=4096kB
Oct 15 23:18:32 HOSTNAME kern.warn kernel: 209085 total pagecache pages
Oct 15 23:18:32 HOSTNAME kern.warn kernel: 61 pages in swap cache
Oct 15 23:18:32 HOSTNAME kern.warn kernel: Free swap  = 2403676kB
Oct 15 23:18:32 HOSTNAME kern.warn kernel: Total swap = 2418012kB
Oct 15 23:18:32 HOSTNAME kern.warn kernel: 393086 pages RAM
Oct 15 23:18:32 HOSTNAME kern.warn kernel: 167906 pages HighMem/MovableOnly
Oct 15 23:18:32 HOSTNAME kern.warn kernel: 5917 pages reserved
Oct 15 23:18:32 HOSTNAME kern.info kernel: Tasks state (memory values in pages):
Oct 15 23:18:32 HOSTNAME kern.info kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Oct 15 23:18:32 HOSTNAME kern.info kernel: [    163]     0   163      704      594    12288        0         -1000 udevd
Oct 15 23:18:32 HOSTNAME kern.info kernel: [   1263]     0  1263      683      299    12288        0         -1000 udevd
Oct 15 23:18:32 HOSTNAME kern.info kernel: [   1264]     0  1264      683      299    12288        0         -1000 udevd
Oct 15 23:18:32 HOSTNAME kern.info kernel: [   1369]     0  1369     8726     1275    28672        0             0 vmtoolsd
Oct 15 23:18:32 HOSTNAME kern.info kernel: [   1446]     0  1446    64462     8653   163840        0             0 BESRelay
Oct 15 23:18:32 HOSTNAME kern.info kernel: [   1502]     0  1502    30362    12260   102400        0             0 BESClient
Oct 15 23:18:32 HOSTNAME kern.info kernel: [   1509]     0  1509      884       23    16384        0             0 syslogd
Oct 15 23:18:32 HOSTNAME kern.info kernel: [   1648]     0  1648      884       21    16384        0             0 klogd
Oct 15 23:18:32 HOSTNAME kern.info kernel: [   6680]     0  6680      884       25    16384        0             0 crond
Oct 15 23:18:32 HOSTNAME kern.info kernel: [   6813]     0  6813      884      221    16384        0             0 getty
Example of crond invoking oom killer:

Code: [Select]

Oct 16 01:55:00 HOSTNAME kern.warn kernel: crond invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=0, oom_score_adj=0
Oct 16 01:55:00 HOSTNAME kern.warn kernel: CPU: 0 PID: 16703 Comm: crond Not tainted 6.1.2-tinycore #612
Oct 16 01:55:00 HOSTNAME kern.warn kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Oct 16 01:55:00 HOSTNAME kern.warn kernel: Call Trace:
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc0778e5e
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc0778e7a
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc07701ec
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  ? 0xc07804c2
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  ? 0xc0760bab
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc020c5d7
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  ? 0xc020c48a
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc020cafe
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc023cdea
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc0143e73
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc02292d6
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc022b926
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  ? 0xc0759d01
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc02265f2
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc022721a
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc0227370
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc025fa5b
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc025fb1a
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  ? 0xc077d742
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc0260bb6
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc026159f
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc077904c
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  0xc078197d
Oct 16 01:55:00 HOSTNAME kern.warn kernel: EIP: 0xb7d125d4
Oct 16 01:55:00 HOSTNAME kern.warn kernel: Code: fc ff 8d 65 f4 5b 5e 5f 5d c3 66 90 66 90 66 90 66 90 66 90 66 90 53 8b 54 24 10 8b 4c 24 0c 8b 5c 24 08 b8 0b 00 00 00 cd 80 <5b> 3d 01 f0 ff ff 0f 83 90 d0 f7 ff c3 55 57 56 53 e8 32 55 07 00
Oct 16 01:55:00 HOSTNAME kern.warn kernel: EAX: ffffffda EBX: 09a2e1a0 ECX: bffbf9b0 EDX: 09a2f0c0
Oct 16 01:55:00 HOSTNAME kern.warn kernel: ESI: b7de3ff4 EDI: 00000000 EBP: bffbf9f8 ESP: bffbf998
Oct 16 01:55:00 HOSTNAME kern.warn kernel: DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000297
Oct 16 01:55:00 HOSTNAME kern.warn kernel: Mem-Info:
Oct 16 01:55:00 HOSTNAME kern.warn kernel: active_anon:6209 inactive_anon:1184 isolated_anon:0
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  active_file:4516 inactive_file:197988 isolated_file:0
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  unevictable:0 dirty:9 writeback:0
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  slab_reclaimable:4567 slab_unreclaimable:2977
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  mapped:4772 shmem:1387 pagetables:77
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  sec_pagetables:0 bounce:0
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  kernel_misc_reclaimable:0
Oct 16 01:55:00 HOSTNAME kern.warn kernel:  free:162929 free_pcp:184 free_cma:0
Oct 16 01:55:00 HOSTNAME kern.warn kernel: Node 0 active_anon:24836kB inactive_anon:4736kB active_file:18064kB inactive_file:791952kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:19088kB dirty:36kB writeback:0kB shmem:5548kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:872kB pagetables:308kB sec_pagetables:0kB all_unreclaimable? yes
Oct 16 01:55:00 HOSTNAME kern.warn kernel: DMA free:3588kB boost:0kB min:236kB low:292kB high:348kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:1804kB unevictable:0kB writepending:0kB present:15992kB managed:5496kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Oct 16 01:55:00 HOSTNAME kern.warn kernel: lowmem_reserve[]: 0 839 1495 1495
Oct 16 01:55:00 HOSTNAME kern.warn kernel: Normal free:42488kB boost:0kB min:42704kB low:53380kB high:64056kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:280kB inactive_file:785248kB unevictable:0kB writepending:32kB present:884728kB managed:871556kB mlocked:0kB bounce:0kB free_pcp:736kB local_pcp:736kB free_cma:0kB
Oct 16 01:55:00 HOSTNAME kern.warn kernel: lowmem_reserve[]: 0 0 5247 5247
Oct 16 01:55:00 HOSTNAME kern.warn kernel: DMA: 9*4kB (ME) 10*8kB (UME) 3*16kB (E) 1*32kB (E) 1*64kB (E) 2*128kB (E) 0*256kB 0*512kB 1*1024kB (E) 1*2048kB (E) 0*4096kB = 3588kB
Oct 16 01:55:00 HOSTNAME kern.warn kernel: Normal: 2102*4kB (UE) 1192*8kB (UE) 538*16kB (UME) 136*32kB (UE) 51*64kB (UME) 43*128kB (UE) 5*256kB (UME) 1*512kB (E) 1*1024kB (M) 0*2048kB 0*4096kB = 42488kB
Oct 16 01:55:00 HOSTNAME kern.warn kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=4096kB
Oct 16 01:55:00 HOSTNAME kern.warn kernel: 203952 total pagecache pages
Oct 16 01:55:00 HOSTNAME kern.warn kernel: 61 pages in swap cache
Oct 16 01:55:00 HOSTNAME kern.warn kernel: Free swap  = 2403676kB
Oct 16 01:55:00 HOSTNAME kern.warn kernel: Total swap = 2418012kB
Oct 16 01:55:00 HOSTNAME kern.warn kernel: 393086 pages RAM
Oct 16 01:55:00 HOSTNAME kern.warn kernel: 167906 pages HighMem/MovableOnly
Oct 16 01:55:00 HOSTNAME kern.warn kernel: 5917 pages reserved
Oct 16 01:55:00 HOSTNAME kern.info kernel: Tasks state (memory values in pages):
Oct 16 01:55:00 HOSTNAME kern.info kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Oct 16 01:55:00 HOSTNAME kern.info kernel: [    163]     0   163      704      594    12288        0         -1000 udevd
Oct 16 01:55:00 HOSTNAME kern.info kernel: [   1263]     0  1263      683      299    12288        0         -1000 udevd
Oct 16 01:55:00 HOSTNAME kern.info kernel: [   1264]     0  1264      683      299    12288        0         -1000 udevd
Oct 16 01:55:00 HOSTNAME kern.info kernel: [   1369]     0  1369     8726     1275    28672        0             0 vmtoolsd
Oct 16 01:55:00 HOSTNAME kern.info kernel: [   1446]     0  1446    64462     8632   163840        0             0 BESRelay
Oct 16 01:55:00 HOSTNAME kern.info kernel: [   1509]     0  1509      884       23    16384        0             0 syslogd
Oct 16 01:55:00 HOSTNAME kern.info kernel: [   1648]     0  1648      884       21    16384        0             0 klogd
Oct 16 01:55:00 HOSTNAME kern.info kernel: [   6680]     0  6680      884       25    16384        0             0 crond
Oct 16 01:55:00 HOSTNAME kern.info kernel: [   6813]     0  6813      884      221    16384        0             0 getty
Oct 16 01:55:00 HOSTNAME kern.info kernel: [  16703]     0 16703      884       25    16384        0             0 crond
    [EDIT]: Added code tags.  Rich

[Rich]

Hi xipcore
Please use  Code Tags  when posting commands and responses seen in a terminal. To use  Code Tags  click on the  #  icon
above the reply box and paste your text between the  Code Tags  as shown in this example:

[code][   36.176529] pcm512x 1-004d: Failed to get supply 'AVDD': -517
[   36.176536] pcm512x 1-004d: Failed to get supplies: -517
[   36.191753] pcm512x 1-004d: Failed to get supply 'AVDD': -517[/code]

It will appear like this in your post:

Code: [Select]

[   36.176529] pcm512x 1-004d: Failed to get supply 'AVDD': -517
[   36.176536] pcm512x 1-004d: Failed to get supplies: -517
[   36.191753] pcm512x 1-004d: Failed to get supply 'AVDD': -517
Code Tags  serve as visual markers between what you are trying to say and the information you are posting. They also preserve
spacing so column aligned data displays properly. Code tags also automatically add horizontal and or vertical scrollbars
to accommodate long lines and listings.

[Rich]

Hi xipcore
Welcome to the forum.

... On a separate VM, I put a job in place that runs ‘cache-clear’ every day at midnight.  Manually clearing the cache this way has prevented the OOM killer from being invoked. 

In another test, I gave one of the systems 3GB of RAM and the same thing happened. ...

When you say the same thing happened, do you mean the OOM killer was not invoked?

It's possible you just need more memory. OOM occurs when the system
feels nothing that is currently in RAM can be swapped out and no RAM
can currently be released. Some things can't be swapped, such as the
kernel, drivers, and the initrd which is the RAM based section of the
file system to name a few. I suspect any virtualization code falls into
the same category.

[xipcore]

Rich, thank you for your reply.  I apologize for the confusion.  When I doubled the memory to 3GB, the OOM killer was invoked; the same as when the system had only 1.5GB of memory.

The Tiny Core v12 and Tiny Core v14 systems are configured identically from both a software (meaning ancillary extensions) and hardware perspective.  However, v14 is the only one experiencing these issues. 

Given that the OOM output in my initial post shows normal free memory of 42.5MB, I expected to see this reflected in the process memory usage, but I don't (based on the output in my initial post).  On a 1.5GB system, I see the memory buff/cache continue to grow until it hits about 1.3GB while the available memory stays at 1.2GB.  This is the same on Tiny Core v12 and v14.  However, v12 has never experienced this issue.  I had thought maybe something changed in v14 that was causing this to behave differently.

I am continuing to dig into this and appreciate any further insight you may have. 

[Rich]

Hi xipcore

... When I doubled the memory to 3GB, the OOM killer was invoked; the same as when the system had only 1.5GB of memory. ...

Thanks for clearing that up.

You could download the kernel config files to see what changed:
http://tinycorelinux.net/12.x/x86/release/src/kernel/config-5.10.3-tinycore
http://tinycorelinux.net/14.x/x86/release/src/kernel/config-6.1.2-tinycore

Code: [Select]

diff -u config-5.10.3-tinycore config-6.1.2-tinycore > config.txt
Someone recently posted this link:
https://www.phoronix.com/news/MGLRU-LPC-2022
Turns out it was added and enabled in TC14:

Code: [Select]

+CONFIG_LRU_GEN=y
+CONFIG_LRU_GEN_ENABLED=y
+# CONFIG_LRU_GEN_STATS is not setI'm not saying that's the problem, just highlighting it as a possibility.

The diff shows about 1500 additions and 900 removals.
Searching config.txt for some of the following might help
to narrow it down:
KVM, VIRTUAL, CACHE, RAM, BUFFER, MMU

[Rich]

Hi xipcore
Does this stop the OOM killer:

Code: [Select]

echo 0 | sudo tee /sys/kernel/mm/lru_gen/enabled
Found here:
https://github.com/raspberrypi/linux/issues/5395

[curaga]

It's entirely possible it's a kernel bug, given that sounds like the only variable. The multi-gen LRU Rich mentioned is worth a try, but other than that there may not be much you can do. Tracking it down would mean git bisecting the kernel, or getting lucky with google if somebody else has already found it.

edit: if it is the RPI issue linked, the fix was "Multi-gen LRU: fix per-zone reclaim" which was backported to the 6.1.54 kernel about a month ago. So you could also build the latest 6.1 kernel, though if it is that bug, disabling the multi-gen LRU would also solve it.

[xipcore]

Thank you both for the helpful information.  I just disabled multi-gen LRU and will wait to see what happens.

[Rich]

Hi xipcore

... After two-five days of uptime, the OOM killer kicks in and starts killing processes. ...

It's too bad the symptoms are so intermittent. Please let us
know what happens.

[gadget42]

several years old but still interesting.

https://4sysops.com/archives/esxi-7-0-free-new-features-limitations-upgrade/

[xipcore]

UPDATE:

Disabling multi-gen LRU has fixed the memory issue.

I have two test boxes with multi-gen LRU disabled that have been up for around five days now.  Over the past five days, both systems have been automatically releasing some of their buffer/cache memory.  This has kept a healthy amount of free memory available which has prevented the OOM-killer from being invoked.

Thank you to everyone in this thread for your input.  This was a massive help as I have several hundred of these systems that I need to update to version 14/6.1.2. 

[Rich]

Hi xipcore
Thank you for confirming that it worked. I've taken the liberty
of marking your thread as solved.