[Solved] not seeing all my Memory on 14.x x86

[polikuo]

Does that affect any TCE submission?

Short answer.
No, it doesn't.

Long answer.
Does it matter ?
Does it run on older machine ?
Well, hard to tell.
I once compiled a program (that I can't remember) on a CPU that support SSE instructions.
Even though I told GCC to build for i486 like every package in TC does (-march=i486),
the product just wouldn't run on Celeron Mendocino which only support MMX.
With the help of my friend, we were able to nail down the cause.
GCC simply ignore my request and use SSE instructions for the binary.
I rebuilt the binary with exactly the same setting, exactly the same versions of extensions provided by TC, on that said Mendocino computer.
Everything was good, it ran without any trouble.
My friend checked the assembly and said there's no SSE instructions anymore.
In brief, if GCC decide to compile against a newer instruction set even though you told it not to, the only way to get a more backward compatible binary is, well, build it on an older machine.
This type of trouble is bond to happen, and I think there's really not much you can do.
It's really likely that part of extensions on TC already has this problem, but no one complaining since not many sane people would stick with machine that old to run complex programs.

[aus9]

Hi Rich

I am wrong again
Your reply 7 may explain a new thing while I was attempting to compile something. I did finally compile it but it was loading pulseaudio-dev that pulls in heaps and then I got this

Code: [Select]

mount: mounting /dev/loop291 on /tmp/tcloop/libcap failed: Cannot allocate memory
root@box:/home/tc# free -m
              total        used        free      shared  buff/cache   available
Mem:           2466         287        1972          38         207        1899
Swap:          8000           0        8000
So I thought I had memory and dismissed your reply 7.

DOH

I have memory but clearly for above error.....it is not seen on my TC32

Thats a pita.

Hi Rich  can you mark it as not solved as others may be affected. I now know another member with TC32
AMD APU but he may not be compilting bloat that  I do. 

[Rich]

Hi aus9
Your  free  command shows plenty of free RAM and that
your swap space hasn't been touched.

Could you recreate the error and then post the result of:

Code: [Select]

dmesg | tail -n 25

[aus9]

Hi Rich

EDIT I think I have solved it next post

thanks for looking into this for me. I have duplicated error but slightly different package but here goes

Code: [Select]

mount: mounting /dev/loop292 on /tmp/tcloop/pci-utils failed: Cannot allocate memory


tc@box:~$ dmesg | tail -n 25 # changed to larger number
loop291: detected capacity change from 0 to 272
loop292: detected capacity change from 0 to 544
vmap allocation for size 49152 failed: use vmalloc=<size> to increase size
mount: vmalloc error: size 42284, vm_struct allocation failed, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
CPU: 2 PID: 14481 Comm: mount Not tainted 6.1.2-tinycore #612
Hardware name: Micro-Star International Co., Ltd MS-7B86/B450 GAMING PLUS MAX (MS-7B86), BIOS H.C0 05/17/2021
Call Trace:
 0xc0778e5e
 0xc0778e7a
 0xc023c774
 0xc0238527
 0xc02383e7
 ? 0xf78ca3fc
 0xc023841b
 ? 0xf78ca3fc
 0xf78ca3fc
 0xf78c9a05
 0xf78c951f
 0xf78c8fa2
 0xc025d5c8
 ? 0xf78c8c92
 0xf78c8b42
 0xc025c6c8
 0xc0274e5f
 0xc0274f46
 0xc02751ba
 0xc077904c
 0xc078197d
EIP: 0xb7e6045e
Code: 90 66 90 66 90 66 90 66 90 66 90 90 57 56 53 8b 7c 24 20 8b 74 24 1c 8b 54 24 18 8b 4c 24 14 8b 5c 24 10 b8 15 00 00 00 cd 80 <5b> 5e 5f 3d 01 f0 ff ff 0f 83 04 92 f4 ff c3 66 90 90 57 56 53 8b
EAX: ffffffda EBX: 0893d1c0 ECX: bfb91e0d EDX: bfb91e26
ESI: 00008001 EDI: 00000000 EBP: bfb91e0d ESP: bfb90880
DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000246
Mem-Info:
active_anon:78731 inactive_anon:19719 isolated_anon:0
 active_file:42748 inactive_file:40843 isolated_file:0
 unevictable:0 dirty:1759 writeback:0
 slab_reclaimable:7700 slab_unreclaimable:17476
 mapped:39999 shmem:15056 pagetables:638
 sec_pagetables:0 bounce:0
 kernel_misc_reclaimable:0
 free:382351 free_pcp:4102 free_cma:0
Node 0 active_anon:314924kB inactive_anon:78876kB active_file:170992kB inactive_file:163372kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:159996kB dirty:7036kB writeback:0kB shmem:60224kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 65536kB writeback_tmp:0kB kernel_stack:2440kB pagetables:2552kB sec_pagetables:0kB all_unreclaimable? no
DMA free:5424kB boost:0kB min:236kB low:292kB high:348kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:5500kB mlocked:0kB bounce:0kB free_pcp:76kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 828 2449 2449
Normal free:702292kB boost:0kB min:42128kB low:52660kB high:63192kB reserved_highatomic:0KB active_anon:0kB inactive_anon:176kB active_file:0kB inactive_file:10036kB unevictable:0kB writepending:28kB present:882004kB managed:860008kB mlocked:0kB bounce:0kB free_pcp:16332kB local_pcp:4524kB free_cma:0kB
lowmem_reserve[]: 0 0 12969 12969
DMA: 2*4kB (M) 3*8kB (M) 1*16kB (M) 2*32kB (M) 1*64kB (M) 1*128kB (M) 2*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 1*4096kB (M) = 5424kB
Normal: 123*4kB (UM) 229*8kB (U) 34*16kB (UE) 9*32kB (UM) 21*64kB (UE) 11*128kB (UME) 6*256kB (UE) 7*512kB (UME) 5*1024kB (UM) 5*2048kB (UM) 165*4096kB (M) = 702228kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=4096kB
98647 total pagecache pages
0 pages in swap cache
Free swap  = 8191996kB
Total swap = 8191996kB
639510 pages RAM
415013 pages HighMem/MovableOnly
8120 pages reserved
SQUASHFS error: Failed to allocate zlib workspace
squashfs image failed sanity checkk
at now I have

Code: [Select]

free -m
              total        used        free      shared  buff/cache   available
Mem:           2466         640        1389          69         437        1511
Swap:          8000           0        8000

[aus9]

I found a grub entry to change vmalloc=512M so will do a full reboot to see if error goes....ref
http://thinking-electron.blogspot.com/2015/05/how-to-increase-vmalloc-size-vmalloc.html

well that seems to be the trick!

but I noticed that when I loaded pulseaudio-dev and other stuff...my used memory was lower than what I just posted.
so I loaded an even bigger amount ffmpeg4-dev  and now

Code: [Select]

free -m
              total        used        free      shared  buff/cache   available
Mem:           2467         630        1176          71         660        1546
Swap:          8000           0        8000


dmesg | tail -n 25
loop333: detected capacity change from 0 to 16
loop334: detected capacity change from 0 to 360
loop335: detected capacity change from 0 to 56
loop336: detected capacity change from 0 to 584
loop337: detected capacity change from 0 to 168
loop338: detected capacity change from 0 to 72
loop339: detected capacity change from 0 to 784
loop340: detected capacity change from 0 to 936
loop341: detected capacity change from 0 to 1592
loop342: detected capacity change from 0 to 88
loop343: detected capacity change from 0 to 760
loop344: detected capacity change from 0 to 40
loop345: detected capacity change from 0 to 168
loop346: detected capacity change from 0 to 392
loop347: detected capacity change from 0 to 416
loop348: detected capacity change from 0 to 408
loop349: detected capacity change from 0 to 80
loop350: detected capacity change from 0 to 104
loop351: detected capacity change from 0 to 96
loop352: detected capacity change from 0 to 9944
loop353: detected capacity change from 0 to 2208
loop354: detected capacity change from 0 to 2160
loop355: detected capacity change from 0 to 104
loop356: detected capacity change from 0 to 424
loop357: detected capacity change from 0 to 776
I notice that my loop count crashed about loop 293 ignoring sizes and have got past that and no crash

I think we have the solution

add bootcode vmalloc=512M

[Rich]

Hi aus9
Seems this has come up before:
https://forum.tinycorelinux.net/index.php/topic,19415.msg120213.html#msg120213

He fixed it in reply #46:

... Increased vmalloc to 256MB ...

[jazzbiker]

Hi Rich,

In the thread You've mentioned vmalloc was hiding not solving. The solution was using Core64 mode on the box with a lot of RAM and lot of GPU RAM. Did I understand right?

@aus9: Are You observing this error in Core64 mode?

[Rich]

Hi jazzbiker
From  https://mjmwired.net/kernel/Documentation/kernel-parameters.txt :

Code: [Select]

vmalloc=nn[KMG] [KNL,BOOT] Forces the vmalloc area to have an exact
size of <nn>. This can be used to increase the
minimum size (128MB on x86). It can also be used to
decrease the size and leave more room for directly
mapped kernel RAM.The way I read it you are setting aside a larger (in this case) contiguous
address space aside for handling memory allocations.

... The solution was using Core64 mode on the box with a lot of RAM and lot of GPU RAM. Did I understand right? ...

Yes, either Core64 or Corepure64. The issue appears to be available address space, not
available RAM. If you look at my reply #7 you'll see where I showed most of the RAM
was mapped out of reach above the 4 Gbyte address range.

[aus9]

Hi jazzbiker

The solution was using Core64 mode on the box with a lot of RAM and lot of GPU RAM. Did I understand right?
@aus9: Are You observing this error in Core64 mode?

hmm  thats no help for dumbos like me 

How do I know what is a lot of GPU RAM. If you notice I reduced the motherboard RAM reserved for my APU to give my system more RAM and allegedly more free RAM

I have never seen this in TC64 but I am trying to stay kind of pure Tinycore x86 while compiling.
And personal taste....I do not want to run a 64 bit kernel with 32 bit apps in case my brain gets confused.

If there is a smarter way let me know.

Incidently slightly off topic.....I saw a recent post at
https://www.phoronix.com/

so I am thinking maybe I can build something related to "containers" like snap appimage flatpak etc
not good for compiling against but quite good for those wanting a quick up-to-date software
-negative is built against /usr and no stripping and a fair amount of duplication.

Hi Rich
Altho I mentioned your post 7 and thought that was the explanation to my compile issue.....I can only report what I see.
And what I am now seeing......is that my new setup is allowing me to load quite large make dependencies and still not crash

[Rich]

Hi aus9

... hmm  thats no help for dumbos like me  ...

Let me try explaining it another way:
1. For x86, your CPU can address 4 Gbytes of stuff. It's a 4 lb sack.
2. You have 8 Gbytes of RAM. Lets call that 8 lb of shit.
3. Your peripherals (video, sound, network, disk, etc.) want another
   3 Gbytes of space. That's an additional 3 lb of crap.
 
That takes care of the groundwork.

You can't fit 11 lb of shit and crap into a 4 lb sack.

It's first come first served, therefore:
If your 4 lb sack (x86 CPU) contains 3 lb of crap (peripherals), it can only hold 1 lb of shit (RAM).

Result:
The other 7 lb of shit (RAM) remain unused.

Note: For a metric explanation, replace each occurrence of  lb  with  kg.

[aus9]

Hi Rich

I am not seeing 4 Kilos in total?

Code: [Select]

free -m
              total        used        free      shared  buff/cache   available
Mem:           2467         809        1097          72         561        1371
Swap:          8000           0        8000

[Rich]

Hi aus9
That's because your peripherals also use address space. Every address
that a peripheral uses is one less address that can be used to access RAM.

Total is installed RAM or the amount of address space space available for
RAM, whichever is less.

[aus9]

Hi Rich

Ahhh   well that is good to know and me thinks that explains why I was confused....other than losing brain cells to aussie beer

OK I reverse my reluctance to go 64 bit kernel and modules. Thanks for caring

[aus9]

Hi Rich

Ok I am on TC32 with K=64bit and now we have

Code: [Select]

tc@box:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:           7400         657        6405          53         337        6459
Swap:          8000           0        8000
Rich....Not sure if its intentional but I can not spot a difference in the files for /etc/os-release or /etc/issue
I can prove I am on TC32 using a readelf command....not that anyone cares but thought I would mention it here

To anyone interested

trivia my 40_custom grub entries
TC64 is a chainload to grub in PBR for partition 3
next is TC32 running 64 bit kernel and modules
bottom is "normal for me" TC32 with 32 bit kernel
I decided to keep the virtual memory allocation vaule as I now have plenty of RAM
I now blacklist my sound module so I have a one liner in my bootlocal.sh to disable hdmi

menuentry "sda3" {
set root=(hd0,msdos3)
chainloader +1
}

menuentry "sda4K64" {
set root=(hd0,msdos4)
linux /grub/vmlinuz64 tce=sda4 home=sda4 waitusb=5 nozswap blacklist=snd_hda_intel vmalloc=512M
initrd /grub/amd-ucode.img /grub/rootfs.gz /grub/modules64.gz
}

menuentry "sda4onboot" {
set root=(hd0,msdos4)
linux /grub/vmlinuz tce=sda4 home=sda4 waitusb=5 nozswap blacklist=snd_hda_intel vmalloc=512M
initrd /grub/amd-ucode.img /grub/core.gz
}

thanks for reading

Just compiled a package and it works over reboot on TC32 kernel so I am good to go.

[Rich]

Hi aus9

... Rich....Not sure if its intentional but I can not spot a difference in the files for /etc/os-release or /etc/issue ...

You are running Kernel64 which loads Modules64+rootfs.
Aside from the base 32 bit libraries (/lib/), rootfs also contains  /etc/  and
all of the other root directories you see if you run:

Code: [Select]

ls /
rootfs  is the 32 bit component in this mix that allows you to run 32 bit programs.
It is the same  rootfs  used in 32 bit Tinycore.