Re: Tiny Core v17.0 upgrade issues

[Rich]

Hi Stefann

Code: [Select]

tce-load -wi advcomp

[Stefann]

Hi Stefann
Install  advcomp.tcz.

Almost!…… This worked:

Code: [Select]

tc@hp510:~$ tce-load -wi advcomp.tczBut thanks for the tcz name. I would not have found that otherwise

But no luck…

Code: [Select]

tc@hp510:/krubo/work/TC/linux-6.18.28/arch/x86/boot$ advdef -z4 bzImage
File type not supported on bzImage [at void convert_inplace(const string&):redef.cc:498]
Note…
Probably no activity anymore today.
As said I’m on a trip.
Social obligations now and sleep after that (living in Europe/netherlands)

[Paul_123]

It wants the file name to end in .gz    But if you did not have advcomp.tcz installed when compiling the kernel, then it did not get fully compressed.

With advcomp installed just remove the old bzImage from arch/x86/boot.    Then run "make bzImage" again

It should go faster as it should not have to recompile most of the code from the last run.

[Stefann]

It wants the file name to end in .gz    But if you did not have advcomp.tcz installed when compiling the kernel, then it did not get fully compressed.

With advcomp installed just remove the old bzImage from arch/x86/boot.    Then run "make bzImage" again

It should go faster as it should not have to recompile most of the code from the last run.

Done, but not really different size

Code: [Select]

tc@hp510:/krubo/work/TC/linux-6.18.28/arch/x86/boot$ ls -l bz*
-rw-r--r--    1 tc       staff      6312448 May 28 21:43 bzImage
Ehh… I get in social trouble if I spend more time on this now :-)
Any input will be followed up tomorrow

[Paul_123]

Lets look at compressed and uncompressed kernel.  Does vmlinux.bin match your build?

Code: [Select]

tc@Devel:~/linux-6.18.28/arch/x86/boot/compressed$ ls -l vmlinux*
-rwxr-xr-x 1 tc staff  6107368 May 29 23:38 vmlinux
-rwxr-xr-x 1 tc staff 12469728 May 29 23:38 vmlinux.bin
-rw-r--r-- 1 tc staff  6046727 May 29 23:38 vmlinux.bin.gz
-rw-r--r-- 1 tc staff     3077 May 29 23:38 vmlinux.lds
-rw-rw-r-- 1 tc staff     2308 May  8 06:40 vmlinux.lds.S
-rw-r--r-- 1 tc staff   666652 May 29 23:38 vmlinux.relocs


[Stefann]

Here you go…
Looks like you are right (but please check, I don’t know whether I compare the correct files)

Compressed vmlinux and vmlinux.bin.gz are different
But uncompressed other files have same size.

That’s good!
If it’s the compressing I doubt that that would cause the crashing

Code: [Select]

tc@hp510:/krubo/work/TC/linux-6.18.28/arch/x86/boot/compressed$ ls -l vm*
-rwxr-xr-x    1 tc       staff      6308072 May 27 12:42 vmlinux
-rwxr-xr-x    1 tc       staff     12469728 May 27 12:42 vmlinux.bin
-rw-r--r--    1 tc       staff      6248050 May 27 12:42 vmlinux.bin.gz
-rw-r--r--    1 tc       staff         3077 May 27 12:42 vmlinux.lds
-rw-r--r--    1 tc       staff         2308 May  8 08:40 vmlinux.lds.S
-rw-r--r--    1 tc       staff       666652 May 27 12:42 vmlinux.relocs
On the crashing…
As said I’m now on a trip AND have a major 1wire hardware breakdown to fix on return.
So… expect things to go a little bit slower…

With that said…. I will more thorough test whether recompiled kernel with usb-serial-6.18.2-tinycore.tcz works.
If indeed my “kernel making” is identical to your “kernel making” than my custom kernel should be identical to the published kernel and should crash. In that case the crashing is related to usb-serial “included in kernel versus by kext”.
That is entirely possible because I included only a few modules in the kernel while the usb-serial-6.18.2-tinycore.tcz has much more.

[Stefann]

small update..
- I returned from my trip
- located 1wire HW issue, the usb-to-1wire interface adapter was broken so the full network was down. I luckily had a spare. It's running again (hurray!)

I picked up the TC17 trouble shooting on usb-serial:
- running on standard configuration self compiled TC17 6.18.2 kernel with usb-serial-6.18.2-tinycore.tcz
- started my application without serial read PLUS dedicated crash-test-program with 200x accelerated serial read

running under kernel-from-downloadlocation this crashed 2x on a row, after 14hr and 12hr.
So... This "should" also crash within at least 24hr
If so... usb-serial-6.18.2-tinycore.tcz would be suspect
But...... no speculation... just test...

[Stefann]

small update
After 24hr still no crash.
I was expecting a crash:
- kernel from TC17.0 download location + usb-serial-6.18.2-tinycore.tcz had crashed 2x in a row after 14hr and 12hr
- now self compiled TC17.0 kernel +usb-serial-6.18.2-tinycore.tcz had NO crash for 24hrs after which I manually stopped it.

However... I realized I had recompiled my application under TC15 because I had been running TC15 during my trip.
Already many weeks ago I had found that "application compiled under TC15 does not crash on TC17"

So...
As every little damn detail counts...
I decided to do a full clean start.

deleted self compiled kernel and made a clean start for TC17.0 linux 6.18.2:

Code: [Select]

download http://tinycorelinux.net/17.x/x86/release/src/kernel/linux-6.18.2/config-6.18.2-tinycore
download http://tinycorelinux.net/17.x/x86/release/src/kernel/linux-6.18.2-patched.tar.xz

tc@hp510:/krubo/work/TC$ tar xvf  linux-6.18.2-patched.tar.xz
tc@hp510:/krubo/work/TC$ cd linux-6.18.2
tc@hp510:/krubo/work/TC/linux-6.18.2$
tc@hp510:/krubo/work/TC/linux-6.18.2$ cp ../config-6.18.2-tinycore .config
tc@hp510:/krubo/work/TC/linux-6.18.2$ make mrproper
  CLEAN   .config
tc@hp510:/krubo/work/TC/linux-6.18.2$ cp ../config-6.18.2-tinycore .config
tc@hp510:/krubo/work/TC/linux-6.18.2$ make oldconfig
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/confdata.o
  HOSTCC  scripts/kconfig/expr.o
  LEX     scripts/kconfig/lexer.lex.c
  YACC    scripts/kconfig/parser.tab.[ch]
  HOSTCC  scripts/kconfig/lexer.lex.o
  HOSTCC  scripts/kconfig/menu.o
  HOSTCC  scripts/kconfig/parser.tab.o
  HOSTCC  scripts/kconfig/preprocess.o
  HOSTCC  scripts/kconfig/symbol.o
  HOSTCC  scripts/kconfig/util.o
  HOSTLD  scripts/kconfig/conf
#
# configuration written to .config
#
tc@hp510:/krubo/work/TC/linux-6.18.2$ make bzImage
...................lots of logging................
tc@hp510:/krubo/work/TC/linux-6.18.2$

tc@hp510:/krubo/work/TC/linux-6.18.2$ cd arch/x86/boot
tc@hp510:/krubo/work/TC/linux-6.18.2/arch/x86/boot$ ls -l bz*
-rw-r--r--    1 tc       staff      6087168 Jun  2 17:00 bzImage

tc@hp510:/krubo/work/TC/linux-6.18.2/arch/x86/boot$ cd compressed
tc@hp510:/krubo/work/TC/linux-6.18.2/arch/x86/boot/compressed$ ls -l vm*
-rwxr-xr-x    1 tc       staff      6082840 Jun  2 17:00 vmlinux
-rwxr-xr-x    1 tc       staff     12428768 Jun  2 16:55 vmlinux.bin
-rw-r--r--    1 tc       staff      6025154 Jun  2 17:00 vmlinux.bin.gz
-rw-r--r--    1 tc       staff         3106 Jun  2 16:55 vmlinux.lds
-rw-r--r--    1 tc       staff         2296 Dec 18 14:03 vmlinux.lds.S
-rw-r--r--    1 tc       staff       663872 Jun  2 16:55 vmlinux.relocs
 
After that:

Code: [Select]

- copy to production machine
- reboot towards this new kernel
- recompile application
- recompile accelerated crash program
- start application
- start accelerated crash program
And now its running again....
It should crash in 12..14hr.
if not the "self recompile" is guilty (very very strange)
Note, self compiled kernel (vmlinuz17C2x) now has identical size to kernel from download area (vmlinuz):

Code: [Select]

tc@huis:/mnt/sda1/tce/boot$ ls -l vm*
-rwxrw-r--    1 tc       staff      6087168 Feb 10 13:10 vmlinuz
-rw-rw-r--    1 tc       staff      6087168 Jun  2 17:11 vmlinuz17C2xSo... apparently the install of advcomp.tcz fixed that

Lets wait...

[Stefann]

small update....
it's weird...
I'm running under self compiled TC17.0 kernel with usb-serial-6.18.2-tinycore.tcz, linux 6.18.2
I'm running my application without serial read
In parallel I'm running a dedicated crash-program with 200x accelerated read (reading blocks of 2 bytes versus 812byte blocks with my normal application)
I made this kernel "very clean" (see last post); I fully recompiled my application and crash program under this kernel.

It's now running 22.5hr without issue:

Code: [Select]

mon day:  6  2 | hh:mm:ss: 20:18:50 | Still alive after 0k + 1 loops | 0k + 2 bytes read
mon day:  6  3 | hh:mm:ss: 18:41:05 | Still alive after 31324k + 802 loops | 62588k + 776 bytes read
May 20-23 I was running this exact same test under kernel copied from iso from TC download location and it crashed 2x in a row after 14hr and 12hr.

Comparing kernels: vmlinuz from TC download location versus self compiled vmlinux17C2x:

Code: [Select]

tc@huis:/mnt/sda1/tce/boot$ ls -l vm*
-rwxrw-r--    1 tc       staff      6087168 Feb 10 13:10 vmlinuz
-rw-rw-r--    1 tc       staff      6087168 Jun  2 17:11 vmlinuz17C2x
tc@huis:/mnt/sda1/tce/boot$ cmp vmlinuz vmlinuz17C2x
vmlinuz vmlinuz17C2x differ: char 105, line 1So: size is exactly the same but they still differ

Anyway...
- I will keep this running for max 3 extra days. If it survives 4 days I will "declare it stable"
- IF it does not crash than the problem is not in the linux source but likely in the TC make sequence
- In that case I will for TC17.0 stop trouble shooting but analyse again with beta release of TC17.1 as I expect the TC team is more active with that release.

[Rich]

Hi Stefann

... So: size is exactly the same but they still differ ...

It's possible a timestamp or some other unique identifier gets added
to vmlinuz.

That's easy enough to confirm. Compile another vmlinuz and compare
it to the one you previously compiled.

[Stefann]

Thanks @Rich
I also thought... "maybe I just have 1 corrupted byte in my downloaded vmlinuz", I actually never checked a checksum and/or redownloaded

So...
That's what I did... redownload and compare:

Code: [Select]

tc@huis:/tmp$ mkdir TC
tc@huis:/tmp$ cd TC
tc@huis:/tmp/TC$ wget --header='Accept-Language: en-us,en;q=0.5' http://tinycorelinux.net/17.x/x86/release/Core-17.0.iso
Connecting to tinycorelinux.net (128.127.66.77:80)
saving to 'Core-17.0.iso'
Core-17.0.iso        100% |***************************************************************************| 19.5M  0:00:00 ETA
'Core-17.0.iso' saved

tc@huis:/tmp/TC$ wget --header='Accept-Language: en-us,en;q=0.5' http://tinycorelinux.net/17.x/x86/release/Core-17.0.iso.md
5.txt
Connecting to tinycorelinux.net (128.127.66.77:80)
saving to 'Core-17.0.iso.md5.txt'
Core-17.0.iso.md5.tx 100% |***************************************************************************|    48  0:00:00 ETA
'Core-17.0.iso.md5.txt' saved
tc@huis:/tmp/TC$ md5sum -c Core-17.0.iso.md5.txt
Core-17.0.iso: OK

tc@huis:/tmp/TC$ mkdir tc17
tc@huis:/tmp/TC$ ls
Core-17.0.iso  tc17/
tc@huis:/tmp/TC$ sudo mount Core-17.0.iso tc17
tc@huis:/tmp/TC$ cd tc17
tc@huis:/tmp/TC/tc17$ cd boot
tc@huis:/tmp/TC/tc17/boot$ ls
core.gz   isolinux/ vmlinuz
tc@huis:/tmp/TC/tc17/boot$ cp vmlinuz /mnt/sda1/tce/boot/vmlinuz_org2
tc@huis:/tmp/TC/tc17/boot$ cd /mnt/sda1/tce/boot

tc@huis:/mnt/sda1/tce/boot$ ls -l vm*
-rwxrw-r--    1 tc       staff      6087168 Feb 10 13:10 vmlinuz
-rw-rw-r--    1 tc       staff      6087168 Jun  2 17:11 vmlinuz17C2x
-r--r--r--    1 tc       staff      6087168 Jun  4 08:35 vmlinuz_org2
tc@huis:/mnt/sda1/tce/boot$ cmp --verbose vmlinuz vmlinuz_org2
tc@huis:/mnt/sda1/tce/boot$                          // no respons so they are equalso... no... the vmlinuz that has been crashing so often is exactly equal to a freshly downloaded one.

Also checked difference between my selfmade vmlinuz17C2x and downloaded vmlinuz in some more detail:

Code: [Select]

tc@huis:/mnt/sda1/tce/boot$ cmp -b vmlinuz vmlinuz17C2x
vmlinuz vmlinuz17C2x differ: byte 105, line 1 is 271 M-9 211 M-^I

tc@huis:/mnt/sda1/tce/boot$ cmp --verbose vmlinuz vmlinuz17C2x
    105 271 211
    106 201 210
    589 363 302
    590 350 357
    613 314 234
    614 101 110
    617 234 174
    618 205 214
   9463 210 200
   9469 334 324
   9565 210 200
   9572 334 324
  10472 210 200
  10479 334 324
  10539 210 200
  10545 334 324
  10594 334 324
  10670 210 200
  10701 334 324
  10903 210 200
  10914 334 324
  11658 174 164
  11665 224 214
  11672 160 150
  11679 210 200
  11686 144 134
  12357 110 100
  13099 260 250
  13848 314 304
  15381 116 150
  15382 125 160
  15383 103  65
  .............. and it keeps going......
  ........ just skipping the middle part....
  6085719   0 123
6085721   0 145
6085723   0 145
6085725   0 144
6085729   0 123
6085731   0 145
6085733   0 143
6085735   0 165
6085737   0 162
6085739   0 145
6085741   0 102
6085743   0 157
6085745   0 157
6085747   0 164
6085751   0 123
6085753   0 145
6085755   0 164
6085757   0 165
6085759   0 160
6085761   0 115
6085763   0 157
6085765   0 144
6085767   0 145...... so... the filesize being equal is likely related to snapping to some blocksize
They are really different

Anyway...
Its still did not crash
I keep it running...

And also as said in previous post:
- If this turns out to be stable my conclusion would be "sources do not have bug", it's in the "make process". I will than keep using the custom TC17.0 kernel. There is not really a downside on that.
- I will than wait for TC17.1 beta to check further. "maybe" custom make will than match make by TC-dev-team and "if not" that would be a more appropriate time to investigate.

[Stefann]

@rich,
I think you are right with "some ID differences" if you remake.
bzImage is a compressed bucket from many files. All compiled files will have a "now" timestamp so for sure the compressed bucket will have byte-level differences.
I remade bzImage and compared to earlier self made kernel
- size is equal
- lots of byte differences
- however.... the "length" until zero byte content begins is different

Anyway....

program & crashtest are still running after 65 hrs (2.5day)
I stick to my earlier plan:
- keep running
- if it survives 4days I declare it stable
- I will start using this custom made kernel. As it is generated from same source as formal TC17.0 I'm not missing out on functionality
- Its probably some "snatch" in the make process, I will explore a bit further with the release of TC17.1 beta

Code: [Select]

tc@huis:/mnt/sda1/tce/boot$ ls -l vmlinuz17C2*
-rw-rw-r--    1 tc       staff      6087168 Jun  2 17:11 vmlinuz17C2x
-rw-r--r--    1 tc       staff      6087168 Jun  5 12:51 vmlinuz17C2xx
tc@huis:/mnt/sda1/tce/boot$ cmp vmlinuz17C2x vmlinuz17C2xx
vmlinuz17C2x vmlinuz17C2xx differ: char 105, line 1

tc@huis:/mnt/sda1/tce/boot$ cmp --verbose vmlinuz17C2x vmlinuz17C2xx
    105 211 171
    589 302 261
    613 234 214
    617 174 134
  15395 124 106
  15396 165 162
  15397 145 151
  15404  62  65
  15407  66  61
  15409  65  60
  15410  65  64
  15412  61  60
  15413  62  63
  16527 130 110
4956393 334  33
4956394 106 107
4956396 145 345
4956397 352 324
4956398 174 371
4956399 245 112
4956400 117 237
4956401 201   2
4956402 311 231
4956403 226  55
4956404  52 325
4956405  63  44
4956407 222 226
4956411  75  35
4956412 316 317
4956413 233 313
4956414 255 326
  .............. and it keeps going......
  ........ just skipping the middle part....
6085688  56   0
6085689  12 145
6085691 151 145
6085693 156 144
6085695 151   0
6085697 164 123
6085699 162 145
6085701 144 143
6085703  75 165
6085705   0 162
6085707 122 145
6085709 141 102
6085711 156 157
6085713 144 157
6085715 157 164
6085717 155   0
6085723 145 164
6085725 144 165
6085727   0 160
6085729 123 115
6085731 145 157
6085733 143 144
6085735 165 145
6085737 162   0
6085739 145   0
6085741 102   0
6085743 157   0
6085745 157   0
6085747 164   0
6085751 123   0
6085753 145   0
6085755 164   0
6085757 165   0
6085759 160   0
6085761 115   0
6085763 157   0
6085765 144   0
6085767 145   0