Can't start Xorg after installing open-vm-tools-desktop: "Illegal instruction"

[denbkh]

Hello!
I downloaded the v9.0 CorePlus-current.iso and installed it using vmware workstation 12.5.9 build-7535481. Then I installed open-vm-tools-desktop(10.2.5.3619) and did a reboot. After that I can't get gui working anymore.
When I try to start Xorg I have this error:

Code: [Select]

[   141.834] (EE) AIGLX: reverting to software rendering
[   141.840] (EE)
[   141.840] (EE) Backtrace:
[   141.840] (EE) 0: /usr/local/lib/xorg/Xorg (xorg_backtrace+0x43) [0x8141cef]
[   141.840] (EE) 1: /usr/local/lib/xorg/Xorg (0x8048000+0x11ce65) [0x8164e65]
[   141.840] (EE) 2: linux-gate.so.1 (__kernel_rt_sigreturn+0x0) [0xb7fd5b20]
[   141.840] (EE) 3: /usr/local/lib/dri/swrast_dri.so (0xb4c8b000+0x1ee0ba) [0xb4e790ba]
[   141.840] (EE) 4: /usr/local/lib/dri/swrast_dri.so (0xb4c8b000+0x1ee419) [0xb4e79419]
[   141.840] (EE) 5: /usr/local/lib/dri/swrast_dri.so (0xb4c8b000+0x1ee9ce) [0xb4e799ce]
[   141.840] (EE) 6: /usr/local/lib/dri/swrast_dri.so (0xb4c8b000+0x1eeaf4) [0xb4e79af4]
[   141.840] (EE) 7: /usr/local/lib/xorg/modules/extensions/libglx.so (0xb7a5c000+0x17321) [0xb7a73321]
[   141.840] (EE) 8: /usr/local/lib/xorg/modules/extensions/libglx.so (0xb7a5c000+0x1c32a) [0xb7a7832a]
[   141.840] (EE) 9: /usr/local/lib/xorg/Xorg (InitExtensions+0x5e) [0x80d80bb]
[   141.840] (EE) 10: /usr/local/lib/xorg/Xorg (0x8048000+0x1ffae) [0x8067fae]
[   141.840] (EE) 11: /lib/libc.so.6 (__libc_start_main+0x156) [0xb7afa7c6]
[   141.840] (EE) 12: /usr/local/lib/xorg/Xorg (0x8048000+0x20791) [0x8068791]
[   141.840] (EE)
[   141.840] (EE) Illegal instruction at address 0xb4fc54a3

Please check full Xorg log file in attachment.

[andyj]

Do you have 3D hardware acceleration enabled under the display adapter options? Is this 32-bit or 64-bit? What host are you using, and does it have a working 3D video driver? There are known problems with software rendering. More precisely, we know there are problems, we just haven't found THE problem or any solutions yet. Also, you may need to add:

Code: [Select]

mks.gl.allowBlacklistedDrivers = "TRUE"
to your ~/.vmware/preferences file.

[denbkh]

Host: Windows 7 Pro (x64) SP1 - with all updates
Host video driver: NVIDIA 390.77 - works without any problems in 3d apps
Tinycore: 9.0 x86
uname -a: Linux box 4.14.10-tinycore #876 SMP Mon Jan 1 15:00:32 UTC 2018 i686

My first run was with 3d acceleration disabled in VM display settings.
But with 3d acceleration enabled I have the same "illegal instruction" bug. Please check the attached log file.

Adding mks.gl.allowBlacklistedDrivers = "TRUE" to the preferences file did not help, got the same error.

I also attached the .vmx file with VM settings, hope this can be useful.

[andyj]

I have tried to replicate the problem on my work laptop. It's a Dell 7510. According to the NVIDIA system information dialog, it is Windows 7 Pro 64-bit SP1 (version 6.1.7601) and NVIDIA video driver version 390.77. I am using VM Player 12.5.7, a new VM with TC 9.0 32-bit and open-vm-tools-desktop 10.2.5.3619, so other than the slight VM Player version difference so far everything is about the same.

What is different is that it works for me. I cannot reproduce the problem. My onboot.lst file looks like this:

Code: [Select]

open-vm-tools-desktop.tcz
flwm_topside.tcz
aterm.tcz
wbar.tcz

Somewhere in your dmesg output you should have something like:

Code: [Select]

[drm] DX: yes.
[drm] Atomic: yes
fbcon: svgadrmfb (fb0) is primary device
Console: switching to colour frame buffer device 160x48
[drm] Initialized vmwgfx 2.14.0 20170612 for 0000:00:0f.0 on minor 0

If DX is not yes, then 3d is not enabled.

[denbkh]

I have the same DX: yes in dmesg. I attached the dmesg log file.
Changing the onboot.lst did not fix the bug.
Could you please upload your VM to any file hosting service (dropbox, google drive, etc.. ) and post a link? I will try to run it.

[denbkh]

Thank you for the VM link in pm message.
I have run it and got the same bug with "Illegal instruction at address 0xb4fc54a3"

Can you try to run it with the latest VMware 12.5.9?
In the VMware changelog they mention adding some "Spectre" (CVE-2017-5715) mitigation ... maybe this could be the source of the problem ...

p.s. I have tried to reply on your pm message twice, but it look like broken - can't see my reply in the "sent items" (http://forum.tinycorelinux.net/index.php?action=pm;f=sent)

[Rich]

Hi denbkh

... p.s. I have tried to reply on your pm message twice, but it look like broken - can't see my reply in the "sent items" (http://forum.tinycorelinux.net/index.php?action=pm;f=sent)

When sending a PM there is a checkbox for saving a copy of your message.

[andyj]

I upgraded my VM Player to 12.5.9 and the VM still works. Decreasing the CPU's to one didn't create the problem either. Neither did decreasing the amount of video memory. We both have Intel CPU's. There are two problems here. One is that when a VM is running on your system X doesn't detect DRM. The second is that software DRM fallback is crashing. When I disable 3D on my system DRM is not detected (DX: no) and software rendering works OK:

Code: [Select]

[     8.150] (II) vmware(0): Initialized VMWARE_CTRL extension version 0.2
[     8.172] (WW) vmware(0): Failed to initialize Gallium3D Xa. No render acceleration available.
[     8.172] (WW) vmware(0): Skipped initialization of direct rendering due to lack of render acceleration.
[     8.172] (--) vmware(0): Render acceleration is disabled.
[     8.172] (==) vmware(0): Rendercheck mode is disabled.
[     8.172] (--) vmware(0): Direct rendering (3D) is disabled.
[     8.172] (==) vmware(0): Backing store enabled
[     8.172] (==) vmware(0): Silken mouse enabled
[     8.174] (II) vmware(0): RandR 1.2 enabled, ignore the following RandR disabled message.
[     8.174] (==) vmware(0): DPMS enabled
[     8.174] (II) vmware(0): No 3D acceleration. Not setting up textured video.
[     8.174] (--) RandR disabled
[     8.174] (II) AIGLX: Screen 0 is not DRI2 capable
[     8.174] (EE) AIGLX: reverting to software rendering
[     8.358] (II) IGLX: enabled GLX_MESA_copy_sub_buffer
[     8.358] (II) IGLX: Loaded and initialized swrast
[     8.358] (II) GLX: Initialized DRISWRAST GL provider for screen 0

I cannot reproduce the problem. Maybe you have something else installed on your system which is interfering with VMware? Another hypervisor like VirtualBox also installed with services running in the background? Maybe check BIOS settings? Maybe update system BIOS?

[denbkh]

... Maybe you have something else installed on your system which is interfering with VMware? Another hypervisor like VirtualBox also installed with services running in the background? Maybe check BIOS settings? Maybe update system BIOS?

Hmm, I don't have VirtualBox on my PC.
The interesting thing is that open-vm-tools-desktop in my LUbuntu 17.10 x86 VM works without any problems with full 3d acceleration. Checked that with glxinfo and glmark2. So that I think the problem is not with my BIOS.
Any ideas how can I help you reproduce the bug?

[andyj]

Those are good points. Did you upgrade VM Player from a previous version, or was it a clean install? When X crashes at startup does it fall back to a console prompt? If so, maybe you could use strace with startx to get some more info? How do you know that Ubuntu is using 3D and not software? Have you checked the Xorg.0.log?

[denbkh]

Did you upgrade VM Player from a previous version, or was it a clean install?

clean install

When X crashes at startup does it fall back to a console prompt?

yes, see the screenshot https://yadi.sk/i/HEQpTH8f3UYdwF

If so, maybe you could use strace with startx to get some more info?

could you please provide the full command?

How do you know that Ubuntu is using 3D and not software?

$ glxinfo | grep "direct rendering"
direct rendering: Yes

I have attached the logs from LUbuntu if they can help.

[andyj]

After it crashes, if you type startx at the console prompt does it work or crash again? Have you tried 64-bit TinyCore? For strace, load the strace extension and try:

Code: [Select]

strace -D -f -s 254 -o strace-x.log startx
Maybe some AV is doing something? I've tried every combination of settings in VM Player I can find and I can't reproduce the problem. It works in spite of everything I do to try to break it.

[denbkh]

After it crashes, if you type startx at the console prompt does it work or crash again?

Yes, it crashes with the same error.

Have you tried 64-bit TinyCore?

Yes. It crashes with the same "Illegal instruction" error, but with 64-bit address.

Maybe some AV is doing something?

I tried to boot VM with disabled AV(Kaspersky free 18.0.0.405(g)) - got the same error.

Please check the log files with strace for both x64 and x86 VMs.
https://yadi.sk/d/0YxnzAaf3UZaYG (can't attach big files here)

[andyj]

Looks like things go bad around here:

Code: [Select]

7467  stat64("/dev/dri/renderD128", {st_mode=S_IFCHR|0660, st_rdev=makedev(226, 128), ...}) = 0
7467  readlink("/sys/dev/char/226:128/device/subsystem", "../../../bus/pci", 4096) = 16
7467  openat(AT_FDCWD, "/sys/dev/char/226:128/device/uevent", O_RDONLY|O_LARGEFILE) = 17
7467  fstat64(17, {st_mode=S_IFREG|0644, st_size=4096, ...}) = 0
7467  read(17, "DRIVER=vmwgfx\nPCI_CLASS=30000\nPCI_ID=15AD:0405\nPCI_SUBSYS_ID=15AD:0405\nPCI_SLOT_NAME=0000:00:0f.0\nMODALIAS=pci:v000015ADd00000405sv000015ADsd00000405bc03sc00i00\n", 4096) = 161
7467  close(17)                         = 0
7467  openat(AT_FDCWD, "/sys/dev/char/226:128/device/vendor", O_RDONLY|O_LARGEFILE) = 17
7467  fstat64(17, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
7467  read(17, "0x15ad\n", 4096)        = 7
7467  close(17)                         = 0
7467  openat(AT_FDCWD, "/sys/dev/char/226:128/device/device", O_RDONLY|O_LARGEFILE) = 17
7467  fstat64(17, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
7467  read(17, "0x0405\n", 4096)        = 7
7467  close(17)                         = 0
7467  openat(AT_FDCWD, "/sys/dev/char/226:128/device/subsystem_vendor", O_RDONLY|O_LARGEFILE) = 17
7467  fstat64(17, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
7467  read(17, "0x15ad\n", 4096)        = 7
7467  close(17)                         = 0
7467  openat(AT_FDCWD, "/sys/dev/char/226:128/device/subsystem_device", O_RDONLY|O_LARGEFILE) = 17
7467  fstat64(17, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
7467  read(17, "0x0405\n", 4096)        = 7
7467  close(17)                         = 0
7467  getdents64(16, /* 0 entries */, 32768) = 0
7467  close(16)                         = 0
7467  SYS_355(0xbff36444, 0x4, 0x1, 0x91cf720, 0x4) = 4
7467  openat(AT_FDCWD, "/etc/drirc", O_RDONLY|O_LARGEFILE) = 16
7467  read(16, "<!--\n\n============================================\nApplication bugs worked around in this file:\n============================================\n\n* Unigine Heaven 3.0 and older contain too many bugs and can't be supported\n  by drivers that want to be complia"..., 4096) = 4096
7467  SYS_355(0xbff36414, 0x4, 0x1, 0x91cf720, 0x4) = 4
7467  read(16, "tion name=\"allow_glsl_builtin_variable_redeclaration\" value=\"true\" />\n        </application>\n\n        <application name=\"Dead Island Riptide Definitive Edition\" executable=\"DeadIslandRiptideGame\">\n            <option name=\"allow_glsl_builtin_variable_red"..., 4096) = 4096
7467  read(16, "<option name=\"mesa_glthread\" value=\"true\"/>\n        </application>\n\n        <application name=\"Borderlands 2\" executable=\"Borderlands2\">\n            <option name=\"mesa_glthread\" value=\"true\"/>\n        </application>\n\n        <application name=\"Civilizati"..., 4096) = 4096
7467  read(16, "l_sync_control\" value=\"true\" />\n        </application>\n    </device>\n    <device driver=\"radeonsi\">\n        <application name=\"ARK: Survival Evolved (and unintentionally the UE4 demo template)\" executable=\"ShooterGame\">\n            <option name=\"radeonsi"..., 4096) = 348
7467  read(16, "", 4096)                = 0
7467  close(16)                         = 0
7467  openat(AT_FDCWD, "/root/.drirc", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
7467  --- SIGILL (Illegal instruction) @ 0 (0) ---

Your CPU is "Intel(R) Xeon(R) CPU E5472  @ 3.00GHz (family: 0x6, model: 0x17, stepping: 0x6)" which I wouldn't think would be a problem.

Did you change any of the NVIDIA driver settings from their defaults? What NVIDIA chip do you have?

[Rich]

Hi andyj
The listing in the first post of this thread looks virtually identical to what is shown here:
http://forum.tinycorelinux.net/index.php/topic,21737.msg136290.html#msg136290