Re: Tiny Core v17.0 upgrade issues

[Stefann]

Boring update...
it still runs..

Code: [Select]

mon day:  5  8 | hh:mm:ss: 15:07:48 | Still alive after 0k + 1 loops
.............
mon day:  5  9 | hh:mm:ss: 15:09:47 | Still alive after 219k + 471 loops

[Stefann]

Bingo! crashed!
After 45 hours
The screensaver was not active but it did not help me much
flwm was active but did simply show a frozen screen.
I have no mouse but keyboard did not give any response.

I just restart without too much intelligence for reason I need the automizations that my application is running.
I will think of proper followup later, no time now.
I will anyways start it in text mode. "maybe console shows more clues when crashing in text mode"

But.... this is first time I was able to get TC17 crashing with a very simple isolated "crash testcase program" (as published few posts back).
==> I would say... "this is definitely a kext usb-serial-6.18.2-tinycore.tcz TC17 issue"

Any advice is welcome

Code: [Select]

First logging: mon day:  5  8 | hh:mm:ss: 15:07:48 | Still alive after 0k + 1 loops

.............

Last logging: mon day:  5 10 | hh:mm:ss: 12:21:54 | Still alive after 414k + 476 loops

[Rich]

Hi Stefann

... The screensaver was not active but it did not help me much
flwm was active but did simply show a frozen screen.
I have no mouse but keyboard did not give any response. ...

Did you try hitting  Ctrl-Alt-F1 ?
Normally that should switch you to the console so you can see messages.

Ctrl-Alt-F2  returns you to the GUI>

[Stefann]

Hi Stefann

... The screensaver was not active but it did not help me much
flwm was active but did simply show a frozen screen.
I have no mouse but keyboard did not give any response. ...

Did you try hitting  Ctrl-Alt-F1 ?
Normally that should switch you to the console so you can see messages.

Ctrl-Alt-F2  returns you to the GUI>

Ah.. no..
And I rebooted already.
I did not really have time to explore a lot.

Anyway..
It’s now running in textmode. Maybe it will crash as well.
At least “getting it to crash” is kind of predictable.
Both my application and this isolated crash program crashed after about 45 hrs.
I keep it running for 48 hrs again. If it does not crash I try again with gui and take your inputs.

Progress is slow but steady.

At some point I will also test TC16 to see whether it did break with TC16 or TC17

[Stefann]

boring update... its still running..

Code: [Select]

mon day:  5 10 | hh:mm:ss: 13:49:56 | Still alive after 0k + 1 loops
.....
mon day:  5 12 | hh:mm:ss: 15:04:58 | Still alive after 453k + 670 loops
Its now running significantly longer than previous run that crashed after about 414k loops.

Configuration changes are:
- I removed the "interrupt stress inducing lines" from the code because that did not seem to add much. It crashed after 45hours while "with the read() as part of my application" without additional stress it also crashed after 45hrs
- I now booted into text mode instead of GUI

So...
- either its natural variation
- or the lower stress increases the "time to crash" (maybe to indefinite)

I leave it running for an other day. If it did not crash by than I bring it back to previous configuration and try to crash it and look into console using the CTR-ALT-F1 & F2 keys.
Note,... It would have been much handier to keep the system in crashed state and await suggestions on this forum BUT the system is actively running my home automation. The home automation crashes it goes into all kind of "safe fallback modes" but I donot want to have it stalled for to long time. Also depends on my agenda. If I'm home I can afford it to be down better than when I'm not home.

[Stefann]

Ok... lots to say...

Yesterday I was off. Today the crash program was still running. I manually stopped it

Code: [Select]

mon day:  5 10 | hh:mm:ss: 13:49:56 | Still alive after 0k + 1 loops
.....
mon day:  5 14 | hh:mm:ss: 13:19:37 | Still alive after 878k + 109 loops
So:
- with crashprogram that no longer has "stress inducing interrupts" and "TC17 with text boot option" I see no crash in 4 days.

Next step I tried:
- install TC16
However... Install worked, but I did not get it running.
I tried TC16.0 and TC16.1 but for some reason I could not get them to run.

syslog did start but it looks like it stalls when trying to load core.gz.

I noticed that on my MacBook TC16 core.gz is downloaded as core.cpio. I tried booting core.cpio but without success.
I changed name in syslog.conf to core.cpio; I also tried renaming core.cpio to core.gz but neither worked.
>> I abandon that path.

Next step..
I downloaded sources from TC15, TC16 and TC17 and looked into ./drivers/usb/serial
I did run a diff on usb-serial.c
>> note, I have no clue whether this is the file I need, I just guessed based on filename.

The most suspicious difference I did see:

Code: [Select]

TC15:
> tty = tty_port_tty_get(&port->port);
> if (tty) {
> tty_vhangup(tty);
> tty_kref_put(tty);
> }

TC17:
< tty_port_tty_vhangup(&port->port);
This (I think) means that TC15 is testing tty to be valid before doing tty_vhangup() by calling tty while TC17 directly closes the port without testing whether the associated tty exists.

I did some google and found on https://docs.kernel.org/driver-api/tty/tty_port.html:

The tty port has a different lifetime to the tty so must be kept apart. In addition be careful as tty -> port mappings are valid for the life of the tty object but in many cases port -> tty mappings are valid only until a hangup so don’t use the wrong path.

please understand, I have no clue about the logic of this code, but I "can" imagine a hangup_command that is run in kernel mode on a non-existant port to cause a crash.

Next.....
- for now I started the crash-test again including stress commands and including gui. With a little luck it fails again in 45hr and than I can see whether I can jump out the gun with CNTR-ALT-F1 as suggested by @Rich.

Followup, and I could use some help here...
1/
It would be nice if I could understand why TC16 does not run. The diff between TC16 and TC17 on usb-serial.c is almost nothing.
> its above mention tty check
> and also use of "guard" instead of "spin_lock"
So if TC16 runs without crash it is a further evidence that the issue is in one of above 2 changes.

So... any advice on "why boot gets stuck on TC16" is welcome

2/
would there be any way that I could recompile the core with usb-serial.c fixed for the tty check?
Please understand that I have zero experience in compiling core.
I would also be very happy to test a core that someone prepared for me.

3/
Can anyone confirm that usb-serial-6.18.2-tinycore.tcz is indeed the usb-serial.c file??
Note,... would simply using  usb-serial-6.6.8-tinycore.tcz  (TC15) or  usb-serial-6.12.11-tinycore.tcz  (TC16) have any potential to work??
Especially the TC16 should have potential as the only difference is in the tty_vhangup() and the spinlock.

And/or... any way I could get a usb-serial-x.y.z-tinycore.tcz compiled for TC17 based on TC16 usb-serial.c?

For information
diff of usb-serial.c TC17 TC15

Code: [Select]

$$ TC17 crash work % diff TC17_drivers_usb_serial/usb-serial.c TC15_drivers_usb_serial/usb-serial.c
709c709
< guard(mutex)(&usb_dynids_lock);
---
> spin_lock(&drv->dynids.lock);
711a712
> spin_unlock(&drv->dynids.lock);
714a716
> spin_unlock(&drv->dynids.lock);
1178a1181
> struct tty_struct *tty;
1193c1196,1200
< tty_port_tty_vhangup(&port->port);
---
> tty = tty_port_tty_get(&port->port);
> if (tty) {
> tty_vhangup(tty);
> tty_kref_put(tty);
> }
1455c1462
<  * __usb_serial_register_drivers - register drivers for a usb-serial module
---
>  * usb_serial_register_drivers - register drivers for a usb-serial module
1457d1463
<  * @owner: owning module
1464,1466c1470,1472
< int __usb_serial_register_drivers(struct usb_serial_driver *const serial_drivers[],
<   struct module *owner, const char *name,
<   const struct usb_device_id *id_table)
---
> int usb_serial_register_drivers(struct usb_serial_driver *const serial_drivers[],
> const char *name,
> const struct usb_device_id *id_table)
1511d1516
< (*sd)->driver.owner = owner;
1519c1524
< rc = driver_attach(&udriver->driver);
---
> rc = driver_attach(&udriver->drvwrap.driver);
1530c1535
< EXPORT_SYMBOL_GPL(__usb_serial_register_drivers);
---
> EXPORT_SYMBOL_GPL(usb_serial_register_drivers);
diff of usb-serial.c TC17 TC16

Code: [Select]

$$ diff TC17_drivers_usb_serial/usb-serial.c TC16_drivers_usb_serial/usb-serial.c
709c709
< guard(mutex)(&usb_dynids_lock);
---
> spin_lock(&drv->dynids.lock);
711a712
> spin_unlock(&drv->dynids.lock);
714a716
> spin_unlock(&drv->dynids.lock);
1178a1181
> struct tty_struct *tty;
1193c1196,1200
< tty_port_tty_vhangup(&port->port);
---
> tty = tty_port_tty_get(&port->port);
> if (tty) {
> tty_vhangup(tty);
> tty_kref_put(tty);
> }

[Stefann]

to add,
I'm chatting with my brother, who is a bit of linux guru..

He also thinks these are suspicious:

<    guard(mutex)(&usb_dynids_lock);
---
>    spin_lock(&drv->dynids.lock);
711a712
>          spin_unlock(&drv->dynids.lock);
714a716
>    spin_unlock(&drv->dynids.lock);
1178a1181

Especially because the "spin_lock()" seems to be replaced by "guard()" but its unclear what happened to the 2x "spine_unlock()"

This folds back to Commit 0b3144d
https://github.com/torvalds/linux/commit/0b3144da31f855fce652303f588416a60991bdef

USB: make single lock for all usb dynamic id lists
There are a number of places where we accidentally pass in a constant
structure to later cast it off to a dynamic one, and then attempt to
grab a lock on it, which is not a good idea.  To help resolve this, move
the dynamic id lock out of the dynamic id structure for the driver and
into one single lock for all USB dynamic ids.  As this lock should never
have any real contention (it's only every accessed when a device is
added or removed, which is always serialized) there should not be any
difference except for some memory savings.

Note, this just converts the existing use of the dynamic id lock to the
new static lock, there is one place that is accessing the dynamic id
list without grabbing the lock, that will be fixed up in a follow-on
change.

tty-check change folds back to Commit 2b5eac0
https://github.com/torvalds/linux/commit/2b5eac0f8c6e79bc152c8804f9f88d16717013ab

tty: introduce and use tty_port_tty_vhangup() helper
This code (tty_get -> vhangup -> tty_put) is repeated on few places.
Introduce a helper similar to tty_port_tty_hangup() (asynchronous) to
handle even vhangup (synchronous).

And use it on those places.

In fact, reuse the tty_port_tty_hangup()'s code and call tty_vhangup()
depending on a new bool parameter.

I traced the code on this. As far as I can see the new code does no longer have the check on "tty being existing"