Tiny Core Linux

Tiny Core Extensions => TCE Bugs => Topic started by: GNUser on October 26, 2023, 02:42:41 PM

Title: [Solved] in havoc terminal emulator: yes causes broken pipes
Post by: GNUser on October 26, 2023, 02:42:41 PM
I'm experimenting with wayland on TCL14 x86_64.

Any shell script that relies on the  yes  command causes the  havoc  terminal emulator (with default configuration) to be littered with "Broken pipe" warnings. This includes the  tce-load  shell script:

Code: [Select]
$ tce-load -wil bmon
bmon.tcz.dep OK
Downloading: libconfuse.tcz
Connecting to gnuser.ddns.net (24.0.182.246:80)
saving to 'libconfuse.tcz'
libconfuse.tcz       100% |********************************| 24576  0:00:00 ETA
'libconfuse.tcz' saved
libconfuse.tcz: OK
yes: Broken pipe
Downloading: bmon.tcz
Connecting to gnuser.ddns.net (24.0.182.246:80)
saving to 'bmon.tcz'
bmon.tcz             100% |********************************| 49152  0:00:00 ETA
'bmon.tcz' saved
bmon.tcz: OK
yes: Broken pipe

I've never seen this issue when using  tce-load  in X session, regardless of the terminal emulator being used.

In the case of  tce-load  the warnings are cosmetic. However, the same issue causes some of my shell scripts (those that use  yes  to give encryption password to bcrypt) to hang indefinitely (with upward tick in CPU usage and laptop's fans coming on).

Any ideas what could be causing this "Broken pipe" warning related to  yes  (and how to fix it)?
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: Juanito on October 26, 2023, 03:55:16 PM
The same thing happens with the weston terminal.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: GNUser on October 26, 2023, 03:58:44 PM
Good to know--so it isn't anything specific to havoc. I'll keep digging.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: patrikg on October 26, 2023, 05:03:07 PM
Have you google it??
Can it be like this ??
https://stackoverflow.com/questions/20573282/hudson-yes-standard-output-broken-pipe (https://stackoverflow.com/questions/20573282/hudson-yes-standard-output-broken-pipe)
That the command that yes should send the yes to, have exit before ??
So what command is being used in the script to send yes to ??

Some text from the stackoverflow:
Quote
You could also report the bug against the component that runs the pipeline. The bug is in not restoring SIGPIPE to the default handler after the child is forked. It is what programs expect when they are run in a terminal on POSIX systems. Though I don't know whether there is a standard way to do it for a java-based program. jvm probably raises an exception for every write error so not-dying on SIGPIPE is not a problem for a java program.
So maybe it's the error comes from the shell that handling the signal correct.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: jazzbiker on October 26, 2023, 05:56:48 PM
That the command that yes should send the yes to, have exit before ??

Of course, it is the only way for yes to exit :-)

@GNUser, busybox yes dies silently. Are You sure that exactly busybox yes is envoked? Can You ensure this with the full path call?
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: GNUser on October 26, 2023, 06:43:43 PM
Hi patrikg. Yes, I have done an internet search.
Hi jazzbiker. I don't have coreutils loaded. The only  yes  on my system is the busybox applet.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: curaga on October 27, 2023, 02:10:56 AM
That stackoverflow link does look like the most likely explanation. IOW, it's a bug in those two terminals, havoc and weston terminal.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: GNUser on October 27, 2023, 10:17:52 AM
IOW, it's a bug in those two terminals, havoc and weston terminal.
Darn. In that case, I created this trivial wrapper script for yes, which does away with the errors when using tce-load:
Code: [Select]
#!/bin/sh
exec /usr/bin/yes "$@" 2>/dev/null
But that doesn't solve scripts where yes is needed multiple times in same pipeline. Even with the wrapper script, something like this:
Code: [Select]
yes "password" | bcrypt somefileseems to enter an endless loop, driving CPU usage way up. Workaround for this situation is this kludge:
Code: [Select]
i=1
while true; do
echo "password"
[ $i -eq 2 ] && break
i=$(( $i + 1 ))
done | bcrypt somefile
If I figure out how to properly fix the bug in the terminal emulators, I'll post it here.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: Juanito on October 27, 2023, 10:24:16 AM
Why not file a bug report against weston or havoc?
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: Rich on October 27, 2023, 10:37:13 AM
Hi GNUser
... But that doesn't solve scripts where yes is needed multiple times in same pipeline. Even with the wrapper script, something like this:
Code: [Select]
yes "password" | bcrypt somefileseems to enter an endless loop, driving CPU usage way up. ...
What if you do it like this:
Code: [Select]
echo -e  "password\npassword" | bcrypt somefile
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: GNUser on October 27, 2023, 11:03:38 AM
Why not file a bug report against weston or havoc?
Good idea. Will do.

What if you do it like this:
Code: [Select]
echo -e  "password\npassword" | bcrypt somefile
Wow. Yes, that works and is much more elegant. Thank you!
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: jazzbiker on October 27, 2023, 11:09:20 AM

Darn. In that case, I created this trivial wrapper script for yes, which does away with the errors when using tce-load:
Code: [Select]
#!/bin/sh
exec /usr/bin/yes "$@" 2>/dev/null
But that doesn't solve scripts where yes is needed multiple times in same pipeline. Even with the wrapper script, something like this:
Code: [Select]
yes "password" | bcrypt somefileseems to enter an endless loop, driving CPU usage way up. Workaround for this situation is this kludge:
Code: [Select]
i=1
while true; do
echo "password"
[ $i -eq 2 ] && break
i=$(( $i + 1 ))
done | bcrypt somefile
If I figure out how to properly fix the bug in the terminal emulators, I'll post it here.

Maybe trap instruction in the wrapper will help? Something like
Code: [Select]
#!/bin/sh
trap 'exit 0' SIGPIPE
exec /usr/bin/yes "$@" 2>/dev/null
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: Rich on October 27, 2023, 12:42:44 PM
Hi GNUser
You are welcome. Thanks for confirming it worked.

If you feel like satisfying my curiosity, what happens if you do this:
Code: [Select]
echo -e  "password\npassword\npassword" | bcrypt somefileI expect another  Broken pipe  message, but reality may differ from my expectations.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: GNUser on October 27, 2023, 03:39:23 PM
Hi jazzbiker. No luck with the trap. But it's not surprising because with the exec statement, the script no longer exists in RAM when /usr/bin/yes executes.

So I also tried your suggestion without exec, like this:
Code: [Select]
#!/bin/sh
trap 'exit 0' SIGPIPE
/usr/bin/yes "$@" 2>/dev/null
But no luck with that, either.

Hi Rich. The superfluous passwords do not cause a Broken pipe message, surprisingly:
Code: [Select]
$ echo -e "tinycore\ntinycore\ntinycore\ntinycore\ntinycore\ntinycore" | bcrypt test.txt
Encryption key:
Again:
$
So the problem does not seem to be due to  yes  not knowing when to stop.

I'm happy with the wrapper script to fix the cosmetic issue when using  tce-load  and Rich's workaround for bcrypt. Thank you all :)
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: jazzbiker on October 27, 2023, 04:02:48 PM
Hi GNUser,

I see that the problem is solved, so maybe if it is still interesting for You and You have a spare minute You may try not use the wrapper for yes but try declaration of the SIGPIPE handler inside tce-load script.

Edit: maybe better to set ignore for the SIGPIPE:
Code: [Select]
trap '' SIGPIPE
Hi Rich. The superfluous passwords do not cause a Broken pipe message, surprisingly:
Code: [Select]
$ echo -e "tinycore\ntinycore\ntinycore\ntinycore\ntinycore\ntinycore" | bcrypt test.txt
Encryption key:
Again:
$
Probably the pipe is buffered and the string is small enough to be placed in the buffer as a whole.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: GNUser on October 27, 2023, 11:30:48 PM
Hi jazzbiker. yes-related "Broken pipe" messages still appear if I add this to /usr/bin/tce-load:
Code: [Select]
trap '' SIGPIPEI have submitted a bug report to the havoc developer.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: GNUser on October 28, 2023, 01:14:02 PM
Hi Juanito. havoc developer says the pipe of interest is created by the shell, not the terminal emulator (see here (https://github.com/ii8/havoc/issues/50)). Moving blame from terminal emulator to shell helps explain why havoc and weston-terminal are both affected.

I use havoc + busybox ash 99.9% of the time. I tried havoc + bash as a test and problem did not go away.

Since all terminal emulators and shells seem to be affected, I think the problem may have something to do with how weston or labwc is being started. Can you suggest a way to start the graphical session different from this (which is what weston.tcz.info suggests):
Code: [Select]
sudo -- seatd -g staff -n /run/seatd.sock & XDG_RUNTIME_DIR=/run/user/$(id -u) weston
?
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: jazzbiker on October 28, 2023, 04:00:50 PM
Hi!

I've made some experiments trying to reproduce the observed by GNUser behaviour of yes. As it was described in the link by patrikg it looks rather counterintuitive.
I'm testing in TC14 x86 under Xorg.

Code: [Select]
tc@box:~$ yes | lua -e 'print(io.read())'
y
tc@box:~$ yes no | lua -e 'print(io.read())'
no
tc@box:~$ trap '' SIGPIPE # ignore signal
tc@box:~$ yes | lua -e 'print(io.read())'
y
yes: Broken pipe
tc@box:~$ yes no | lua -e 'print(io.read())'
no
here the pipeline hangs, interrupted by Ctrl-C
Code: [Select]
^Ctc@box:~$  trap - SIGPIPE # restore default handler
tc@box:~$ yes | lua -e 'print(io.read())'
y
tc@box:~$ yes no | lua -e 'print(io.read())'
no
tc@box:~$

So looks like havoc envokes the shell with SIGPIPE ignored. The cure may be restoring the default handler.

@GNUser, You may try adding
Code: [Select]
trap - SIGPIPEto tce-load or to yes wrapper.

Looks like it is havoc's fail cause it leaves the shell without the SIGPIPE handler. I can't guess why the same behaviour is exhibited by another wayland terminal emulator, looks like coincidence.

You can notice the changes in the yes behaviour when it is supplied with the output string.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: jazzbiker on October 28, 2023, 04:37:24 PM
You can notice the changes in the yes behaviour when it is supplied with the output string.

In fact the difference in the yes behaviour is between single-char and multi-char output:
Code: [Select]
tc@box:~$ trap '' SIGPIPE
tc@box:~$ yes | lua -e 'print(io.read())'
y
yes: Broken pipe
tc@box:~$ yes n | lua -e 'print(io.read())'
n
yes: Broken pipe
tc@box:~$ yes no | lua -e 'print(io.read())'
no
^Ctc@box:~$ # hung up and interrupted by Ctrl-C
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: Rich on October 28, 2023, 04:56:44 PM
Hi jazzbiker
Nice find. Works under TC10 x86 too:
Code: [Select]
tc@E310:~$ busybox yes | head -n 3
y
y
y
tc@E310:~$ trap '' SIGPIPE
tc@E310:~$ busybox yes | head -n 3
y
y
y
yes: Broken pipe
tc@E310:~$ trap - SIGPIPE
tc@E310:~$ busybox yes | head -n 3
y
y
y
tc@E310:~$

... Looks like it is havoc's fail cause it leaves the shell without the SIGPIPE handler. I can't guess why the same behaviour is exhibited by another wayland terminal emulator, looks like coincidence. ...
Maybe they share a common code base. That can happen when
someone forks a program to implement changes they want.

Or maybe it's being caused by something else they share in
common, Wayland itself.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: GNUser on October 28, 2023, 07:10:07 PM
Hi jazzbiker and Rich. Thank you. It's good to have more than one workaround.

But I confess I'm no closer to understanding the root problem.

Looks like it is havoc's fail cause it leaves the shell without the SIGPIPE handler.
I'm not sure it is havoc's fault. The developer tried hard but was unable to reproduce the issue. See here:
https://github.com/ii8/havoc/issues/50#issuecomment-1783859048
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: Rich on October 28, 2023, 08:06:27 PM
Hi GNUser
Quote from https://github.com/ii8/havoc/issues/50#issuecomment-1783907710
Quote
In havoc, ls | sort and all other pipelines work fine. Strangely, the problem seems limited to pipelines containing yes command. ...
Nothing strange about it:
Code: [Select]
yes "password" | bcrypt somefileyes  starts writing  "password"s  to the pipe (FIFO).
bcrypt  reads the first  "password"  and then the second one to confirm.
bcrypt  encrypts the file and then exits.
When  bcrypt  exits, the pipe gets destroyed.
When  yes  attempts the next write, it encounters a write error and terminates.

The way to stop  yes  is to terminate it with Ctrl-C, kill, break the pipe, etc.
The broken pipe is the normal way to terminate  yes , not an error in this case.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: GNUser on October 28, 2023, 08:22:35 PM
Hi Rich. I can follow all of that and it makes perfect sense. What I'm trying to understand is why these "broken pipe" messages only appear in terminal emulators on TCL in wayland session. Here is the situation:

console on TCL: no error
terminal emulator on TCL in X session: no error
terminal emulator on other distro in wayland session (per havoc's developer): no error
terminal emulator on TCL in wayland session: error
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: GNUser on October 28, 2023, 08:24:11 PM
When  bcrypt  exits, the pipe gets destroyed.
When  yes  attempts the next write, it encounters a write error and terminates.
That's not what I'm seeing.  yes  is not exiting and enters an infinite loop after bcrypt gets the two lines it needs, driving CPU usage up and turning on my laptop's fans. Pressing Control+c does not stop the infinite loop. Only way to stop the infinite loop is to close the terminal emulator window:

Code: [Select]
$ echo "this is a test" >test.txt
$ yes "password" | bcrypt test.txt
Encryption key:
Again:
^C^C^C^C^C^C^C^C^C^C^C^C^C^C[laptop fans start running here, I close terminal emulator window, cpu usage goes down, fans stop]
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: Rich on October 28, 2023, 08:30:03 PM
Hi GNUser
Forgot about the infinite loop case. No idea about that.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: Rich on October 28, 2023, 09:13:53 PM
Hi GNUser
I see you added an example to your post.

This is interesting:
Code: [Select]
tc@E310:~$ trap '' SIGPIPE  # Disable handler
tc@E310:~$ echo "this is a test" >test.txt
tc@E310:~$ busybox yes "password" | bcrypt test.txt
Encryption key:
Again:
^Ctc@E310:~$ # busybox yes enters infinite loop, Ctrl-C breaks us out.
tc@E310:~$
tc@E310:~$ echo "this is a test" >test.txt
tc@E310:~$ yes "password" | bcrypt test.txt
Encryption key:
Again:
yes: standard output: Broken pipe
tc@E310:~$ # GNU yes works.
tc@E310:~$
tc@E310:~$ trap - SIGPIPE  # Enable handler
tc@E310:~$ echo "this is a test" >test.txt
tc@E310:~$ busybox yes "password" | bcrypt test.txt
Encryption key:
Again:
tc@E310:~$ # busybox yes works.
tc@E310:~$
tc@E310:~$ echo "this is a test" >test.txt
tc@E310:~$ yes "password" | bcrypt test.txt
Encryption key:
Again:
tc@E310:~$ # GNU yes still works.

This seems to suggest there may be a bug in  busybox yes  when
there is no signal handler for  SIGPIPE.

If you use  GNU yes , does the problem go away?
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: GNUser on October 28, 2023, 10:23:31 PM
Hi Rich.
If you use  GNU yes , does the problem go away?
No.

I think I figured it out. Juanito sees this error in weston-terminal because that terminal ignores SIGPIPE (see here (https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/253)). In my case, I am seeing the error in havoc terminal because the terminal emulator inherits its environment from my compositor, labwc (version 0.6.5), which has been rigged to ignore SIGPIPE since version 0.6.3 (see here (https://github.com/labwc/labwc/blob/master/NEWS.md)).

If  yes  is running in an environment where SIGPIPE has been disabled, it will either error out with "Broken pipe" or will enter an infinite loop, depending on the situation. Either way, it will not behave as expected.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: Rich on October 28, 2023, 10:40:30 PM
Hi GNUser
So just to be clear, installing  coreutils  and executing the
following still results in an endless loop:
Code: [Select]
$ echo "this is a test" >test.txt
$ /usr/local/bin/yes "password" | bcrypt test.txt
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: GNUser on October 28, 2023, 10:44:42 PM
No. In that scenario an error is displayed but no endless loop. It seems only busybox  yes  enters the endless loop when used with  bcrypt  .

Code: [Select]
bruno@x230:~$ tce-load -wil coreutils
coreutils.tcz.dep OK
Downloading: coreutils.tcz
Connecting to gnuser.ddns.net (24.0.182.246:80)
saving to 'coreutils.tcz'
coreutils.tcz        100% |********************************| 2304k  0:00:00 ETA
'coreutils.tcz' saved
coreutils.tcz: OK
yes: Broken pipe
bruno@x230:~$ echo "this is a test" >test.txt
bruno@x230:~$ /usr/local/bin/yes "password" | bcrypt test.txt
Encryption key:
Again:
/usr/local/bin/yes: standard output: Broken pipe
bruno@x230:~$
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: Rich on October 28, 2023, 10:49:28 PM
Hi GNUser
And that's the point I was trying to make. GNU yes  prints
the error message after  bcrypt  completed successfully.
That's why I suggested it may be a  busybox  issue.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: GNUser on October 28, 2023, 11:04:16 PM
Hi Rich. Now I understand you 100%.

The missing piece for me was that we are dealing with an environment where SIGPIPE has has been disabled. My TCL14 instance with Xorg has an intact SIGPIPE and none of these  yes  shenanigans.

So it seems busybox yes sometimes errors-out and sometimes enters an endless loop in an environment where SIGPIPE is disabled. GNU yes seems to consistently error-out and not enter endless loop.

Thread may be marked as solved. Thank you all.
Title: Re: [Solved] in havoc terminal emulator: yes causes broken pipes
Post by: Rich on October 28, 2023, 11:22:48 PM
Hi GNUser
Marked as solved.

By the way,  busybox yes  does not appear to be aliased. So if
you extract the  yes  executable from  coreutils  , copy it to
/usr/local/bin/  and add it to your backup, it might fix it.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: jazzbiker on October 29, 2023, 01:15:22 AM
So it seems busybox yes sometimes errors-out and sometimes enters an endless loop in an environment where SIGPIPE is disabled. GNU yes seems to consistently error-out and not enter endless loop.
Not sometimes. If the string to output is single-byte it exits with error, but if the string to includes more than one byte, yes hangs up.
Aren't You in the mood to send the bug report to busybox?
Title: Re: [Solved] in havoc terminal emulator: yes causes broken pipes
Post by: GNUser on October 29, 2023, 08:11:18 AM
Hi jazzbiker. Yes (pun intended), that is a more precise statement of the problem. Thank you :)
I'm a bit worn out by this at the moment. Maybe I'll find the inspiration later. If you'd like to give the busybox developers a bug report, please go ahead.

The developer of my compositor fixed the issue with this:
https://github.com/labwc/labwc/pull/1210
I have recompiled labwc with this fix and resubmitted it for the repo.
Title: Re: in havoc terminal emulator: yes causes broken pipes
Post by: Juanito on October 29, 2023, 11:01:29 AM
Since all terminal emulators and shells seem to be affected, I think the problem may have something to do with how weston or labwc is being started. Can you suggest a way to start the graphical session different from this (which is what weston.tcz.info suggests):

Weston used to use weston-launch to start, but this is deprecated and still had the error. I believe sway uses seatd-launch, but I haven’t tried it.

Gnome-session uses the mutter wayland compositor and doesn’t exhibit the error in gnome-terminal, but I haven’t tried it with havoc.

Perhaps it’s worth moving the XDG_RUNTIME_DIR statement to /etc/profile and/or .ashrc to see if that helps?
Title: Re: [Solved] in havoc terminal emulator: yes causes broken pipes
Post by: Juanito on October 30, 2023, 07:20:48 AM
havoc does not have the broken pipe error with gnome-session