Tiny Core Linux

Tiny Core Extensions => TCE Q&A Forum => Topic started by: emmi on November 25, 2019, 02:15:27 PM

Title: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: emmi on November 25, 2019, 02:15:27 PM
I ran an gtk2 application on TinyCore 5.4 and everything worked as expected, including german umlauts, within the window, the input fields and, and ... I updated to version 10.1 and now all titles in a popup window shows ? instead of the expected umlaut.  All the other occurrences of umlauts seem to be correct. (I also had version 6.4.1 available, which shows the same wrong titles.) Obviously something changed. But what and how can I fix it or work around it?

I only install (zip-unzip and) gtk2, nothing else, no locale, no font, no nothing.

Installing gtk2 is different, there seem to be different dependencies which trigger different extensions to be installed and a different order to install the common ones. But I failed to figure out, if and which one/s is/are missing in the10.1 (or 6.4.1) extensions.

I hope someone with more insight into gtk2 can help.

Thanks.
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: Rich on November 25, 2019, 06:19:37 PM
Hi emmi
...  (I also had version 6.4.1 available, which shows the same wrong titles.) Obviously something changed. ...
Starting with version 6  shared-mime-info.tcz  was removed from the dependency file. In addition, the following files no longer
showed up in the  gtk2.tcz.tree  file:
Code: [Select]
libFS.tcz
libICE.tcz
libSM.tcz
libXaw.tcz
libXfont.tcz
libXmu.tcz
libXpm.tcz
libXres.tcz
libXss.tcz
libXt.tcz
libXtst.tcz
libXv.tcz
libXvmc.tcz
libXxf86dga.tcz
libdmx.tcz
libfontenc.tcz
libxkbfile.tcz
libxml2.tcz
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: curaga on November 25, 2019, 11:35:19 PM
Did the WM change? The WM is responsible for drawing the title bars.
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: GNUser on November 26, 2019, 07:33:49 AM
emmi, I was not able to reproduce your issue: I booted my Pure64 10.1 box with a minimal onboot.lst (only Xorg-7.7, my intel graphics extension, and thunderbird-appimage.tcz as a representative gtk2 application) and umlauts show up fine in all of thunderbird's boxes and title bars. So in my experience they show up fine even without loading mylocale.tcz and without loading any fonts. My WM is fluxbox.

The only places where I find I need special care is needed to display Unicode characters is in my web browser (iridium-browser.tcz, which needs a Unicode font--I use dejavu-fonts-ttf.tcz) and in my terminal emulator (rxvt.tcz, which needs a UTF-8 locale). aterm does not support Unicode (see here (https://wiki.netbsd.org/tutorials/unicode/#index11h2)).

The extension getlocale.tcz walks the user through creating mylocale.tcz extension. You'll need a boot code that matches the locale in mylocale.tcz (for example, if mylocale.tcz contains en_US.UTF-8, then you need the boot code lang=en_US.UTF-8).
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: emmi on November 26, 2019, 02:03:29 PM
Thanks for getting back to  me on this. I will try to install the removed extensions. Some of them seem to be installed anyway, maybe as an indirect dependency.

As I said, the error is reproducible at will: just booting the iso image, adding gtk2 and running the app. I can make screenshots. I could patch the app and replace the utf-8 ö, two bytes, with oe and that correctly showed. But I don't want to do that for all these strings.

FWIW, in the app, near the title string, I saw the string "dialog". Knowing  nothing about gtk, I said popup window.

PS: Yes, I entered this in the wrong forum but didn't know how to move it myself.
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: emmi on December 04, 2019, 01:33:30 PM
I installed all the removed tces as listed here http://forum.tinycorelinux.net/index.php/topic,23351.msg146250.html#msg146250 (http://forum.tinycorelinux.net/index.php/topic,23351.msg146250.html#msg146250) but that didn't change anything. As can be seen from the appended screenshot, the title bar of an aterm correctly shows the umlaut, the text in the window and boxes show the umlauts, but the dialog/popup box has it wrong. I suspect there is something broken, very likely in gtk2.
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: GNUser on December 05, 2019, 07:54:54 AM
Your WM (window manager) can properly display umlauts in titlebars, as your screenshot shows.

GTK2 is only involved here if it's the toolkit that your WM uses. FWIW, GTK2 supports UTF-8 out of the box (see https://wiki.gentoo.org/wiki/UTF-8#KDE.2C_GNOME.2C_and_Xfce).

What WM are you using? It seems that your WM's graphical toolkit and this specific application are butting heads, probably over encoding--i.e., the application used one encoding to store the characters, but WM's toolkit is assuming a different encoding. The two encodings agree on how to represent basic ASCII characters (I think most encodings agree on that), but not the expanded Latin characters. That's my theory, anyway.

If you specify a locale (see my prior post) does it make the problem go away?

Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: emmi on December 05, 2019, 03:51:10 PM
Yes, for aterm the WM displays the UTF-8 encoded umlaut in the title without any problem.

And, as mentioned before, the application worked without any problem on tcl 5.4, I only installed GTK2, nothing else.

The WM is whatever comes with tcl. Because GTK2 is not installed by default, the WM doesn't use it. As mentioned before, there is UTF-8 in the application, the shown message is hard coded. If I change the encoded umlaut to two ASCII characters, they are shown. For the aterm example I used the encoded umlaut from the application, modified a script with hexedit.

I installed getlocale, selected de_DE.UTF-8, loaded mylocale, nothing changed. Again, locale was not necessary on tcl 5.4.
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: GNUser on December 06, 2019, 03:59:30 AM
emmi - sorry to hear using the locale didn't help.

It sounds like you are familiar with the application's source code. My last thought is to check the encoding of the file that contains the strings used in title bar. If it is something other than UTF-8, change the encoding to UTF-8 then recompile your application.

Take a look here: https://www.shellhacks.com/linux-check-change-file-encoding/

P.S. The file utility is in file.tcz, iconv utility is in glibc_apps.tcz
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: emmi on December 06, 2019, 02:32:10 PM
I'm not a GTK2 programmer but managed to copy an example from the net and adjust it for demonstration purpose. The program was compiled on a different Linux system. The binary, the object, the source and how it was built is in the appended ZIP file. The binary works as expected on 5.4, just install GTK2; it does not work on 6.4.1 (and newer versions) with GTK2 and locale installed. It uses the gtk_window_set_title() with an UTF-8 encoded umlaut.

If someone wants to try it...
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: Rich on December 06, 2019, 05:36:34 PM
Hi emmi
Attaching binaries to posts is against forum rules. See item 2 here:
http://forum.tinycorelinux.net/index.php/topic,7738.0.html
I repackaged your attachment to include only the source file and build script.
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: emmi on December 08, 2019, 05:15:53 AM
OK. The intention was to provide a environment similar to the one I have with the real application, which is only available as a binary. I don't know if compiling on tcl will make a difference. I hope it  will not and I hope someone can identify the problem with this small reproducer, fix it or give a workaround.
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: Rich on December 08, 2019, 06:41:03 AM
Hi emmi
I understand. If someone prefers to try the binary without compiling it themselves, they are free to say so. You can email them a
copy or upload it to a file hosting site and PM them a download link.
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: GNUser on December 09, 2019, 05:40:59 AM
I think I've found a different symptom of the same problem.

In all GTK2 and GTK3 applications, I get an error ("Invalid file name") in file selection box whenever I include any Unicode character in the filename. Here's an example when using a GTK3 application:

(http://files.dantas.airpost.net/public/save_file.jpg)

I'm using a Unicode locale:
Code: [Select]
bruno@box:~$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

In Devuan, the same GTK2 and GTK3 applications accept Unicode characters in file names.

In TCL, I do not have this problem with applications that use a different graphical toolkit (e.g., xfe file manager uses fox toolkit; xfe can create new files and directories with Unicode characters in their names).

Maybe current GTK2 and GTK3 versions in Pure64 repository were not compiled with full Unicode support?
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: GNUser on December 09, 2019, 09:49:32 AM
@Rich - I'm not sure whether the problem in my prior post is related to OP, so I'll open a new thread. Feel free to delete my post above.

If I find a fix for my issue, I'll share it in this thread so that emmi can try it. I think that would be the easiest way to know whether his issue and mine are related.
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: emmi on January 04, 2020, 12:26:19 PM
It took me a while to do some more testing with my x application (binaries removed from this forum) and different window managers:
As mentioned before, the "old" flwm_topside in TinyCore 5.4 works as well.

To me this looks like a regression in flwm_topside.
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: emmi on February 01, 2020, 11:58:28 AM
Some updates ...

I found the sources for flwm at https://github.com/tinycorelinux/flwm.git, but I couldn't find any relationship between the tcl versions and the commits in the repository. I found the current sources for fltk, but none for 5.x.

I generated flwm and flwm_topside from the sources and ran them. As before, my demo application does not work with flwm_topside but "works" with flwm. As before aterm with an utf-8 encoded title works with flwm_topside, however, I did't try before, it doesn't work with flwm.

So I added some fprintf's showing the to be printed title. For both, flwm_topside and flwm, it showed the ISO-LATIN1 character 'ü' for my demo application and the utf-8 encoded 'ü' for the aterm.

That is, flwm_topside prints utf-8 encoded titles and flwm doesn't. And vice versa, flwm prints ISO-LATIN1 characters and flwn_topside doesn't.

Obviously there are different ways how the title set in aterm and the title set in my gdk demo application makes it to the windows manager, flwm. I didn't look at aterm. Gtk sets the title in X11 atoms, XA_WM_NAME and _NET_WM_NAME. The latter is set to the utf-8 encoded title, the first to a "compound_text" type of text, which here is a ISO-LATIN1 encoded text.

Flwm only retrieves the XA_WM_NAME atom. This explains why I see the ISO-LATIN1 character 'ü' in my debug output. This also means that aterm sets the utf-8 title in XA_WM_NAME. (I assume, it sets whatever it finds in the -title option.)

Retrieving the _NET_WM_NAME atom is not possible, as it is not supported by the default X11 server, Xvesa. (Which to me also means, that gtk fails to set it, but that doesn't matter, here.)

Because my demo application works on 5.x of tcl, it is very likely that flwm_topside for that version printed the ISO-LATIN1 title. As it
no longer does, it is either a regression in flwm_topside or in fltk's fl_draw(), which is used, here - at least in the sources I have. Without having the sources of flwm and fltk for 5.x it is hard to tell where the regression is.

In the current version, for the title, flwm (left side) uses  draw_rotated90(), defined in Rotated.C. (fltk is configured with --disable-xft, so there is no HAVE_XFT defined when compiling flwm.)

So I changed the sources for flwm_topside to use similar code as in draw_rotated90() that is finally calling draw_rotated() with angle 0 for the title. Now flwm_topside behaves as flwm, which is an acceptable behavior for me (and aterm with an utf-8 title no longer
works, which is OK for me, too).

It seems the default X11 server and flwm do not really support utf-8.
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: curaga on February 02, 2020, 01:04:58 AM
Just noting, the X server does not restrict atoms. You can create arbitrary-named atoms just fine.
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: Rich on February 02, 2020, 07:09:53 AM
Hi emmi
... I found the sources for flwm at https://github.com/tinycorelinux/flwm.git, but I couldn't find any relationship between the tcl versions and the commits in the repository. I found the current sources for fltk, but none for 5.x. ...
Attached is a source package from an earlier time frame. I converted it from  .tgz  to  .tar.bz2  so it would be small enough that I
could attach it. The files in  flwm-1.14.tar.bz2  were last modified in 2010 and are likely from TC2 and TC3.

If you look here:
http://tinycorelinux.net/4.x/x86/release/src/
you'll find  flwm-1.14.tgz  and  flwm_topside-1.14.tgz.  These packages contain files with 2012 modification dates and were likely
used for TC5 as well as TC4.

Ignore the  1.14  version number on all of the packages I mentioned. The 3 packages all have differences.
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: emmi on February 02, 2020, 11:55:47 AM
Just noting, the X server does not restrict atoms. You can create arbitrary-named atoms just fine.

Not being an X11 programmer and _NET_WM_NAME not being in /X11/Xatom.h I  incorrectly concluded, that it isn't supported. From what you say, gtk can set this atom and in deed it does. I tried XGetAtoms() for_NET_WM_NAME and it returned an atom, which I then used with getProperty(), which in turn got me the utf-8 title.

Dunno if this is the right way to do this, but it enables me to write a fix and not just a hack or workaround.
Title: Re: gtk2 and german umlauts (utf-8) in the title of a popup window
Post by: emmi on February 02, 2020, 01:26:32 PM
This works for me, but a more experienced X11 programmer should review it.

Code: [Select]
diff --git a/Frame.C b/Frame.C
index a52d314..383ea51 100644
--- a/Frame.C
+++ b/Frame.C
@@ -508,6 +508,15 @@ Frame::~Frame() {
 void Frame::getLabel(int del) {
   char* old = (char*)label();
   char* nu = del ? 0 : (char*)getProperty(XA_WM_NAME);
+// fl_draw() works with utf-8, draw_rotated90() does not
+#if defined TOPSIDE || defined HAVE_XFT
+  Atom a = XInternAtom(fl_display, "_NET_WM_NAME", True);
+  if (a != None && a != BadValue && a != BadAlloc) {
+    char *utf8 = del ? 0 : (char*)getProperty(a);
+    if (utf8)
+      nu = utf8;
+  }
+#endif
   if (nu) {
     // since many window managers print a default label when none is
     // given, many programs send spaces to make a blank label.  Detect