Umlauts and special characters in icingaweb2

So, I’ve replaced my MySQL 8 container now with a MariaDB 10.3.25-focal (docker) container. Everything else is the same as before. Umlauts also work pretty fine. So I’d rule that out.

Which browser are you using? Have you tried a different one?
The locale output from above, was that from the machine that’s running the temperature check? If not, please check the locale on this machine as well.

Chrome, firefox and old IE show the same behavior.

Check, DB, webinterface and core are all running on the same VM.

Well, then I’m out of ideas. :roll_eyes:

:poop:

I’ll set up a CentOS and Ubuntu VM for testing, when I find the time and report back :slight_smile:
Thanks for your help so far!

For python unicode support see: Unicode HOWTO — Python 3.9.1 documentation
You can look at the code of the plugin. Maybe you only have to add the encoding to an open statement:
open(‘unicode.txt’, encoding=‘utf-8’)

Hmm, everything works good for me
image | image
But it seems I’ve the same setup as @nilmerg

What do you see in API?
I see "__name":"tt\u00e4esting \u00b0C \u00df\u00d7h\u00e4esting \u00b0C \u00df\u00d7h" for host called täesting °C ß×h which in database is stored as täesting °C ß×h

My settings

Stored in database

select comment_data from icinga_comments where comment_id = 120;  
+------------------------------------------------+
| comment_data                                   |
+------------------------------------------------+
| Umlaut ä
some special chars ¤×°C          |
+------------------------------------------------+

MariaDB

Server version: 10.3.27-MariaDB-0+deb10u1 Debian 10

locale

LANG=en_US.UTF-8
LANGUAGE=en_US:en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

database charsets

SELECT default_character_set_name FROM information_schema.SCHEMATA WHERE schema_name = "icinga2";
| default_character_set_name |
+----------------------------+
| utf8mb4                    |

MariaDB [(none)]> SELECT CCSA.character_set_name FROM information_schema.`TABLES` T, information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA WHERE CCSA.collation_name = T.table_collation  AND T.table_schema = "icinga2" AND T.table_name = "icinga_comments";
| character_set_name |
+--------------------+
| utf8mb4            |


MariaDB [(none)]> SELECT character_set_name FROM information_schema.`COLUMNS`  WHERE table_schema = "icinga2" AND table_name = "icinga_comments"  AND column_name = "comment_data";
| character_set_name |
+--------------------+
| latin1             |

ido settings

[icinga_ido]
type = "db"
db = "mysql"
host = "localhost"
dbname = "icinga2"
username = "***"
password = "***"
charset = "latin1"
use_ssl = "0"

If I change charset = "utf8" the symbols are displayes wrongly

Just set up icinga on a ubuntu 20.04 test system again. All done with the ansible stuff as well.
No problems here, umlauts are displayed correctly.
The DB shows them not correctly though:
image

char_set is latin1

MariaDB [(none)]> SELECT default_character_set_name FROM information_schema.SCHEMATA WHERE schema_name = "icinga2";
+----------------------------+
| default_character_set_name |
+----------------------------+
| latin1                     |
+----------------------------+
1 row in set (0.001 sec)


MariaDB [(none)]> SELECT CCSA.character_set_name FROM information_schema.`TABLES` T, information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA WHERE CCSA.collation_name = T.table_collation  AND T.table_schema = "icinga2" AND T.table_name = "icinga_comments";
+--------------------+
| character_set_name |
+--------------------+
| latin1             |
+--------------------+
1 row in set (0.002 sec)

MariaDB [(none)]> SELECT character_set_name FROM information_schema.`COLUMNS`  WHERE table_schema = "icinga2" AND table_name = "icinga_comments"  AND column_name = "comment_data";
+--------------------+
| character_set_name |
+--------------------+
| latin1             |
+--------------------+
1 row in set (0.001 sec)

will keep this in mind for future setups, but I don’t know why/how this is happening.
Maybe on the problematic VM something went wrong during the icinga setup or something is wrong with the VM setup itself (was not setup be me, so I don’t know the exact settings)

1 Like

Glad to hear!

That’s to be expected. The comment is now with utf8 encoded, but the database tries to interpret it as latin1. Try SET NAMES latin1 before you run your query. That’s the same what Icinga Web 2 does and why it’s necessary to set the resource’s charset to latin1. This way mysql won’t try to transcode the comment and you’ll get the raw (correct) value.

2 Likes

@nilmerg sorry for pulling this up again :sweat_smile:

I just experienced a strange behavior.
System on Ubuntu 20.04.2.

when having “latin1” in charset option of resource:
image

when removing “latin1” for charset option
image

As I am using Graphite I need to set “latin1” in the charset option, otherwise the graphs are not displayed.

Setup was create DBs with default MariaDB settings and them schema import.

MariaDB [(none)]> SELECT default_character_set_name FROM information_schema.SCHEMATA WHERE schema_name = "icinga2";
+----------------------------+
| default_character_set_name |
+----------------------------+
| utf8mb4                    |
+----------------------------+
1 row in set (0.001 sec)

MariaDB [(none)]> SELECT CCSA.character_set_name FROM information_schema.`TABLES` T, information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA WHERE CCSA.collation_name = T.table_collation  AND T.table_schema = "icinga2" AND T.table_name = "icinga_comments";
+--------------------+
| character_set_name |
+--------------------+
| utf8mb4            |
+--------------------+
1 row in set (0.001 sec)

MariaDB [(none)]> SELECT character_set_name FROM information_schema.`COLUMNS`  WHERE table_schema = "icinga2" AND table_name = "icinga_comments"  AND column_name = "comment_data";
+--------------------+
| character_set_name |
+--------------------+
| latin1             |
+--------------------+
1 row in set (0.001 sec)

Can you make any sense of that?

Not really. But as we experienced it before, Icinga 2 just accepts any and all encodings and puts them straight into the database no matter its charset. Icinga Web 2 just reads it as utf-8. If the real encoding is a subset of utf-8, all is fine albeit it not being utf-8. If it overlaps with another encoding not compatible with utf-8 (e.g. latin1 (aka ISO-8859-1, cp1252)) … yes, really, latin1, as mentioned previously, that config option only prevents transcoding … it will fail badly. Or may produce weird results because a code point of such an encoding is indeed part of utf-8, but will represent an entirely different glyph. (char)

So, with that said, make sure your check plugins all accept utf-8 (input) and put out utf-8 (output).

1 Like

Thanks for the quick feedback.

Seems like I was to fixed on the “latin1”.
Putting utf8 into the charset option solves both problems:

  • the umlauts and special characters are displayed
  • the graphite graphs are displayed