Icinga 2.16 upgrade: sudden flood of warnings ("Service object name too long" / ctrl characters) on config that was clean in 2.15

Icinga 2
1

Hello,

Since upgrading our Icinga servers from 2.15 to 2.16.0-1, we are seeing a strong increase of config warnings during validation, while the exact same configuration was working fine before on 2.15.

The warnings look like this:

warning/config: Object name of type 'Service' is too long (length: 315, max: 255). 
warning/config: Object name of type 'Service' contains ctrl or newline characters, which may cause problems in some contexts.

They are triggered from an apply Service for (...) rule:

apply Service for (port in host.vars.access_ports) to Host {

Example hosts impacted:

Important points:

  • Hostnames themselves are normal length and do not contain special characters.

  • This happens on many/all hosts using this rule.

  • On Icinga 2.15, same config produced no such warnings.

  • The issue only appeared after upgrading the Icinga servers.

Our suspicion is that one of the generated values inside host.vars.access_ports may now be interpreted differently in 2.16, or that validation became stricter.

This is causing operational issues for us because we use an internal tool that generates and deploys Icinga config automatically. It runs the validation command remotely through Paramiko, and because of the huge amount of warnings, command execution becomes very long and eventually times out while waiting for the full output.

So our questions are:

  1. Did something change in Icinga 2.16 regarding object name generation for apply for rules?

  2. Were checks on control/newline characters tightened recently?

  3. Is there a recommended way to sanitize generated service names coming from arrays/dictionaries in apply for loops?

  4. Could this be considered a regression if the exact same config was accepted cleanly in 2.15?

Any guidance would be appreciated.

Thanks.

Nathan

2

The database schema only allows object names of up to 255 characters. Prior to Icinga 2 version 2.16.0, this would result in errors further down the pipeline, either in Icinga DB or in the deprecated IDO. The only change in Icinga 2 was the introduction of the warning, as such long object names have always resulted in potential errors.

Btw, the errors are about the service names, as the log shows and not for your short hosts.

More details are available in Icingadb crashes on too long service name · Issue #1101 · Icinga/icingadb · GitHub and Warn on problematic object names · Issue #10768 · Icinga/icinga2 · GitHub.

3

Thank you for your reply, that makes sense regarding the 255-character database limit and the fact that the warning was only introduced in Icinga 2.16.

However, in our case, based on our configuration, we do not understand how the generated service name could exceed 255 characters.

Our apply rule is:

apply Service for (port in host.vars.access_ports) to Host {
  import if (port.service_template) {port.service_template} else {"service-5m"}
  name = port.name
  display_name = if(port.display_name) {port.display_name} else {"Port "+port.real_name}
  check_command = "check_switch_port"
  vars.port_name = port.real_name
  vars.ifindex = if(port.ifindex) {port.ifindex} else {port.real_name}
  vars.check_state = if (port.monitoring) {"yes"} else {"no"}
  vars.relative_state = if (port.absolute_state) {"no"} else {"yes"}
  vars.warning = if (port.warning) {port.warning} else {"85,-1,-1"}
  vars.critical = if (port.critical) {port.critical} else {"95,-1,-1"}
  vars.service_groups = if (port.service_groups) {port.service_groups} else {[]}
  vars.notifications = if (port.notifications) {port.notifications} else {[]}
  vars.type = if (port.type) {port.type} else {""}
}

And port.name values are short strings such as:

port_gigabitethernet1_0_1
port_gigabitethernet1_0_10
port_gigabitethernet1_0_11

So if I understand correctly, the generated service object name should be something like:

acc-fo-gran-01.net.test.net!port_gigabitethernet1_0_1

Character count:

  • acc-fo-gran-01.net.test.net = 27

  • ! = 1

  • port_gigabitethernet1_0_1 = 24

Total = 52 characters

Even with longer interface names such as:

acc-fo-gran-01.net.test.net!port_tengigabitethernet1_1_1

This would still remain far below 255 characters.

We also initially suspected that some hidden or non-printable characters could be present in the generated data (for example in host.vars.access_ports or related fields), which could potentially influence how object names are evaluated. However, even after validating and sanitizing this part, the warnings still occur.

That is why we are struggling to understand where lengths such as 315 or 323 could come from, or what exactly is being concatenated internally to reach those values.

Is there any change in Icinga 2.16 regarding how object names are constructed or validated for apply for rules that could explain this behavior?

Thank you :slight_smile:

4

Please feel free to share the complete (but redacted) log output. This might help.

Furthermore, you can add a log DSL function call to the Service object, helping identifying the length. For your previously pasted Service snippet, just add the following line somewhere within the object body:

log(host.name + "!" + name + "\t" + len(host.name + name))

Afterwards, fire an icinga2 daemon -C --dump-objects and scroll through the output.

As another idea, use the Icinga 2 API to query all service objects. Take the following as an inspiration, where $user is an Icinga 2 ApiUser with enough permissions to query all objects and $pass its password. Please check the output.

curl -k -s -S \
    -u "$user:$pass" \
    -H 'Accept: application/json' \
    'https://localhost:5665/v1/objects/services/' \
  | jq -r '.results[].name | [length, @text] | @tsv' \
  | sort -n -k 1 -r \
  | head
5

Maybe verify the actual service names. You can do this pretty easy in the icinga2 console:

icinga2 console --connect https://root:password@localhost:5665/

get_services(get_host(NodeName)).map(s => s.name)
[ "1", "disk", "disk /", "http", "icinga", "load", "ping4", "ping6", "procs", "ssh", "swap", "users" ]
6

That’s not quite correct! The length restriction doesn’t apply on the host.name!service.name part but on each of them separately. The name = port.name statement won’t be executed at the time when Icinga 2 performs such kind of name validations. That assignment is executed much later, when the object body actually gets evaluated. What is going on with your apply rule is as follows: First, looking at your apply rule and its usages, I assume some of your hosts have access_ports custom vars defined as follows:

vars.access_ports = [
    {
      name = "port_gigabitethernet1_0_1"
      real_name = "..."
      ...
    },
    {
      name = "port_gigabitethernet1_0_10"
      real_name = "..."
       ...
    },
    {
      name = "port_gigabitethernet1_0_11"
      real_name = "..."
      ...
    }
]

So, when Icinga 2 evaluates the apply Service for (port in host.vars.access_ports) statement, the port variable is assigned the value of each of the elements of the access_ports array, which are dictionaries (and may contain much more attributes in your case than the ones shown above). Icinga 2 then use port as a unique identifier for the service instances being created, and thus performs name validations on the port variable, which is a dictionary, and not on the name attribute of the dictionary as you might have expected. In this case, the port dictionary converted to a string will look something like this (exactly how it’s defined):

{
    name = "port_gigabitethernet1_0_11"
    real_name = "..."
    ...
}

So, if that dictionary has a lot of attributes, the resulting string will be very long, and thus the name validation will fail with the error message you are seeing. The warnings about the ctrl/newline characters are also related to the fact that the string representation of the port dictionary is prettified with newlines and indentation, which is not really suitable for a service name. See example of the log output below: (EDIT: you won’t see the faulty object name in your case since Icinga 2 doesn’t include it in the logs as it can be huge and possibly mess up the logs).

warning/config: Object name '{
    name = "port_gigabitethernet1_0_11"
    real_name = "..."
    ...
}' of type 'Service' contains ctrl or newline characters, which may cause problems in some contexts.
Location: in icinga2.conf: 70:1-70:50
icinga2.conf(70): apply Service for (port in host.vars.access_ports) {
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To fix this issue, you should change the type of the host.vars.access_ports variable to be a dictionary instead of an array like this:

vars.access_ports = {
    "port_gigabitethernet1_0_1" = {
      real_name = "..."
      ...
    },
    "port_gigabitethernet1_0_10" = {
      real_name = "..."
       ...
    },
    "port_gigabitethernet1_0_11" = {
      real_name = "..."
      ...
    }
}

Then, change your apply rule to a dictionary based iteration instead of the array elements like this:

apply Service for (name => port in host.vars.access_ports) {
    ...
}

You can then use the port variable to access the other attributes just like you were doing before, but now the name variable will contain the unique identifier for the service instance, so you don’t need to assign it manually via the name = port.name statement anymore. That should fix all your warnings!