Quad A (AAAA) DNS requests

I am trying to get icinga2 to stop making AAAA dns requests and have been doing a bunch of adjusting to eliminate this.
I am 90% icinga2 is the cause, since stopping the service stops all the requests in the live tcpdump.

my first fix was to change the generic host template

template Host "generic-host" {
  max_check_attempts = 3
  check_interval = 1m
  retry_interval = 30s

  check_command = "hostalive4"
  vars.slack_notifications = "enabled"
}

so it is now using hostalive4.
all my ping checks are ping4

nagios    105840  105505  0 12:14 ?        00:00:00 /usr/lib/nagios/plugins/check_ping -4 -H host1.internal -c 200,15% -w 130,5%
nagios    105841  105840  0 12:14 ?        00:00:00 /bin/ping -4 -n -U -w 10 -c 5 host1.internal

Does anyone know where else I may need to adjust my configuration to prevent AAAA dns requests?
if there some other check that is causing these requests?

  • Version used (version: r2.13.2-1)
  • Operating System and version : ubuntu 20.04
  • Enabled features api checker ido-mysql influxdb mainlog notification
  • Icinga Web 2 version and modules (System - About)
  • Config validation : valid
  • If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes

Hi,

in general you could try to use $host.address$ instead of $host.name$ or something else in your checks to use the IP instead of hostname/dns. Depending on the checks there are specific vars for this. e.g. the check hostalive uses the var ping_address. But if you look into the the docs Icinga Template Library - Icinga 2, the standard for this would be already $address$.
Instead of using hostalive as check command, you could try icmp (Icinga Template Library - Icinga 2). This check is much faster. Anyway this check works for us.
If this doesn’t help, maybe there is another reason why the check is not using the IP.

1 Like

@stevie-sy Thank you for the suggestion and if nothing else works, i may switch to having the checks use IP. But I would prefer that they continue using fqdn just not make “AAAA” dns requests, they should be only making “A” DNS request (ipv4), not AAAA (ipv6)

https://support.dnsimple.com/articles/aaaa-record/

The AAAA record is conceptually similar to the A record, but it allows you to specify the IPv6 address of the server, rather than the IPv4.

I had though my first idea, the fact the generic-host was using hostalive was the culprit because

Check command object for the check_ping plugin with host check default values. This variant uses the host’s address attribute if available and falls back to using the address6 attribute if the address attribute is not set.

The hostalive command will fall back to ipv6 (even though address6 is not set for any host). And to add more credence to that idea i could see ping commands being run that did not have the preceding -4 (force ipv4). After testing via both versions of the check_ping command (with -4 and without), and using a tcpdump to watch the traffic, running check_ping without the -4 would immediately cause AAAA dns requests. On the other hand check_ping with the -4 would not.
So all of this lead me to swap the generic host to hostalive4. but sadly it did not seem to eliminate the AAAA requests. Something else somewhere (another check? some internal icinga2 mechanism?) is still making ipv6 dns requests. Oh and disabling ipv6 entirely on the host, doesn’t fix the issue.

The point is, that icinga doesn’t directly do this. Icinga is only calling a installed check plugin. You’re installing it on your own. Depending on the OS “nagios-plugins” or “monitoring-plugin”. So maybe you’re create a feature request/issue there?
In other words Icinga has no influence to the check plugin your installed. Icinga only trigger it with the passed params you’re configure in your service check.

1 Like

ok, so that confirms that it is not icinga2 or some core module, but the issue is in some check plugin. So I just need to figure out which one. how exactly I go about doing that, I am not 100% but iI will keep digging.

I wouldn’t even say the AAAA queries are coming from any specific check (except where it is explicitly performing queries, e.g. check_dns). It is likely the plugin simply calling something that eventually needs to call getaddrinfo or similar function to translate a name to an address (which the plugin may not even be doing explicitly either).

There may be some approaches to achieve what you say you want, but it would likely help to know what issues you are trying to solve or avoid by doing so to propose some appropriate options.

I am trying to eliminate all AAAA dns queries. nothing should be calling AAAA since I don’t use ipv6 in the network. There are just extraneous network traffic and dns server load.

  1. I disable ivp6 on the host completely via ubuntu grub2 config options “ipv6.disable=1”
  2. I changed all hostalive configurations to use hostalive4 (which should avoid ipv6)
  3. I verified that all my ping calls go to ping4 and have ping6 commented out in the icinga2 configuration.

This may help: How to Disable IPv6 on Ubuntu Linux I would focus on your OS and not plugins or Icinga2, since it all filters down to there anyway.

I’m still not clear why you want to disable ipv6. If it’s simply performance, there are ways around that without disabling the entire stack. It’s generally not recommended except for specific cases and I haven’t seen a specific case made here. Just my two cents.

Edit: Fixed I would focus on the OS. That is largely where the magic happens.

As I already indicated i have completely disabled ipv6 at the os level via the grub2 config option “ipv6.disable=1”, I did that for testing reasons, in the hopes that it would stop all the spurious AAAA dns requests. disabling ivp6 had zero (0) impact on the AAAA, they continue unabated.

The reason for getting rid of the AAAA, is that they are noise on the network and they are adding load to the DNS server for no reasons, since we don’t use AAAA anywhere. Some icinga2 plugin is causing tons of AAAA requests that serve no purpose and simply add network traffic.

At this point, after some thinking, my suspicions is the “ssh” plugin/check. This is because the AAAA are on the nms master, so it has to be a check executed on master, and there are not that many that execute on master. Most of my check execute on the agents.

I don’t see anything in the thread so far which tells us which hostname is
being looked up for these AAAA requests. I think that should help to narrow
down what might possibly be doing the lookups.

Antony.

The quad AAA requests are leaving the nms master for, as far as I can tell, every host configured in icinga2. So as an example host1.internal (see the original post for a ping request to the same host),

note: the hostnames have been changed to protect the innocent :slight_smile:

I have managed to reduce the volume by significant amount by switching both the ssh and http checks to v4 only

apply Service "ssh" {
  import "generic-service"

  check_command = "ssh"
  vars.ssh_ipv4 = true
  vars.sla = "24x7"

  assign where (host.address || host.address6) && host.vars.os == "Linux"
}
apply Service for (http_vhost => config in host.vars.http_vhosts) {
  import "generic-service"

  check_command = "http"
  vars.http_ipv4 = true

  vars += config
}

the remaining AAAA traffic is minimal, so I deem this issue resolved.