How to increase service return text + metrics max string length

Icinga 2
1

Hi,

we run icinga2 r2.10.2-1. I found that services can only return a string of max 1024 chars (including the return text + performance metrics). Everything after the 1024’s char gets ignored.

I want to return a lot of metrics from a single service and now I don’t get all of them in icinga2 and the graphs.

How to increase that length limit?

Thx.
Robert

2

Hello @robert.ortel and thank you for reporting!

Please provide instructions how to reproduce this on v2.12.0.

Best,
AK

3

Hello,
I admit i’m quite sceptic about it, i can sometime get several megabytes of data output for checks without problems.

Where do you observe this limitation ? on api results or icingaweb dashboards ?
Is this a particular check ? which one ? (there may be an output limitation bound to it, it’s oftenly the case for nrpe check for example)

1 Like
4

Hi,

I cannot easily upgrade or check that on 2.12.0. If you can tell me that a higher icinga2 version has already fixed it … would be a good outcome for me.

This is an example output of my service:

[root@solv248 ~]# /usr/lib64/nagios/plugins/contrib/check_mssql_data_size.sh -s sqlprod02 -u wxyz -p abcdefg
OK: Total data size: 74245 MB. See details in metrics.| ADFSARTIFACTSTORE=4.19 ADFSCONFIGURATIONV3=10.19 ADSYNC=984.00 DEV_ABLAUFVERFOLGUNG=856.31 DEV_ASPSTATE=4.19 DEV_DRK_CRM_API_AUTHORISIERUNG=707.69 DEV_DRK_CRM_API_BEARBEITER=4.19 DEV_DRK_CRM_API_BERATUNGSDOKUMENTATION=156.63 DEV_DRK_CRM_API_CHANGELOG=4.19 DEV_DRK_CRM_API_CONTACTDATA=244.13 DEV_DRK_CRM_API_EWS=7.19 DEV_DRK_CRM_API_IDENTITY=5.00 DEV_DRK_CRM_API_KEX=5.19 DEV_DRK_CRM_API_PERSON=45.44 DEV_DRK_CRM_API_TEMPLATE=4.19 DEV_DRK_CRM_API_TESTPERSON=4.19 DEV_DRK_CRM_API_UEBERLEITUNG=9.00 DEV_DRK_CRM_API_USER=4.19 DEV_DRK_CRM_CRON=23.63 DEV_DRK_LARS_API_AUTH=4.19 DEV_UPDATECRM=4497.19 DEV_UPDATECRM_DESIGNER=78.13 DEV_UPDATECRM_TFS=5.00 DRK_CRM_API_CHANGELOG_HIST=4.19 DWDATA=14054.13 DWLOGGING=137.25 DWNOTIFICATION=3.25 DWSYSTEM=277.25 DWTHUMBNAIL=1126.25 DWWORKFLOWENGINE=2705.00 EINFUEHRUNG=280.94 HPS=258.19 HP-SERVICE=15.19 JOBROUTER_PROD=27270.00 KOSTENDB=11.50 MASTER=5.38 MODEL=8.00 MSDB=2321.38 MYWOWI_RELION=4472.00 PSRCONFIGDB=229.88 PWSDRKPK01=72.00 REPORTSERVER=8.00 REPORTSERVERTEMPDB=8.00 STAG_DRK_CRM_API_AUTHORISIERUNG=83.69 STAG_DRK_CRM_API_BEARBEITER=4.19 STAG_DRK_CRM_API_BERATUNGSDOKUMENTATION=20.81 STAG_DRK_CRM_API_CHANGELOG=4.19 STAG_DRK_CRM_API_CONTACTDATA=118.19 STAG_DRK_CRM_API_EWS=26.13 STAG_DRK_CRM_API_IDENTITY=5.00 STAG_DRK_CRM_API_KEX=4.19 STAG_DRK_CRM_API_PERSON=157.88 STAG_DRK_CRM_API_TEMPLATE=4.19 STAG_DRK_CRM_API_UEBERLEITUNG=5.00 STAG_DRK_CRM_API_USER=4.19 STAG_DRK_CRM_CRON=25.94 STAG_UPDATECRM=12740.75 STAG_UPDATECRM_DESIGNER=72.56 TEMPDB=32.00 TOTAL_DATA_SIZE=74245
[root@solv248 ~]#

From the metrics I can see in on the icinga website there are only displayed 41 of them, even though their count is 60. Additionally there is one wrongly captured metric. On the website there is PWSDRKPK01=7 displayed, even though the actual metric value is 72.00. However … that char ‘7’ is the 1024’s char in the whole string.

The data inside the graphite db does reflect the same issue. Remote checks are done with NRPE on our site and it is not NRPR which truncates the string.

Regards,
Robert

5

2020-09-25_08h47_46
2020-09-25_08h47_46513×1279 9.01 KB

6

What about modifying check_mssql_data_size.sh to not send perfdata within one but multiple lines as described here?

7

No, multiline output as described there does not work. I do not read it from that doc, but I have to put a “|” on the start of each additional line to make the output been recognized as metric values.

However those additional chars result in less actual metrics being detected by icinga2, as the 1024’s char cuts of more metrics.

Thx.

8

My biggest result has 9320 chars but has no perfdata and i can see everything inside Icinga.

I’am using 2.12, but cant remember that i ever had such a problem.

9

This is a known limitation of NRPE, please check this :
https://support.nagios.com/kb/article.php?id=518

Currently, your options are :

  • change your max payload length if your version allow you to do this (v3+)
  • get rid of nrpe agent and use icinga agent to executer directly the check on your sql server, there are no check output limitation (at my knowledge), thus you can execute checks that return big amount of data without problems.
10

Thanks for the hints. We are actually using NRPE 3.2.1, but however forcing it to use v2 packets. I will see what I can do.

Thank you :slight_smile:.