I have integrated Icinga to my ELK cluster using Icingabeat. Everything is working as expected, but now I don’t want to include some of the available fields that are sent to Elasticsearch from Icingabeat.
Reason being I want to use only the fields which are of my use and wants to discard rest of the fields and hence wants to optimize the size of my Icingabeat indexes, which is currently taking GBs for a single day of storage.
I have tried to remove some of the fields (like check_result.schedule_end, check_result.schedule_start) from fields.yml inside the icingabeat directory, and restarted the icingbeat service. But when i checked the Kibana dashboard then those fields are still there. I don’t know whether I am doing the right things or not to achieve what I need.
Please help, as no information I have found on neither Google nor on your forum.
I tried the one you have suggested. I have added the fields which I want to exclude by adding a processor into the icingabeat.yml, but nothing has been changed for me. Still the fields are showing in the Elasticsearch.
Below is the one that I have added and restarted the icingabeat after that.
################### Icingabeat Configuration Example ########################
############################# Icingabeat ######################################
icingabeat:
# Defines the Icinga API endpoint
host: "localhost"
# Defines the port of the API endpoint
port: 5665
# A user with sufficient permissions
user: "api"
# Password of the user
password: "-mySecretApiPassword-"
# Configure SSL verification. If `false` is configured, all server hosts
# and certificates will be accepted. In this mode, SSL based connections are
# susceptible to man-in-the-middle attacks. Use only for testing. Default is
# `true`.
ssl.verify: false
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
########################### Icingabeat Eventstream ##########################
#
# Icingabeat supports capturing of an evenstream and periodical polling of the
# Icinga status data.
# Decide which events to receive from the event stream.
# The following event stream types are available:
#
# * CheckResult
# * StateChange
# * Notification
# * AcknowledgementSet
# * AcknowledgementCleared
# * CommentAdded
# * CommentRemoved
# * DowntimeAdded
# * DowntimeRemoved
# * DowntimeStarted
# * DowntimeTriggered
#
# To disable eventstream, leave the types empty or comment out the option
eventstream.types:
- CheckResult
- StateChange
# Event streams can be filtered by attributes using the prefix 'event.'
#
# Example for the CheckResult type with the exit_code set to 2:
# filter: "event.check_result.exit_status==2"
#
# Example for the CheckResult type with the service matching the string
# pattern "mysql*":
# filter: 'match("mysql*", event.service)'
#
# To disable filtering set an empty string or comment out the filter option
eventstream.filter: ""
# Defines how fast to reconnect to the API on connection loss
eventstream.retry_interval: 10s
########################### Icingabeat Statuspoller #########################
#
# Icingabeat can collect status information about Icinga 2 periodically. Set
# an interval at which the status API should be called. Set to 0 to disable
# polling.
statuspoller.interval: 60s
#================================ General =====================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
env: master
#================================ Drop Fields ================================
event:
processors:
- drop_fields:
fields: ["icinga.last_hard_state", "icinga.last_state", "icinga.current_check_attempt", "icinga.check_result.latency", "icinga.check_result.perfdata.DIS.value", "icinga.check_result.perfdata.DGD.value", "icinga.check_result.perfdata.DGD.max", "icinga.check_result.perfdata.DIS.max", "icinga.check_result.perfdata.SIReadLock.value", "icinga.check_result.perfdata.SIReadLock.crit", "icinga.check_result.perfdata.ExclusiveLock.warn", "icinga.check_result.perfdata.ExclusiveLock.value", "icinga.check_result.perfdata.Full_GC.crit", "icinga.check_result.perfdata.Full_GC.max", "icinga.check_result.perfdata.Full_GC.min", "icinga.check_result.perfdata.Full_GC.unit", "icinga.check_result.perfdata.Full_GC.value", "icinga.check_result.perfdata.Full_GC.warn"]
#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: true
# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
#setup.dashboards.url:
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: “kibanahost:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
#============================= Elastic Cloud ==================================
# These settings simplify using Icingabeat with the Elastic Cloud.
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user:<pass`.
#cloud.auth:
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: [“elasticsearch1:9200","elasticsearch2:9200","elasticsearch3:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
#----------------------------- Logstash output --------------------------------
#output.logstash:
# The Logstash hosts
#hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
#================================ Processors =====================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
#================================ Logging =====================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: info
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]
#============================== X-Pack Monitoring ===============================
# icingabeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# reporting is disabled by default.
# Set to true to enable the monitoring reporter.
#monitoring.enabled: false
# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Icingabeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:
# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:
#================================= Migration ==================================
# This allows to enable 6.7 migration aliases
migration.6_to_7.enabled: true
I would like to ask you not to push other users to respond, the people here are helping on a completely voluntary basis.
The forum operates on people wanting to share their knowledge and help each other in their free time.
In this specific case, I know that Blerim is on a holiday at the moment and will probably respond when he is back.
