Sorry for the late reply…
I’m not really sure what you mean with “timeline picutre” but I’ll try to explain here.
Let’s say we have a bunch of services for which we are sending notifications to a timeseries database like InfluxDB. On every notification we are sending measurement points to InfluxDB with a few tags like ‘problem_type’, ‘owner’, ‘environment’ etc.
A service becomes critical and we send something like the following to InfluxDB:
notifications,service=the_check,owner=myteam,zone=eu,site=london,host=the-host,notification_type=PROBLEM,state=CRITICAL duration=0 1554288881
A while later someone is acknowledging the problem. Then we send a new measurement point to InfluxDB almost same as in 2. but this time we add duration since the last state change. So the line might look like this:
notifications,service=the_check,owner=myteam,zone=eu,site=london,host=the-host,notification_type=ACKNOWLEDGEMENT,state=CRITICAL duration=637 1554289548
Later on when the problem have recovered and a recovery notification is sent out we send the total duration:
notifications,service=the_check,owner=myteam,zone=eu,site=london,host=the-host,notification_type=ACKNOWLEDGEMENT,state=CRITICAL duration=2943 1554292491
This is not only for this kind of notifications could be really great to include in the notification emails, slack messages or what ever you are using. Why not in the GUI as well so when you check the service / host details you could see how long the service / host have been in the current state.
The issue now is that we need to calculate the “duration” on our own now. For the moment we are using a database table with every notification in. So when we send a notification we add that notification with the receiver and start time to a line. Then when that problem the notification belonged to we use the start time saved in the db table and calculate the duration. That’s the short and lazy description of how we do it.
If there was a variable containing the duration of the last problem it would make life a lot easier.