More by Krzysztof KoniecznyBest Way to Silence Noisy Alerts Using SLOs
| Author: Krzysztof Konieczny
Nobl9 1.25 introduces alert resolution as part of an updated alerting mechanism
Our customers asked for a way to automate closing open alerts and help them notify their engineers about the fact. As a direct response to that feedback, we’ve added the ability to resolve an ongoing alert and inform customers about it via our PagerDuty integration. For those unfamiliar with PagerDuty, it’s a tool used for on-call management and notifications.
Every Nobl9 alert has a state reflecting its lifecycle: triggered, resolved, or canceled.
The diagram above illustrates how those states are achieved. Nobl9 customers are able to manipulate the timings through two parameters:
- lastsFor, which defines how long the alert conditions need to be met for the alert to fire.
- cooldown, which defines how long the system should wait after the alert conditions drop below the threshold to resolve the alert. If the conditions are met again during the cooldown countdown, the countdown is stopped and reset.
Setting these parameters allows customers to prevent triggering false alerts (when the conditions are met but for an insufficient amount of time) or firing duplicate alerts (when the original alert shouldn’t actually have been resolved because the trigger conditions dropped, but not for long enough to cancel it).
Support for sending alert resolution notifications is currently provided by the PagerDuty alert method, and this functionality will be added to more alert methods soon. Customers can easily add this option to new or existing PagerDuty alerts by selecting a checkbox in the Alert Policy Wizard or setting a sendResolution parameter in YAML.
With this update, you’re one click away from getting a good night’s sleep–you won’t be bothered needlessly about potential issues that have already been resolved. Enable alert resolution and go back to bed!
If you’d like to check it out for yourself and see what else Nobl9 has to offer, sign up for Nobl9 Free Edition at nobl9/signup.