A Best Practices Guide to Availability Monitoring

Set business-aligned thresholds

A key to getting your SLIs right is aligning them to business goals, not technical goals. For example, 99.95% success for payment flows (revenue-critical), 99.5% success for search queries, 99.9% for authentication requests. Don't make the mistake of using just HTTP 200 responses as your "good" events; wait for actual business confirmation. Some platforms out in the wild can return an HTTP 200 with an error field.

You can set up these SLIs as actual SLOs in Nobl9 either through their dashboard or with YAML config. The YAML version looks like this:

apiVersion: n9/v1alpha
kind: SLO
metadata:
  name: checkout-success-rate
  displayName: Checkout Success Rate
  project: ecommerce-platform
spec:
  description: "Measures customer checkout completion success"
  indicator:
    metricSource:
      name: prometheus-production
      kind: Agent
  budgetingMethod: Occurrences
  objectives:
  - displayName: "99.95% checkout success"
    target: 0.9995
    countMetrics:
      good:
        prometheus:
          promql: sum(rate(payment_transactions_total{status="completed",confirmation_received="true"}[5m]))
      total:
        prometheus:
          promql: sum(rate(payment_transactions_total[5m]))
  service: checkout-service
  timeWindows:
  - unit: Day
    count: 30
    isRolling: true

• The good query counts payment transactions where status="completed" AND confirmation_received="true". This means you're measuring actual successful purchases, not just API responses. The total query counts all payment attempts.
• Nobl9 calculates your SLI as good events / total events = successful_purchases / all_attempts. With a target of 0.9995, you're setting 99.95% as your reliability threshold, meaning an error budget of 0.05%. The timeWindows section creates a rolling 30-day measurement window.
• The budgetingMethod: Occurrences tracks the volume of good events rather than uptime percentages. When your success rate drops below 99.95%, Nobl9 starts consuming your error budget and triggers alerts based on how fast you're burning through it.

To adapt this to your services, update the promql queries to match your metric names, adjust the targets based on business criticality, and modify the timeWindows for your operational needs. Ensure your "good" events represent user success, not just system responses. Learn more about how to do this from the Nobl9 SLO as code docs.

Once configured, Nobl9 automatically tracks these SLOs, calculates error budgets, triggers burn rate alerts when consumption exceeds normal rates, and provides dashboards showing exactly how much reliability margin remains before breaching your targets.

Prioritize availability monitoring SLOs based on business impact

Not every service failure costs the same money. Payment processing going down stops revenue immediately. Product review comments failing might go unnoticed for hours. This reality drives how you set SLO targets and allocate reliability engineering effort.

Start by researching the amount of revenue lost per minute of downtime for each service you operate. This will help you calculate the cost of each minute of payment downtime versus the cost of search slowness, allowing you to understand the real impact of your reliability efforts. 

For services without direct revenue attribution, such as back-office systems or internal dashboards, measure impact through support ticket volume, dependency analysis that shows how many workflows they block, or internal productivity costs resulting from extended downtime. The table below summarizes four key areas to investigate as you define your SLO targets.

Now you can set targets that actually make sense for your business, like the following:

• Payment processing and user authentication at 99.95% because losing $5,000/minute means you can justify spending big money and time on backups and failovers for services that enable people to spend time and money on your platform.
• Notification service at 99.9% is essential for customer communication, but doesn't block transactions. Payment can be completed even if the confirmation email or sms fails. Out of 1 million events, you can tolerate 1,000 failures.
• Product search at 99.5% can accept a lower SLO because it's annoying when it's slow, but people will try again and most still convert.
• Review system at 99% because it's nice to have, but most customers don't even notice when reviews are down.

If your payment system goes down and costs $5,000 per minute, spending extra on building resilience into your systems to prevent more outages than the business can handle each month makes sense. But don't spend the same money making your product recommendations super reliable if losing them doesn’t affect your bottom line. Your login system also gets high priority because people can't explore your platform and buy anything when it's broken, and because login problems make customers call support and get frustrated. Search problems are annoying, but people will try again. When your review system breaks, most customers don't even notice.

Nobl9's Service Health dashboard showing color-coded service status across different groups of services

Nobl9's Service Health view organizes services by team and business function, using color coding to show priority: 

• Red circles indicate exhausted error budgets requiring immediate attention. 
• Yellow shows at-risk services approaching their thresholds. 
• Green represents healthy services with an available error budget. 

This visual prioritization helps incident response teams focus on business-critical failures first rather than treating all alerts equally.

Establish consistent SLO review processes for availability monitoring

Different teams measure different things. Checkout services track payment confirmations while authentication services track login success. These services interact with users differently and might need different SLIs to measure availability accurately. As products evolve, teams need flexibility to adjust measurements without breaking organizational consistency.

Three practices can help maintain this balance: the SLO Development Lifecycle Cycle (SLODLC) standardizes how teams document and review SLOs, while allowing for different metrics to be used. Error budgets provide comparable reliability metrics across different services. Alerting policies can be shared across teams regardless of what they measure.

Using the SLODLC framework to standardize availability monitoring SLOs 

SLODLC helps teams avoid reinventing the wheel when defining their own SLO creation process. Everyone gets the same worksheet with the same questions to answer before they can set up any reliability targets. Here's what the template might look like for our checkout success SLO:

The checkout team fills this worksheet differently from how the authentication team fills theirs, but both follow the same structure. During incidents, anyone can refer to these worksheets to understand each team's reliability strategy and the reasoning behind their choices.

Some common pitfalls to be wary of when standardizing SLO processes across teams are:

• Skipping the SLODLC or similar worksheet and defining SLOs ad hoc can lead to setting poorly thought-out targets that don't reflect actual user expectations or historical performance.
• Failing to review SLOs after incidents misses opportunities to evaluate and improve measurement accuracy and reliability strategies.
• Not documenting the rationale for SLI choices makes it difficult for other teams to understand your reliability decisions during collaborative incident response.

Using error budgets and alerting policies for organizational consistency

Alert policies work the same way regardless of what you're measuring. You create the policy once, then reuse it across checkout, auth, search, and any other service. Nobl9 automatically calculates and tracks these error budgets from your existing monitoring data. The platform converts your Prometheus, Datadog, or CloudWatch metrics into error budget percentages, showing burn rate trends and remaining budget across all services in a unified dashboard.

A standard alert policy YAML that can be applied to our checkout SLO might look like the one below.

apiVersion: n9/v1alpha
kind: AlertPolicy
metadata:
  name: standard-burn-rate-alert
  project: reliability-policies
spec:
  description: Reusable alert policy for detecting fast and slow error budget burns 
  severity: High
  cooldown: 5m
  conditions:
    - measurement: averageBurnRate
      value: 2
      op: gte
      lastsFor: 1h
    - measurement: averageBurnRate
      value: 1
      op: gte
      lastsFor: 24h
  alertMethods:
    - metadata:
        name: slack-reliability-team
        project: reliability-policies
    - metadata:
        name: pagerduty-oncall
        project: reliability-policies

The 2x burn rate over 1 hour catches fast-burning incidents. The 1x burn rate over 24 hours catches slow degradation. Both thresholds work across all your SLOs. Nobl9 lets you define these policies once and link them to multiple SLOs. 

When checkout, auth, and search all use the same alert policy, they share escalation workflows and notification channels while monitoring completely different user outcomes. You configure alert methods (Slack, PagerDuty, email) independently and reuse them across policies, maintaining consistent incident response regardless of which service degrades.

Integrate availability monitoring SLO dashboards with incident review

Incident postmortems usually turn into arguments about whose service broke everything. SLO dashboards give you actual numbers about user impact so you can stop playing the blame game and figure out what really happened. 

Replace your usual incident timeline with concrete data:

During incidents, teams can:

• Screenshot burn rate views directly into Slack incident channels for real-time user impact visibility
• Pull specific numbers (percentage error budget consumption, and burn rate spikes) into postmortem timelines instead of vague descriptions
• Track recovery progress by watching burn rates return to normal levels (0x-1x range) instead of relying on subjective "feels better" reports.

Teams can make this integration practical by using platforms that provide embeddable burn rate charts showing exactly when and how quickly error budgets were consumed during incidents. For recurring reliability reviews, Nobl9's System Health Review aggregates SLO health across your organization, grouping services by project, region, or custom labels to show which teams have exhausted their budgets and which remain healthy.

Nobl9's system health review report (Source)

Configure these reports to run weekly or monthly so teams walk into reliability meetings with current data instead of scrambling to pull metrics during the meeting.

Nobl9's burn rate summary dashboard showing organizational overview

Nobl9 dashboard showing individual SLO burn rate for Organization Management Latency.

The first dashboard gives incident teams an organizational view during multi-service outages. You can see that 16.6% of services have high burn rates (like Organization Management at 9.16x and Outbox at 5x), while 75% are burning error budgets at acceptable rates. This prioritization data is precisely what you need in your incident review templates.

Nobl9's burn rate dashboards also provide the data teams need for incident postmortems. In the second image, you can see an Organization Management service with a catastrophic 7.5x burn rate, consuming -688.1% of its error budget (meaning it's been unreliable for 462 hours beyond its monthly allowance). The timeline charts show exactly when the burn rate spiked and how reliability dropped to 21.18% against a 90% target.

Have availability monitoring for each dependency

User workflows span multiple services, but traditional monitoring treats them independently. When checkout breaks, you must guess which of your various dependencies caused it. Tracing solves this by following individual requests across all service boundaries, showing how dependencies interact and impact user success.

This approach makes cascade failures visible in real-time. When that authentication provider starts responding slowly, you see the impact ripple through dependent services before users start complaining, showing you the exact failure path instead of forcing you to guess.

Use tracing to map dependencies

Add distributed tracing to your applications to see which dependencies users actually hit when using your applications, and the times each takes to complete successfully. This requires instrumenting your code to generate trace spans and collecting them in a tracing system like Jaeger, Tempo, or your favorite observability platform. 

Once you have traces flowing, you'll start seeing a clearer picture than your architecture diagrams have shown you. Here's what a real checkout trace might look like:

Tempo trace view showing a successful checkout workflow

Failed checkout trace showing payment failure and cascade effects

Tracing reveals how dependencies interact during successful and failed workflows. You see which services get called, in what order, and how third-party services affect user workflows. Instead of guessing from architecture diagrams, you get real data and dashboards about what affects user success and can factor external dependencies into your reliability targets.