Monitoring Microservices: A Best Practices Guide

Microservices are more challenging to monitor than monolithic applications because they function as distributed systems with numerous independent components. Without monitoring, teams risk performance blind spots, inefficient resource use, and unnoticed failures that affect the user experience.

This article breaks down the process of monitoring microservices into three key component areas: metrics, logging, and distributed tracing. We explore and expand on the different types of metrics and examine distributed tracing as a method of identifying request flows through services, logging and logging strategies as an opportunity to capture and analyze interesting events we may want to investigate, and the role of service-level objectives (SLOs) in helping achieve reliability. We conclude by discussing common pitfalls and challenges in monitoring microservices and extensively describe best practices and recommendations.

Summary of monitoring microservices best practices

The three pillars of monitoring microservices

Monitoring microservices starts with three core pillars: metrics, logs, and traces. You can build on these with enrichment, correlation, and advanced querying.

Metrics

Metrics are important in monitoring as they provide visibility into individual service behaviors and overall system performance. Metrics are represented as numbers; they are numerical values measured over a period of time that indicate a specific measurement of a service or system's health.

A metric must include a metric name, value, and timestamp.

Types of metrics in microservices

Logs

Logs can be in a structured or unstructured format. They are streams of timestamped records describing what happened in your system and when. They provide insights into unpredictable behavior in microservices by logging events, transactions, errors, warnings, and user interactions. They are typically used for debugging, troubleshooting issues, auditing activity, and analyzing system behavior, especially in the case of incidents.

A typical log entry might look like:

A structured log sample (source)

Logs can be categorized according to their format into:

• Plaintext logs: Human-readable strings are the most widely used format due to their simplicity.
• Structured logs: These contain key-value pairs or JSON data and are rich in metadata and more straightforward to parse and query.
• Binary logs: These are sets of compact, encoded records optimized for performance and storage. They require specific reading and analysis tools.

Types of logs in microservices

Log aggregation in microservices

Logging is essential in troubleshooting, but it's not simple in microservices architecture since the logs are distributed among different services, each writing to its own log files. Log aggregation overcomes this problem by consolidating all the logs of all services into a unified logging system like Elasticsearch, Splunk, Datadog, or New Relic. This logging system is responsible for collecting, aggregating, and storing the logs, and from this logging system, teams can search, graph, and analyze the logs.

For example, within an ecommerce application built using microservices—such as payment, inventory, and shipping—every service has its own logs. Without aggregation, identifying a problem like a failed order would require accessing the logs on different servers. However, with aggregation, all correlated logs can be queried centrally, like by order ID, so the root cause analysis is much easier.

Log aggregation pipeline

All log aggregation pipelines consist of three major components:

• Source: Logs are generated from different sources; those sources can be applications, infrastructure, or networks. Logging agents such as Fluentd, Logstash, or Alloy, collect the logs and ship them to the centralized logging system.
• Storage: The centralized logging system indexes and stores logs for search and analysis. This could be a file system, a database, or a cloud service. Logging tools like Splunk and Elasticsearch have high write throughput and rapid querying, and they usually have SDKs to extract meaningful information from large-volume logs.
• Consumer: Logs are consumed by several systems and tools, such as Kibana, Grafana, and PagerDuty, to enable the developers and operations teams to use their web UIs for debugging and to check the alerts for errors.

A log aggregation pipeline with ELK stack (source)

Traces and distributed tracing

Traces are essential because they provide insight into the journey of a request from one service to another. Each request is assigned a unique trace ID, which allows the tracing of the request's path through all services and their child calls. This, in turn, helps the team identify performance bottlenecks and root causes more quickly.

Tracing breaks down a request into spans. Each span represents a unit of work done by the service and captures key metadata like operation names, timestamps, and identifiers.

For example, suppose that in an ecommerce application based on microservices—e.g., inventory service, shopping cart, payment service, and order confirmation—a customer reports a slow checkout process that leads to an abandoned cart. The checkout flow involves several services, each handling a part of the request independently, making it hard to find where the delay happened.

With distributed tracing, each request has its unique trace ID that flows through all services, and each request is broken into spans. By analyzing those traces and spans, the team can detect the root cause of the delay, such as by finding that the request was fast through the inventory and payment services but slow during order confirmation.

An order workflow in an ecommerce application showing the included traces and spans.

There are three types of tracing, summarized in the table below.

There are two methods to instrument an application for distributed tracing: manual and automatic

Manual instrumentation

This is a way to implement distributed tracing by adding specific snippets into the application code to collect and send the telemetry data. It gives complete control over data collection and reporting, which is helpful for applications with specific observability needs. It does take time and effort to implement.

For example, a developer can instrument a Java service with the OpenTelemetry SDK to create spans around specific business logic, such as wrapping a span around the checkout() function to measure the time it takes to complete a purchase.

This snippet creates a custom span around the checkout() function. It manually tracks execution time and errors for detailed tracing.

const { trace } = require('@opentelemetry/api');
const tracer = trace.getTracer('checkout-service');

async function checkout(userId) {
  const span = tracer.startSpan('checkout-operation');
  try {
    // Business logic here
  } catch (err) {
    span.recordException(err);
    throw err;
  } finally {
    span.end();
  }
}

Automatic instrumentation

This is a simple way to configure tracing. It involves adding an agent or an SDK to your application without modifying your application code. It is less customizable than manual instrumentation, but it is far simpler and quicker to configure.

For example, a developer can use the OpenTelemetry auto-instrumentation agent for Java by attaching it to the JVM. This agent automatically traces popular libraries like Spring, JDBC, and HTTP clients without changing the application code.

This command starts the app with an agent that auto-instruments libraries without touching the source code:

java -javaagent:opentelemetry-javaagent.jar -jar your-app.jar

Service-level indicators (SLI)

SLIs are the foundation of service-level objectives (SLOs), representing the metrics upon which SLOs are built. Thus, SLIs indicate a service's performance, while the SLOs define the target for that performance.

It’s also worth noting that not every single metric should be considered an SLI metric—only those that reflect the intended user experience need to be involved. The table below describes some examples of the most common SLIs.

Service-level objectives (SLO)

Maintaining reliability and meeting performance targets is critical in distributed systems with multiple dependencies, like microservices applications. This is why service- level objectives (SLOs) are important.

SLOs are predefined targets, often expressed in percentages, that indicate the system’s reliability and performance over a specified period. They allow teams to evaluate how well the services meet their goals and lead to user satisfaction.

SLOs transform collected telemetry data into actionable targets, bridging the gap between metrics and reliability by defining measurable goals that track what matters most, such as response times, error rates, and availability. That's how monitoring and SLOs together create a complete reliability strategy. As monitoring tools like Prometheus, Datadog, or Dynatrace gather the essential data, SLOs provide the framework for understanding and improving system performance.

For example, Nobl9, a SaaS SLO management platform, helps apply this strategy by integrating with different monitoring systems and then establishing SLOs in a consistent, policy-driven manner. Nobl9 also enables alerts on SLO error budgets and creates reliability reports for the whole system.

Nobl9 integrates with various monitoring and observability tools. (source)

Composite SLOs in microservices

In a microservices architecture, each service might have its own SLO, but individual SLOs alone often don't reflect the overall reliability of the application. One solution is to combine SLOs into a composite SLO.

Composite SLOs combine the single SLOs of the multiple services to represent the system's or application's overall reliability. For example, if an application includes a database, an API, and a frontend, each with its own SLO, the reliability of the entire system is determined by combining these into a composite SLO.

Weighting in composite SLOs

In microservices architectures, not all services have an equal impact on the user experience, which is why weighting is needed. This method allows the more critical services to have a greater contribution toward the composite SLO calculations than the less critical services.

By applying appropriate weights, alerting policies can prioritize the services most critical to the user experience. This enables the classification of alerts by business impact, excluding noise from less critical services and reducing alert fatigue. Instead of reacting equally to every minor incident, teams can focus their attention on more important services.

Example 1: SLOs with equal weights

Let's consider an application composed of four microservices (A, B, C, and D), where each service has its own reliability target: Service A aims for 99% uptime, Service B for 98%, Service C for 95%, and Service D for 97%, with equal weights.

Normalized weight = Weight of service / sum of all weights * 100 = 1/4 * 100 = 25 %

Here the composite SLO is (99% * 25%) + (98% * 25%) + (95% * 25%) + (97% * 25%) = 97.52%. Since the weights are the same, this is equal to a simple average of the target SLO numbers.

Example 2: SLOs with varying weights

For the same previous example, we will assign different weights for services A, B, C, and D as 8,6,4 and 2, respectively.

Here, the composite SLO = (99% * 40%) + (98% * 30%) + (95% * 20%) + (97% * 10%) = 97.7%

How to assign weights

There's no universal rule about weighting individual SLOs. The best approach will depend on your system's context and specific requirements. Here are some common approaches for weight assignments:

• Availability-critical SLOs: In most cases, service availability is more important than service response time because a slow but functional service is better than a completely unavailable service. In such cases, it is recommended to assign a higher weight to the availability SLOs than the latency SLOs.
• Latency-critical SLOs: In some specific services, such as online meetings and video conference tools, the response time is as important as the service's availability. In such cases, it is recommended that equal weights be assigned to availability and latency SLOs.
• Business requirements: When creating a composite SLO for a service or application that includes a multi-step user journey, it's recommended to assign weights based on the relative business importance of each step. The steps that have more influence on the business should have higher weights. For example, ecommerce sites might prioritize the order and payment steps.

Calculating composite SLOs in practice can be challenging. Nobl9 is one vendor that has an out-of-the-box solution for creating composite SLOs with customizable weights.

Challenges in monitoring microservices

Monitoring in a distributed architecture presents unique challenges. Here are the key challenges involved.

Complexity of the distributed architecture

Microservices architectures are significantly more complex than monoliths. In a monolithic application, it's possible to monitor the entire application from a single place, but with microservices, it's necessary to track the behavior of multiple independent services that are often deployed across distributed clusters and typically run in containers orchestrated by Kubernetes, layered on virtual machines and physical servers. There may be load balancers, service meshes, and other networking components. 

All of this complexity makes the process of monitoring a complicated one due to several factors:

• Additional abstraction layers: The additional abstraction layers between microservices make it harder to gather the telemetry data from a central agent on the host. For example, an external monitoring agent that runs outside a Kubernetes cluster cannot collect the data from microservices inside the cluster unless each microservice code is instrumented to send monitoring data to the agent.
• Scalability: The Kubernetes clusters that host the microservices are dynamic and scale up or down frequently. For example, Kubernetes pods and containers often scale up and down, and services migrate from one node to another according to availability. While monitoring tools can track these infrastructure changes, it becomes harder to understand the actual end-user experience from infrastructure metrics alone. Application Performance Monitoring (APM) tools help bridge this gap. 
• Service-to-service dependencies: Another critical challenge is tracking service-to-service dependencies, which is essential in systems that may contain a lot of microservices. Distributed transactions often span multiple services, and failure in one component can surface symptoms elsewhere. For example, users might experience video playback issues caused by caching failure in the CDN, not by the frontend service as expected. For another, when an authentication service in an e-commerce platform goes down, users won't be able to check out, put items in their carts, or see personalized recommendations. That leads to the same conclusion that root cause identification is difficult without proper tracing.
• Asynchronous communication patterns: Microservices frequently communicate through asynchronous calls, message queues, and event-driven architectures, which make it difficult to correlate cause and effect from a single monitoring pane, as the impact of failures or performance issues may not be immediately visible.

Diversity in libraries and programming languages

Developers usually build microservices architectures using a variety of programming languages, resulting in polyglot environments that complicate the monitoring process. Those microservices may be in Java, Python, Go, Node.js, or other languages, each with its runtime characteristics and integration requirements.

Maintaining consistent monitoring practices in such environments requires tools that support multiple languages and frameworks. Solutions like Prometheus, Datadog, and many others are well-suited to this challenge, as they offer broad language support and seamless integration across diverse tech stacks and code languages.

Large data volumes

Microservices applications generate large amounts of observability data through logs, metrics, and traces. Though this data is required to understand each service well, the amount of it can be overwhelming. The number of individual data sources and the high volume and cardinality of the telemetry data make storage, processing, and analysis more difficult.

Even with automated data collection, processing this volume of data at scale introduces many obstacles, and analyzing and correlating this data becomes more challenging as it grows.

Best practices in monitoring microservices

Alert on SLOs

To effectively monitor microservices' performance, set up SLO-based alerts. This allows teams to be notified to take action before an SLO breach occurs which is based on end-user impact rather than changes in metrics that may not have any impact on end-users. There are two common ways to alert on SLOs.

Alert on the remaining error budget

Error budget-based alerts track the amount of the error budget consumed and the amount remaining. This method helps clarify how much room remains before an SLO breach.

One option is to alert on the remaining percentage; for example, an alert could be triggered when the service has only 20% of its error budget left. Another is to alert on the remaining duration, such as triggering when only 90 minutes of allowable downtime remain in the SLO window.

Alert on burn rate 

Create alerts based on burn rates to notify teams about failure incidents that cause an error budget to be consumed before the end of the SLO time window.

When alerting on burn rates, it's always recommended to follow the "multi-window, multi-burn-rate" strategy by configuring the alerts using both short- and long-term windows strike a balance between responsiveness to fast-burning incidents and stability in reducing false positives.

Combining multiple alert conditions (e.g., alert if both 5m burn rate > 14 and 1h burn rate > 1) helps:

• Trigger alerts only when the issue is real and sustained
• Reduce false positives due to momentary blips
• Ensure early warning for high-impact events

You can achieve this strategy by setting up two alerting windows: one short (e.g., 5 minutes or 30 minutes) and another long (e.g., 1 hour, 6 hours, or 3 days). For example, you can alert in these cases:

• If the burn rate over the last 1 hour consumes 2% of the budget and the burn rate over the last 5 minutes consumes 2% of the budget
• If the burn rate for the last 6 hours consumes 5% of the budget and the burn rate for the last 30 minutes consumes 5% of the budget
• If the burn rate for the last 3 days consumes 10% of the budget and the burn rate for the last 6 hours consumes 10% of the budget

A burn rate higher than 1× will consume the error budget before the time window ends. (source)

Adapt to rapid changes with automation

In distributed systems, services are continuously deployed, updated, or scaled. Manually configuring the monitoring in such continuously changing environments is inefficient. Here are some recommendations for handling the monitoring automation:

• Automate monitoring configuration: Apply automatic monitoring configuration using libraries and agents that automatically collect metrics without requiring manual configuration for every service. Also, use service discovery mechanisms to automatically detect new services once they are created without the need to instrument those services' code.
• Integrate SLO definition and monitoring into the CI/CD pipelines: Use scripts and automation tools to integrate monitoring easily into your CI/CD workflows. This ensures that monitoring configurations are consistently applied and updated with each deployment.

Regularly review and update

Monitoring microservices is not a set-it-and-forget-it task; it’s essential to make regular updates part of your monitoring lifecycle. Specifically:

• Review your monitoring strategy quarterly: Evaluate which metrics and alerts are still relevant, identify any gaps in visibility, and reassess the effectiveness of your current tools and configurations.
• Conduct monthly check-ins: Use shorter, focused sessions to address recent incidents, fine-tune alert thresholds, and quickly resolve recent monitoring issues.
• Review SLOs on a weekly basis
• Trigger reviews after significant changes: Whenever new services are introduced or significant architectural changes occur, revalidate your instrumentation and alerting logic to ensure continued observability.
• Use historical data: Save the history of performance logs and alerts to detect long-term trends and continuously improve the monitoring approach.

Configure health checks

Health checks are essential for the monitoring of microservices. They are more than just ping tests; they help ensure that only healthy services receive traffic and that unresponsive components are detected and recovered automatically.

• Configure health endpoints: Implement /health and /ready endpoints for every service by sending a test request to the service and checking the response status. For example, you can forward an HTTP request to the /health endpoint in your service, so it returns a 200 OK status code that indicates that the service is healthy.
• Use liveness and readiness probes: In platforms like Kubernetes, configure liveness probes to detect and restart unhealthy containers and readiness probes to control when services are ready to receive traffic.
• Add health checks for dependencies: Extend your health checks to verify the availability of critical dependencies like databases, caches, or external APIs.
• Include the health checks in orchestration: Let health check results drive automated actions like scaling, failover, or traffic rerouting so that you can maintain availability without manual intervention.

Use distributed tracing

As discussed earlier, the complexity of distributed architectures makes tracking and troubleshooting issues across multiple services and infrastructure layers difficult. To overcome this challenge, it is crucial to implement distributed tracing, which provides end-to-end visibility by tracing how a request passes through each service component.

Here are the key recommendations for implementing distributed tracing effectively:

• Choose a distributed tracing tool: To collect and visualize traces, select a solution like Jaeger, Zipkin, OpenTelemetry, or AWS X-Ray.
• Instrument your services: Add tracing instrumentation in each microservice to automatically generate and propagate trace data.
• Assign unique request IDs: Generate a unique ID for every incoming request and ensure that it’s passed and logged consistently across all services.
• Correlate traces with logs and metrics: Include trace and request IDs in your logs and metrics to simplify troubleshooting and performance analysis.
• Analyze trace data regularly: Use traces to identify latency hotspots, service dependencies, and points of failure to guide architectural improvements.

Configure automatic service discovery

In distributed microservices, services are frequently created, scaled, or replaced, so it becomes challenging for a monitoring tool to monitor them. To address this challenge, it's necessary to discover services without any manual configurations automatically.

Here are some essential recommendations for using automatic service discovery:

• Use service discovery tools: Use tools like Consul, etcd, Eureka, or built-in Kubernetes service discovery to automatically register and deregister services as they are created or removed. This ensures that your monitoring system always has an up-to-date view of the available services.

• Integrate service discovery with monitoring tools: Configure your monitoring stack (e.g., Prometheus, Datadog, or Grafana Cloud) to pull updated lists of services automatically from your service registry.

• Tag services dynamically: Assign dynamic metadata and tags such as environment, version, and team at registration time. Those tags help you organize, filter, and analyze monitoring data efficiently.

• Handle the metadata changes automatically: Ensure that changes in service metadata (such as label updates or endpoint changes) are reflected automatically in the monitoring system without requiring redeployment.

Implement crash reporting

In containerized applications, it's always a challenge that when a pod or container crashes, its logs and data often disappear. That's why you must configure a crash reporting strategy into your monitoring stack.

Rely on an automated way for log shipping using agents or sidecars that export logs continuously to a centralized logging system before containers are deleted. This ensures that valuable insights aren't lost if a container crashes unexpectedly.

Use persistent storage for critical logs. Configure services to write crash reports or critical logs to an external, persistent storage volume.

Correlate data and use structured logging

As we discussed earlier, tracking service-to-service dependencies in systems that may contain many microservices can be challenging. To overcome this challenge, it’s essential to correlate logs, metrics, and traces.

Here are some best practices to achieve this.

Assign a unique correlation

Create a new correlation ID for every incoming request. This ID should be created during the start of the call or request (in API Gateway or ingress controller) and propagated across all downstream services. Pass the correlation ID through HTTP headers or message metadata to maintain traceability as the request flows through your microservices.

Log the correlation ID in every log entry related to that request across all services. Ensure that the correlation ID appears in traces and metrics as tags or labels for the best correlation.

Implement contextual logging

Include relevant metadata in every log message, such as userID, request URL, parameters, session ID, method name, or function signature.

The warning below clearly shows that the user was trying to check out without items, while the error indicates that the user data was not duplicated because the email already exists.

[2025-04-14 10:30:00] INFO: User [623] login successful.
[2025-04-14 10:45:00] WARN: User [623] tried to checkout order [524] without items.
[2025-04-14 10:48:10] ERROR: Cannot update user [2531]: Duplicate email 'example@domain.com'.
      

{
  "timestamp": "2025-04-14T11:01:55.789Z",
  "service": "auth-service",
  "level": "INFO",
  "message": "User login successful",
  "userId": "user-4521",
  "requestId": "req-abc123"
}

[DEBUG] Fetching user preferences from DB
[INFO] User [1023] successfully updated their profile
[WARN] Retry attempt for service [payment-gateway] timed out
[ERROR] Failed to process order [8492]: Payment declined
      

Last thoughts

Monitoring microservices requires integrating logs, metrics, and traces to provide a complete system view. In this article, we discussed each of those three pillars of observability. Also, microservices monitoring is not easy; it comes with challenges, such as the complexity of the distributed architecture, additional abstraction layers, the scalability of microservices, and service-to-service dependencies. We also discussed the challenges of using different programming languages in each service and handling the high volume of data generated in containerized environments.

We covered the best practices for monitoring microservices, such as automating monitoring configurations, integrating monitoring into the CI/CD pipeline, using distributed tracing, correlating logs with structured formats, implementing health checks, crash reporting mechanisms, and service discovery automation. Additionally, we discussed how service-level objectives (SLOs) play a crucial role in linking monitoring data to business reliability targets. By using SLO-based alerting strategies, such as error budget and burn rate alerts, teams can shift from reactive to proactive incident response and avoid alert fatigue.