Rust's performance characteristics make it an increasingly popular choice for high-throughput services, APIs, and system-level applications. But high performance without observability is a liability โ you need to know what's happening inside your Rust services in production. This guide covers the complete Rust observability stack: metrics, distributed tracing, structured logging, error tracking, and uptime monitoring.
The Rust Observability Stack
Rust's observability ecosystem has matured significantly. The three pillars of observability โ metrics, traces, and logs โ are well-supported through a combination of first-party crates and the OpenTelemetry standard:
| Pillar | Primary Crate | Backend |
|---|---|---|
| Metrics | metrics / metrics-exporter-prometheus | Prometheus + Grafana |
| Distributed Tracing | tracing + opentelemetry-otlp | Jaeger / Grafana Tempo / Honeycomb |
| Structured Logging | tracing-subscriber (JSON) | Loki / CloudWatch / Datadog |
| Error Tracking | sentry (sentry-rust) | Sentry.io |
| Uptime / API Health | reqwest + tokio (health check endpoint) | Better Stack / APIStatusCheck |
1. Metrics with Prometheus
The metrics crate provides a facade that decouples your application from the specific metrics backend. This means you instrument once and switch backends without changing application code.
Setup: Cargo.toml
[dependencies]
metrics = "0.23"
metrics-exporter-prometheus = "0.15"
axum = "0.7"
tokio = { version = "1", features = ["full"] }Initializing the Prometheus Exporter
use metrics_exporter_prometheus::PrometheusBuilder;
use std::net::SocketAddr;
pub fn init_metrics() -> PrometheusHandle {
PrometheusBuilder::new()
.with_http_listener("0.0.0.0:9000".parse::<SocketAddr>().unwrap())
.install()
.expect("failed to install Prometheus exporter")
}Recording Metrics in Application Code
use metrics::{counter, gauge, histogram};
async fn handle_request(req: Request) -> Response {
let start = std::time::Instant::now();
// Increment request counter with labels
counter!("http_requests_total", "method" => req.method().as_str().to_owned(),
"path" => req.uri().path().to_owned()).increment(1);
let response = process_request(req).await;
// Record latency histogram
let duration = start.elapsed().as_secs_f64();
histogram!("http_request_duration_seconds",
"status" => response.status().as_str().to_owned())
.record(duration);
response
}
// Track active connections
fn update_active_connections(count: f64) {
gauge!("active_connections").set(count);
}Monitor your Rust service health endpoints
Better Stack provides uptime monitoring for any HTTP endpoint with 30-second check intervals. Ideal for Rust services exposing /health or /metrics endpoints.
Try Better Stack Free โ2. Distributed Tracing with OpenTelemetry
The tracing crate is Rust's standard instrumentation library, and tracing-opentelemetry bridges it to the OpenTelemetry ecosystem.
Setup: Cargo.toml
[dependencies]
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
tracing-opentelemetry = "0.27"
opentelemetry = "0.26"
opentelemetry-otlp = { version = "0.26", features = ["http-proto", "reqwest-client"] }
opentelemetry_sdk = { version = "0.26", features = ["rt-tokio"] }Initializing the Tracing Pipeline
use opentelemetry::global;
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_sdk::runtime;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
pub fn init_tracing(service_name: &str, otlp_endpoint: &str) {
let tracer = opentelemetry_otlp::new_pipeline()
.tracing()
.with_exporter(
opentelemetry_otlp::new_exporter()
.http()
.with_endpoint(otlp_endpoint),
)
.with_trace_config(
opentelemetry_sdk::trace::Config::default()
.with_resource(opentelemetry_sdk::Resource::new(vec![
opentelemetry::KeyValue::new("service.name", service_name.to_string()),
])),
)
.install_batch(runtime::Tokio)
.expect("failed to initialize tracer");
let otel_layer = tracing_opentelemetry::layer().with_tracer(tracer);
tracing_subscriber::registry()
.with(tracing_subscriber::EnvFilter::from_default_env())
.with(tracing_subscriber::fmt::layer().json())
.with(otel_layer)
.init();
}Instrumenting Functions
use tracing::{info, instrument, Span};
// Auto-creates a span for the function
#[instrument(fields(user_id = %user_id, email = tracing::field::Empty))]
async fn process_order(user_id: u64, order: Order) -> Result<OrderResult, Error> {
// Add dynamic fields to the current span
Span::current().record("email", tracing::field::display(&order.email));
info!(order_id = %order.id, "processing order");
let result = database.save_order(&order).await?;
info!(order_result = ?result, "order saved");
Ok(result)
}
// Manual span creation
async fn external_api_call(url: &str) -> Result<Response, Error> {
let span = tracing::info_span!("external_api_call", url = %url);
let _guard = span.enter();
// All code here is within the span
reqwest::get(url).await
}3. Structured Logging
The tracing-subscriber crate's JSON formatter outputs machine-parseable logs ideal for log aggregation systems like Grafana Loki, Datadog, or AWS CloudWatch.
use tracing::{debug, error, info, warn};
// Structured fields โ machine-parseable by log aggregators
info!(
user_id = 12345,
action = "login",
ip = "192.168.1.1",
duration_ms = 45,
"user authenticated"
);
// Error with context โ captured automatically when using #[instrument]
error!(
error = %e,
request_id = %req_id,
"database connection failed"
);JSON output from tracing-subscriber looks like:
{
"timestamp": "2026-05-04T09:00:00Z",
"level": "INFO",
"target": "my_service::auth",
"fields": {
"message": "user authenticated",
"user_id": 12345,
"action": "login",
"ip": "192.168.1.1",
"duration_ms": 45
},
"span": {
"name": "handle_request",
"request_id": "abc-123"
}
}๐ก Monitor your Rust service uptime every 30 seconds โ get alerted in under a minute
Trusted by 100,000+ websites ยท Free tier available
4. Health Check Endpoints
Every production Rust service should expose a health check endpoint for uptime monitoring. Here's a standard implementation using Axum:
use axum::{extract::State, http::StatusCode, response::Json, routing::get, Router};
use serde::Serialize;
#[derive(Serialize)]
struct HealthResponse {
status: &'static str,
version: &'static str,
db: bool,
uptime_seconds: u64,
}
async fn health_check(State(state): State<AppState>) -> (StatusCode, Json<HealthResponse>) {
let db_ok = state.db.ping().await.is_ok();
let body = HealthResponse {
status: if db_ok { "ok" } else { "degraded" },
version: env!("CARGO_PKG_VERSION"),
db: db_ok,
uptime_seconds: state.started_at.elapsed().as_secs(),
};
let status = if db_ok { StatusCode::OK } else { StatusCode::SERVICE_UNAVAILABLE };
(status, Json(body))
}
pub fn app(state: AppState) -> Router {
Router::new()
.route("/health", get(health_check))
.route("/metrics", get(metrics_handler)) // Prometheus scrape endpoint
.with_state(state)
}5. Error Tracking with Sentry
The official sentry crate integrates with Rust services to capture panics, errors, and custom events:
// Cargo.toml
// sentry = { version = "0.34", features = ["tracing"] }
fn main() {
let _guard = sentry::init(("https://your-dsn@sentry.io/123", sentry::ClientOptions {
release: sentry::release_name!(),
environment: Some("production".into()),
traces_sample_rate: 0.1, // 10% of transactions
..Default::default()
}));
// Panics are automatically captured
// Use capture_anyhow for explicit error reporting:
if let Err(e) = run_application() {
sentry::capture_error(&e);
}
}6. Rust-Specific Metrics to Monitor
Beyond standard application metrics, monitor Rust-specific runtime characteristics:
Tokio task count
Why it matters: Task accumulation indicates backpressure or missing .await calls
How to measure: tokio-metrics crate or tokio console
Thread pool utilization
Why it matters: Blocking calls on async threads cause latency spikes
How to measure: tokio::task::spawn_blocking calls; monitor separately
Memory allocations
Why it matters: Rust is efficient but custom allocators have overhead
How to measure: jemalloc + malloc_stats; or dhat for heap profiling
Channel queue depth
Why it matters: Full channels cause backpressure that blocks callers
How to measure: Expose gauge metrics from mpsc channel receivers
Connection pool utilization
Why it matters: Pool exhaustion causes request queuing and timeout errors
How to measure: sqlx / deadpool expose pool size and wait metrics
Recommended Alerting Rules for Rust Services
| Alert | Threshold | Severity |
|---|---|---|
| HTTP 5xx error rate | > 1% of requests for 5 minutes | Critical |
| P99 request latency | > 500ms for 10 minutes | Warning |
| Health check failure | Endpoint returns non-200 | Critical |
| Tokio runtime blocked | tokio_blocking_threads > 90% of pool | Warning |
| DB connection pool exhausted | Wait queue > 10 for 2 minutes | Critical |
| Panic count | Any panic in 5 minutes | Critical |
Set up uptime monitoring for your Rust services
Better Stack monitors your /health endpoints every 30 seconds and alerts you the moment your Rust service goes down. Free tier available.
Try Better Stack Free โFrequently Asked Questions
What is the best crate for Rust metrics?
The metrics crate (facade) + metrics-exporter-prometheus is the most popular combination. It decouples your app from the backend โ you can switch from Prometheus to StatsD without changing application code. For direct Prometheus integration, the prometheus crate is the official Prometheus Rust client.
How do I add OpenTelemetry tracing to a Rust application?
Add tracing, tracing-opentelemetry, opentelemetry-otlp, and opentelemetry_sdk to Cargo.toml. Initialize an OTLP exporter pointing at your collector (Jaeger, Grafana Tempo, or Honeycomb). Use #[instrument] on async functions and the span! macro for manual spans.
How do I do structured logging in Rust?
Use tracing-subscriber with JSON output: tracing_subscriber::fmt::layer().json(). This outputs machine-parseable JSON logs with all structured fields. Route stdout to your log aggregation system (Loki, Datadog, CloudWatch). Use info!(), warn!(), error!() macros with key=value fields.
Should I use log or tracing in Rust?
Use tracing for new projects. It's a superset of log with structured data, async-aware context propagation, and OpenTelemetry integration. The tracing-log crate bridges old log-based libraries into the tracing ecosystem, so the two are compatible.
Related Monitoring Guides
Alert Pro
14-day free trialStop checking โ get alerted instantly
Next time your Rust API goes down, you'll know in under 60 seconds โ not when your users start complaining.
- Email alerts for your Rust API + 9 more APIs
- $0 due today for trial
- Cancel anytime โ $9/mo after trial