Skip to content

Observability

Retour provides logging, metrics, and tracing for production monitoring.

Logging

  • Format: JSON (production) or console (development)
  • Levels: trace, debug, info, warn, error, fatal
  • Config: LOG_TYPE, LOG_LEVEL in operator/worker/carrier env
# operator-env ConfigMap
data:
  LOG_TYPE: json
  LOG_LEVEL: info

Metrics (Prometheus)

The operator exposes Prometheus metrics:

Backups and restores

Metric Description
retour_backup_total Backups by phase
retour_backup_duration_seconds Backup duration
retour_restore_total Restores by phase
retour_restore_duration_seconds Restore duration
retour_backup_backend_data_orphaned_total Backend data a prune could not remove, so it does not accumulate unseen

Schedules

Backup counts do not show a schedule that has stopped firing, so each schedule's own state is exported.

Metric Description
retour_backupschedule_created_total Scheduled backups created
retour_backupschedule_last_success_timestamp_seconds When each schedule last produced a successful backup
retour_backupschedule_next_run_timestamp_seconds When each schedule fires next
retour_backupschedule_missed_runs_total Fire times that passed without a run
retour_backupschedule_suspended Whether a schedule is paused

Drills, standby, and backends

Metric Description
retour_restore_drill_total Drill runs by outcome
retour_restore_drill_last_success_timestamp_seconds When a drill last passed, per schedule
retour_restoresync_last_poll_timestamp_seconds When each RestoreSync last polled its backend
retour_restoresync_degraded Whether a RestoreSync is degraded
retour_storagebackend_ready Backend readiness, refreshed by the periodic re-check

Reporting and notifications

Metric Description
retour_report_generated_total Protection reports written
retour_report_schedules_out_of_sla Schedules past their expected interval
retour_report_unprotected_namespaces Namespaces no schedule covers
retour_notification_delivery_attempts_total Failure-webhook deliveries attempted
retour_notification_delivery_failures_total Failure-webhook deliveries that failed

The chart binds the operator's metrics endpoint on port 8443 with operator.metrics.enabled (the default), serving HTTPS with authentication; a scraper needs a token bound to retour-operator-metrics-reader, which the Prometheus Operator arranges for you. Set operator.metrics.secure: false for plain HTTP on the same port.

Shipped Alerts and Dashboard

monitoring.prometheusRule.enabled=true renders a PrometheusRule covering failed backups and restores, drills that failed or went stale, schedules that missed their window or stopped succeeding, unreachable backends, degraded RestoreSyncs, unprotected namespaces, and webhook delivery failures. Each rule carries a description written as a runbook line.

monitoring.grafanaDashboards.enabled=true ships an overview dashboard (outcomes, durations, drill status and staleness, per-schedule activity) through the usual grafana_dashboard: "1" sidecar ConfigMap.

Tracing (OpenTelemetry)

When OTEL_EXPORTER_OTLP_ENDPOINT is set, Retour exports traces to an OTLP collector:

  • Operator: Reconcile spans
  • Worker: Backup/Restore spans
  • Carrier: Rclone operation spans
  • Server: HTTP request spans (otelgin)

Deploy an OpenTelemetry Collector in the retour namespace to receive traces.

Production Overlay

The production Kustomize overlay enables:

  • JSON logging
  • Info log level
  • OTLP endpoint for tracing
  • Leader election
  • Webhook (BackupSchedule validation)