feat: add OpenTelemetry metrics

README.md

@@ -229,6 +229,7 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for how to set up a development environme
 - [Using Cog with Windows 11](docs/wsl2/wsl2.md)
 - [Browse the example models in this repo](docs/examples.md)
 - [Deploy models with Cog](docs/deploy.md)
+- [Configure OpenTelemetry tracing and metrics](docs/observability.md)
 - [`cog.yaml` reference](docs/yaml.md) to learn how to define your model's environment
 - [Run interface reference](docs/python.md) to learn how the `Runner` interface works
 - [Training interface reference](docs/training.md) to learn how to add a fine-tuning API to your model

architecture/01-model-source.md

@@ -36,16 +36,18 @@ concurrency:
   max: 1
 ```
 
-| Field                   | Purpose                                     |
-| ----------------------- | ------------------------------------------- |
-| `build.python_version`  | Python interpreter version (3.10-3.13)      |
-| `build.gpu`             | Enable CUDA support                         |
-| `build.python_packages` | pip packages to install                     |
-| `build.system_packages` | apt packages to install                     |
-| `build.run`             | Arbitrary shell commands during build       |
-| `run`                   | Path to runner class (`module:ClassName`)   |
-| `concurrency.max`       | Max concurrent predictions (requires async) |
-| `observability.config`  | Optional Python tracer-provider factory     |
+| Field                   | Purpose                                                   |
+| ----------------------- | --------------------------------------------------------- |
+| `build.python_version`  | Python interpreter version (3.10-3.13)                    |
+| `build.gpu`             | Enable CUDA support                                       |
+| `build.python_packages` | pip packages to install                                   |
+| `build.system_packages` | apt packages to install                                   |
+| `build.run`             | Arbitrary shell commands during build                     |
+| `run`                   | Path to runner class (`module:ClassName`)                 |
+| `concurrency.max`       | Max concurrent predictions (requires async)               |
+| `observability.traces`  | Opt-in OpenTelemetry framework tracing                    |
+| `observability.metrics` | Opt-in runtime and model OpenTelemetry metrics            |
+| `observability.config`  | Optional Python provider and runtime metric configuration |
 
 The [Build System](./05-build-system.md) uses this configuration to produce an image containing all necessary dependencies, libraries, and the correct Python/CUDA versions.
 

architecture/04-container-runtime.md

@@ -385,17 +385,19 @@ Models can record custom metrics via `self.record_metric(name, value, mode)` in
 
 Metrics appear in the prediction response's `metrics` object alongside the built-in `predict_time`.
 
-## Distributed Tracing
+## OpenTelemetry
 
 Opt-in OpenTelemetry tracing uses a provider in the parent process, a provider in the worker process, and a Python provider installed before predictor import. The parent and worker exchange an optional W3C carrier alongside prediction IPC. Transport context remains separate from the user-owned request `context` map.
 
-Models may provide `observability.config`, which Cog validates and stages at a fixed image path. During worker setup, Cog loads that module, installs the `TracerProvider` returned by `create_tracer_provider()`, and then calls optional `configure_instrumentation()` before predictor import. Cog owns provider flush and shutdown. Configuration failures stop model setup because the user explicitly selected that module.
+Metrics use separate ownership. The parent creates the runtime MeterProvider after worker setup and exports fixed prediction, rejection, setup, and slot-state measurements. The Python worker creates the model MeterProvider for arbitrary model-owned instruments. It returns a runtime metric configuration in its `Ready` control message so the parent can omit selected fixed instruments before recording terminal setup status.
+
+Models may provide `observability.config`, which Cog validates and stages at a fixed image path. During worker setup, Cog loads that module, creates selected Python tracer and meter providers, validates them, installs them globally, and then calls optional `configure_instrumentation()` before predictor import. Cog owns provider flush and shutdown. Configuration failures stop model setup because the user explicitly selected that module.
 
 The framework trace covers HTTP handling, validation, the logical prediction lifetime, worker execution, input preparation, predictor invocation, output upload, and setup. Model code creates ordinary child spans through `opentelemetry.trace`.
 
 The `Prediction` state object owns the logical prediction span so asynchronous and SSE requests can return before the span reaches a terminal state. Signed output uploads never receive trace headers. Webhooks receive the active prediction context.
 
-Framework tracing is inert unless the image enables it and the runtime supplies a collector endpoint. A configured Python provider may run without that endpoint, but it then emits model spans without framework parents. The disabled path creates no provider, exporter, background telemetry thread, connection, or real framework span.
+Framework telemetry is inert unless the image enables the matching signal and the runtime supplies a collector endpoint. A configured Python provider may run without that endpoint, but it then emits model telemetry without framework trace parents. The disabled path creates no provider, exporter, background telemetry thread, connection, runtime instrument, or real framework span.
 
 ## User-Defined Healthchecks
 
@@ -405,16 +407,17 @@ If the healthcheck fails, the HTTP `/health-check` endpoint returns `UNHEALTHY`
 
 ## Environment Variables
 
-| Variable                         | Default | Purpose                                          |
-| -------------------------------- | ------- | ------------------------------------------------ |
-| `PORT`                           | 5000    | HTTP server port                                 |
-| `COG_LOG_LEVEL`                  | INFO    | Logging verbosity (ignored if `RUST_LOG` is set) |
-| `COG_MAX_CONCURRENCY`            | 1       | Number of concurrent prediction slots            |
-| `COG_SETUP_TIMEOUT`              | none    | Setup timeout in seconds (0 is ignored)          |
-| `COG_TRACE_ENABLED`              | false   | Runtime tracing switch for an opted-in image     |
-| `COG_OBSERVABILITY_CONFIG`       | none    | Internal path to staged Python telemetry config  |
-| `COG_THROTTLE_RESPONSE_INTERVAL` | 0.5s    | Webhook response throttling interval             |
-| `LOG_FORMAT`                     | json    | Set to `console` for human-readable log output   |
+| Variable                         | Default | Purpose                                            |
+| -------------------------------- | ------- | -------------------------------------------------- |
+| `PORT`                           | 5000    | HTTP server port                                   |
+| `COG_LOG_LEVEL`                  | INFO    | Logging verbosity (ignored if `RUST_LOG` is set)   |
+| `COG_MAX_CONCURRENCY`            | 1       | Number of concurrent prediction slots              |
+| `COG_SETUP_TIMEOUT`              | none    | Setup timeout in seconds (0 is ignored)            |
+| `COG_TRACE_ENABLED`              | image   | `true` in tracing-enabled images; absent otherwise |
+| `COG_METRICS_ENABLED`            | image   | `true` in metrics-enabled images; absent otherwise |
+| `COG_OBSERVABILITY_CONFIG`       | none    | Internal path to staged Python telemetry config    |
+| `COG_THROTTLE_RESPONSE_INTERVAL` | 0.5s    | Webhook response throttling interval               |
+| `LOG_FORMAT`                     | json    | Set to `console` for human-readable log output     |
 
 ## Where to Look
 

architecture/05-build-system.md

@@ -86,7 +86,7 @@ flowchart LR
 
 The generator produces a Dockerfile from the validated config.
 
-When `observability.config` is set, build orchestration validates the project-local file and stages it in the private `cog_build` context. Generated and custom-Dockerfile wrapper layers copy that artifact to `/.cog/telemetry.py`; the original user path is never exposed to runtime path resolution.
+When tracing or metrics is enabled, build orchestration installs the aligned OpenTelemetry Python packages and writes per-signal image markers. If `observability.config` is set, it validates the project-local file and stages it in the private `cog_build` context. Generated and custom-Dockerfile wrapper layers copy that artifact to `/.cog/telemetry.py`; the original user path is never exposed to runtime path resolution.
 
 #### Generated Dockerfile Sections
 

crates/README.md

@@ -114,9 +114,9 @@ HTTP Request                     Parent Process                    Worker Subpro
        │         │ install audit hook, load predictor, run setup │
        │         └────────────────────────────────────────────────┘
        │
-       ├─▶ Wait for Ready {slots, schema} or Failed {error}
+       ├─▶ Wait for Ready {slots, schema, runtime_metrics} or Failed {error, runtime_metrics}
        │
-       ├─▶ Populate PermitPool with slot sockets
+       ├─▶ Populate PermitPool with slot sockets and initialize parent runtime metrics
        │
        ├─▶ Start event loop (routes responses to predictions)
        │

crates/coglet-python/src/lib.rs

@@ -419,6 +419,8 @@ fn serve_impl(
         if let Some(runtime) = trace_runtime.as_ref() {
             runtime.shutdown();
         }
+        #[cfg(feature = "tracing")]
+        coglet_core::runtime_metrics::shutdown();
         return result;
     };
 
@@ -437,6 +439,8 @@ fn serve_impl(
     if let Some(runtime) = trace_runtime.as_ref() {
         runtime.shutdown();
     }
+    #[cfg(feature = "tracing")]
+    coglet_core::runtime_metrics::shutdown();
     result
 }
 
@@ -491,7 +495,7 @@ fn serve_subprocess(
 
             let setup_service = Arc::clone(&service_clone);
             let setup_span = coglet_core::cog_span!(info_span, "cog.setup");
-            tokio::spawn(
+            let setup_task = tokio::spawn(
                 async move {
                     info!("Spawning worker subprocess");
                     let spawn_start = std::time::Instant::now();
@@ -505,6 +509,17 @@ fn serve_subprocess(
                                 "Worker ready, configuring service"
                             );
 
+                            #[cfg(feature = "tracing")]
+                            coglet_core::runtime_metrics::install(
+                                ready.runtime_metrics.clone(),
+                                Some(Arc::clone(&ready.pool)),
+                            );
+                            #[cfg(feature = "tracing")]
+                            coglet_core::runtime_metrics::record_setup_duration(
+                                "succeeded",
+                                spawn_elapsed,
+                            );
+
                             let num_slots = ready.handle.slot_ids().len();
                             debug!(num_slots, "Setting up orchestrator on service");
 
@@ -549,6 +564,16 @@ fn serve_subprocess(
                                 "Worker initialization failed"
                             );
                             debug!("Transitioning health to SetupFailed");
+                            #[cfg(feature = "tracing")]
+                            coglet_core::runtime_metrics::install(
+                                e.runtime_metrics_config().unwrap_or_default(),
+                                None,
+                            );
+                            #[cfg(feature = "tracing")]
+                            coglet_core::runtime_metrics::record_setup_duration(
+                                "failed",
+                                spawn_elapsed,
+                            );
                             setup_service.set_health(Health::SetupFailed).await;
                             setup_service
                                 .set_setup_result(setup_result.failed(e.to_string()))
@@ -559,9 +584,18 @@ fn serve_subprocess(
                 .instrument(setup_span),
             );
 
-            http_serve(config, service_clone)
+            let server_result = http_serve(config, service_clone)
                 .await
-                .map_err(|e| PyErr::new::<pyo3::exceptions::PyRuntimeError, _>(e.to_string()))
+                .map_err(|e| PyErr::new::<pyo3::exceptions::PyRuntimeError, _>(e.to_string()));
+            if !setup_task.is_finished() {
+                setup_task.abort();
+            }
+            if let Err(error) = setup_task.await
+                && !error.is_cancelled()
+            {
+                tracing::error!(%error, "Setup task failed");
+            }
+            server_result
         })
     })
 }
@@ -665,6 +699,7 @@ fn coglet(py: Python<'_>, m: &Bound<'_, PyModule>) -> PyResult<()> {
     // Static metadata
     m.add("__version__", env!("COGLET_PEP440_VERSION"))?;
     m.add("__build__", BuildInfo::new())?;
+    m.add("_supports_observability_metrics", true)?;
 
     // Frozen server object
     m.add("server", CogletServer {})?;

crates/coglet-python/src/worker_bridge.rs

@@ -7,7 +7,7 @@ use std::thread::JoinHandle;
 use pyo3::prelude::*;
 use pyo3::types::PyDict;
 
-use coglet_core::bridge::protocol::SlotId;
+use coglet_core::bridge::protocol::{RuntimeMetric, RuntimeMetricsConfig, SlotId};
 use coglet_core::worker::{PredictHandler, PredictResult, SetupError, SlotSender};
 
 use crate::predictor::PythonPredictor;
@@ -58,25 +58,45 @@ fn env_true(name: &str, default: bool) -> bool {
     })
 }
 
-fn python_tracing_enabled() -> bool {
-    if !env_true("COG_TRACE_CONFIGURED", false)
-        || !env_true("COG_TRACE_ENABLED", true)
-        || env_true("OTEL_SDK_DISABLED", false)
-    {
-        return false;
-    }
-    if std::env::var_os("COG_OBSERVABILITY_CONFIG").is_some() {
-        return true;
-    }
-    if std::env::var("OTEL_TRACES_EXPORTER").as_deref() == Ok("none") {
+fn python_telemetry_enabled() -> bool {
+    if env_true("OTEL_SDK_DISABLED", false) {
         return false;
     }
-    [
-        "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT",
-        "OTEL_EXPORTER_OTLP_ENDPOINT",
-    ]
-    .iter()
-    .any(|name| std::env::var(name).is_ok_and(|value| !value.trim().is_empty()))
+    (env_true("COG_TRACE_CONFIGURED", false) && env_true("COG_TRACE_ENABLED", true))
+        || (env_true("COG_METRICS_CONFIGURED", false) && env_true("COG_METRICS_ENABLED", true))
+}
+
+fn runtime_metrics_config_from_python(
+    config: &Bound<'_, PyAny>,
+) -> Result<RuntimeMetricsConfig, SetupError> {
+    let enabled = config
+        .getattr("enabled")
+        .and_then(|value| value.extract::<bool>())
+        .map_err(|error| SetupError::setup(error.to_string()))?;
+    let disabled = config
+        .getattr("disabled")
+        .and_then(|values| values.try_iter())
+        .map_err(|error| SetupError::setup(error.to_string()))?
+        .map(|value| {
+            let value = value.map_err(|error| SetupError::setup(error.to_string()))?;
+            let name = value
+                .getattr("value")
+                .and_then(|value| value.extract::<String>())
+                .map_err(|error| SetupError::setup(error.to_string()))?;
+            match name.as_str() {
+                "prediction_count" => Ok(RuntimeMetric::PredictionCount),
+                "prediction_rejected" => Ok(RuntimeMetric::PredictionRejected),
+                "prediction_active" => Ok(RuntimeMetric::PredictionActive),
+                "prediction_duration" => Ok(RuntimeMetric::PredictionDuration),
+                "setup_duration" => Ok(RuntimeMetric::SetupDuration),
+                "slot_count" => Ok(RuntimeMetric::SlotCount),
+                _ => Err(SetupError::setup(format!(
+                    "unsupported runtime metric selector {name:?}"
+                ))),
+            }
+        })
+        .collect::<Result<Vec<_>, _>>()?;
+    Ok(RuntimeMetricsConfig { enabled, disabled })
 }
 
 fn current_trace_carrier() -> Option<HashMap<String, String>> {
@@ -179,6 +199,7 @@ pub struct PythonPredictHandler {
     /// Handle to the asyncio loop thread for joining on shutdown.
     async_thread: Mutex<Option<JoinHandle<()>>>,
     max_concurrency: usize,
+    runtime_metrics: Mutex<Option<RuntimeMetricsConfig>>,
 }
 
 impl PythonPredictHandler {
@@ -193,6 +214,7 @@ impl PythonPredictHandler {
             async_loop: Mutex::new(Some(loop_obj)),
             async_thread: Mutex::new(Some(thread)),
             max_concurrency,
+            runtime_metrics: Mutex::new(None),
         })
     }
 
@@ -207,6 +229,7 @@ impl PythonPredictHandler {
             async_loop: Mutex::new(Some(loop_obj)),
             async_thread: Mutex::new(Some(thread)),
             max_concurrency,
+            runtime_metrics: Mutex::new(None),
         })
     }
 
@@ -354,13 +377,18 @@ impl PredictHandler for PythonPredictHandler {
     async fn setup(&self) -> Result<(), SetupError> {
         Python::attach(|py| {
             let carrier = current_trace_carrier();
-            if python_tracing_enabled() {
-                let trace_module = py
-                    .import("cog._trace")
+            if python_telemetry_enabled() {
+                let telemetry_module = py
+                    .import("cog._telemetry")
                     .map_err(|error| SetupError::setup(error.to_string()))?;
-                trace_module
-                    .call_method0("install_provider")
+                let config = telemetry_module
+                    .call_method0("install_providers")
                     .map_err(|error| SetupError::setup(error.to_string()))?;
+                *self
+                    .runtime_metrics
+                    .lock()
+                    .expect("runtime_metrics mutex poisoned") =
+                    Some(runtime_metrics_config_from_python(&config)?);
             }
             let _trace_guard = PythonTraceGuard::enter(py, carrier.as_ref())
                 .map_err(|error| SetupError::internal(error.to_string()))?;
@@ -406,6 +434,13 @@ impl PredictHandler for PythonPredictHandler {
         self.mode == HandlerMode::Train
     }
 
+    fn runtime_metrics_config(&self) -> Option<RuntimeMetricsConfig> {
+        self.runtime_metrics
+            .lock()
+            .expect("runtime_metrics mutex poisoned")
+            .clone()
+    }
+
     async fn predict(
         &self,
         slot: SlotId,
@@ -793,12 +828,9 @@ impl PredictHandler for PythonPredictHandler {
     }
 
     async fn shutdown(&self) {
-        if !python_tracing_enabled() {
-            return;
-        }
         Python::attach(|py| {
-            if let Ok(trace_module) = py.import("cog._trace") {
-                let _ = trace_module.call_method0("shutdown");
+            if let Ok(telemetry_module) = py.import("cog._telemetry") {
+                let _ = telemetry_module.call_method0("shutdown");
             }
         });
     }