Skip to content

AWS Managed Prometheus (AMP) rejects instant queries with >31d lookback range #522

@i-yevtushenko

Description

@i-yevtushenko

When using --eks-managed-prom with --history_duration greater than ~720h (30 days), KRR fails with:

HTTP Status Code 422: "queries with long day range (32d to 95d) are currently
only supported for range type queries"

Root cause:

In prometheus_metrics_service.py, pod owner lookup queries are capped at 32 days:

days_literal = min(int(period.total_seconds()) // 3600 // 24, 32)
period_literal = f"{days_literal}d"
This generates an instant query like kube_replicaset_owner{...}[32d]. AMP does not support instant queries with a lookback ≥ 32 days. However, AMP does support query_range for periods up to 95 days.

Expected behavior:

KRR should use query_range endpoint (range query) for long history lookups instead of instant queries, similar to how CPULoader explicitly sets query_type: QueryType.QueryRange.

Suggested fix:

Use QueryType.QueryRange for the pod owner lookup, or split the query into smaller chunks.

Environment:

KRR v1.28.0
AWS Managed Prometheus (AMP)
--history_duration 960 (40 days)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions