Add PostgreSQL observability telemetry exposure by DmytroPI-dev · Pull Request #1808 · splunk/splunk-operator

DmytroPI-dev · 2026-04-01T11:06:42Z

Description

Adds PostgreSQL observability telemetry for PostgresCluster using Prometheus pod-annotation-based scraping. Metrics are exposed by CNPG's built-in exporters on PostgreSQL pods (port 9187) and PgBouncer pooler pods (port 9127). The operator controls whether annotations are injected via class- and cluster-level configuration, with no dedicated metrics Service or ServiceMonitor required for PostgreSQL or PgBouncer scraping.

A ServiceMonitor is still supported for operator-controller metrics as an optional step.

Key Changes

api/v4/postgresclusterclass_types.go
Added class-level observability configuration (monitoring.postgresqlMetrics.enabled, monitoring.connectionPoolerMetrics.enabled) that controls whether scrape annotations are injected into CNPG pods.

api/v4/postgrescluster_types.go
Added cluster-level disable-only overrides (spec.monitoring.postgresqlMetrics.disabled, spec.monitoring.connectionPoolerMetrics.disabled) allowing per-cluster opt-out without changing the class.

pkg/postgresql/cluster/core/cluster.go
Wired observability flag resolution into PostgresCluster reconciliation. When enabled, sets InheritedMetadata.Annotations on the CNPG Cluster (for PostgreSQL pods) and Template.ObjectMeta.Annotations on CNPG Pooler resources (for PgBouncer pods).

pkg/postgresql/cluster/core/monitoring.go
Added isPostgreSQLMetricsEnabled / isConnectionPoolerMetricsEnabled flag resolution helpers.
Added buildPostgresScrapeAnnotations / buildPoolerScrapeAnnotations annotation builders.
Added removeScrapeAnnotations for the disable path.

pkg/postgresql/cluster/core/monitoring_unit_test.go
Added unit tests for flag resolution, scrape annotation builders, and annotation removal.

internal/controller/postgrescluster_controller_test.go
Added integration tests verifying that InheritedMetadata annotations are set on the CNPG Cluster when monitoring is enabled and removed when disabled by cluster override.

docs/PostgreSQLObservabilityDashboard.json
Reference Grafana dashboard covering PostgreSQL target count, RW/RO PgBouncer availability, WAL activity, database sizes, PgBouncer client load, controller reconcile metrics, and domain fleet metrics.

docs/postgresSQLMonitoring-e2e.md
End-to-end validation guide for the annotation-based scraping flow on KIND.

Testing and Verification

Added unit tests in pkg/postgresql/cluster/core/monitoring_unit_test.go for:

class/cluster observability enablement logic
scrape annotation builders for PostgreSQL (port 9187) and PgBouncer (port 9127)
annotation removal on the disable path

Added integration tests in internal/controller/postgrescluster_controller_test.go verifying:

InheritedMetadata.Annotations presence when monitoring is enabled
annotation removal when disabled by cluster-level override

Related Issues

CPI-1853 — related JIRA ticket.

Grafana screenshot:

PR Checklist

Code changes adhere to the project's coding standards.
Relevant unit and integration tests are included.
Documentation has been updated accordingly.
All tests pass locally.
The PR description follows the project's guidelines.

github-actions · 2026-04-10T09:54:36Z

CLA Assistant Lite bot:
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contribution License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment with the exact sentence copied from below.

I have read the CLA Document and I hereby sign the CLA

1 out of 2 committers have signed the CLA.
✅ @DmytroPI-dev
❌ @limak9182
_{You can retrigger this bot by commenting recheck in this Pull Request}

mploski · 2026-04-16T07:43:03Z

+	ConnectionPoolerMetrics *FeatureDisableOverride `json:"connectionPoolerMetrics,omitempty"`
+}
+
+type FeatureDisableOverride struct {


could we allign to what we already have https://github.com/splunk/splunk-operator/pull/1808/changes#diff-e68b5e4731e03120f6625b75b8562e7845606ddaeae77144b79fc4368171e848L101?

mploski · 2026-04-16T07:46:11Z

@@ -0,0 +1,321 @@
+# PostgreSQL Monitoring E2E on KIND


this file probably shouldn't be here, it sound like a internal testing approach, maybe move it to confluence?

mploski · 2026-04-16T07:49:29Z

@@ -0,0 +1,434 @@
+# PostgreSQL Monitoring E2E with OTel Collector


mploski · 2026-04-16T07:52:07Z

+  -f test/postgresql/monitoring/prometheus-via-otel-values.yaml
+```
+
+This is important: Prometheus should scrape the OTel Collector exporter, not the PostgreSQL and PgBouncer pods directly. Otherwise Grafana will bypass OTel or you will get duplicate series.


What do you mean by this statement? We dont have Prometheus in this setup?

mploski · 2026-04-16T08:26:46Z

+- `cnpg_pgbouncer_last_collection_error`
+- `cnpg_pgbouncer_pools_cl_active`
+
+## 7. Verify Prometheus is scraping OTel


prometheus shouldnt be involved at all here otel collector should send it directly to grafana: https://grafana.com/docs/opentelemetry/#ingest-otlp-data

mploski · 2026-04-16T08:28:41Z


 func (r *PostgresClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
-	rc := &clustercore.ReconcileContext{Client: r.Client, Scheme: r.Scheme, Recorder: r.Recorder, Metrics: r.Metrics}
+	metrics := r.Metrics


what problem do we solve here? Our reconciler should expect proper metrics being initiated, noop recorder is only for testing

mploski · 2026-04-16T08:29:23Z

 		}
 	}

+	recreateClusterClass := func(modify func(*enterprisev4.PostgresClusterClass)) {


It is not readable, can we make it straightforward? What problem we try to solve here? Why do we need to recreate class? especially in real live class cannot be recreated if it is already used. Maybe instead of this lets create few classes that we use in different scenarios and we change the cluster config itself

mploski · 2026-04-16T08:37:32Z

 		return ctrl.Result{}, err
 	}
-	rc := &dbcore.ReconcileContext{Client: r.Client, Scheme: r.Scheme, Recorder: r.Recorder, Metrics: r.Metrics}
+	metrics := r.Metrics


As previously I dont think we should have it like that. metrics should be initiaited in reconciler

mploski · 2026-04-16T08:50:59Z

 		rc.emitPoolerReadyTransition(postgresCluster, oldConditions)
 	}

+	oldConditions := make([]metav1.Condition, len(postgresCluster.Status.Conditions))


maybe instead of duplicating making a oldConditions copy here and in line 337 we can have it before our switch and copy just once?

mploski · 2026-04-16T08:58:12Z

 		normalized.PgHBA = spec.PostgresConfiguration.PgHBA
 	}
+	if spec.InheritedMetadata != nil && len(spec.InheritedMetadata.Annotations) > 0 {
+		normalized.InheritedAnnotations = spec.InheritedMetadata.Annotations


I probably read it wrong, but why do we set normalized.InheritedAnnotations if we already have it injected into spec?

M4KIF · 2026-04-15T09:36:15Z

+	oldConditions := make([]metav1.Condition, len(postgresCluster.Status.Conditions))
+	copy(oldConditions, postgresCluster.Status.Conditions)
+
+	if err := reconcilePostgreSQLMetricsService(ctx, c, rc.Scheme, postgresCluster, postgresMetricsEnabled); err != nil {


that could potentially be packed into unit of work style of code block, as It's very repeatable across the logic statements, It could get packed into and executable interface which handles the componentMetrics or something in this line of thought.
The code would then get separated into testable blocks and orchestrated cleanly.

M4KIF · 2026-04-15T09:36:58Z

units, nice!

M4KIF · 2026-04-15T09:38:26Z

+		return false
+	}
+	if cluster == nil || cluster.Spec.Monitoring == nil || cluster.Spec.Monitoring.PostgreSQLMetrics == nil {
+		return true


what does that mean? areMetrics enabled return true on nil value's? Could you please explain?

M4KIF · 2026-04-16T09:28:24Z

 		return ctrl.Result{}, errors.Join(err, statusErr)
 	}

+	postgresMetricsEnabled := isPostgreSQLMetricsEnabled(postgresCluster, clusterClass)


Just an idea, could that be merged into a general port like "Are the component displaying metrics == enabled"? Because there is always a possibility to expand then without adding isComponent1MetricsEnabled, isComponentNMetricsEnabled. Just the method, getComponentMetricsSettings(...) return map[string(component)]bool. One config poll for any future coming.

M4KIF · 2026-04-16T09:32:05Z

 }

 func normalizeCNPGClusterSpec(spec cnpgv1.ClusterSpec, customDefinedParameters map[string]string) normalizedCNPGClusterSpec {
 	normalized := normalizedCNPGClusterSpec{


we could potentially map It via json contract. Unless we have tags busy in our specs, which we probably have. If not, It would be mapped straight into cnpg spec.
Btw. wha do ingeritedAnnotations mean tech wise?

inherited annotations set a anottations on the k8s pod. Thanks do this we have a way to discover every pod with those annotations and this is what otel collector use to find pod endpoints to scrape from

M4KIF · 2026-04-16T09:37:23Z

+	assert.Equal(t, postgresMetricsPortString, cluster.Spec.InheritedMetadata.Annotations[prometheusPortAnnotation])
 }

 func TestClusterSecretExists(t *testing.T) {


nit: Isn't It more readable to split the units into chunks?
a naming idea, just for reference. TestClusterSecret_with_n_expected_rwro_poolers_exist

DmytroPI-dev requested review from M4KIF, limak9182 and mploski April 1, 2026 11:06

DmytroPI-dev force-pushed the postgres-operator-monitoring branch from a1b796f to 976ecd1 Compare April 2, 2026 14:08

DmytroPI-dev changed the title ~~Create ServiceMonitor and basic Grafana dashboard for metrics~~ Add PostgreSQL observability telemetry exposure via ServiceMonitors Apr 2, 2026

limak9182 reviewed Apr 8, 2026

View reviewed changes

limak9182 reviewed Apr 10, 2026

View reviewed changes

Comment thread pkg/postgresql/cluster/core/cluster.go