instrumentation scope dropped from metrics

[stevesea]

context:

java spring boot app, instrumented via OTel java agent 2.10.0
(spring boot 3.4.0, java temurin 21.0.1)

in the prometheus data exposed via Grafana, I don't see instrumentation scope in the metrics.

in my application's startup, if I enable additional exporters:
OTEL_METRICS_EXPORTER=otlp,prometheus,console

and query the prom endpoint, I see the instrumentation scope there:

❯ http localhost:9464/metrics| grep micrometer | grep applica
application_ready_time_seconds{main_application_class="my.package.MyApplication",otel_scope_name="io.opentelemetry.micrometer-1.5"} 2.022
application_started_time_seconds{main_application_class="my.package.MyApplication",otel_scope_name="io.opentelemetry.micrometer-1.5"} 2.013

and, in the console output, I see instrumentation scope in the OTLP export:

.... INFO io.opentelemetry.exporter.logging.LoggingMetricExporter - metric: ImmutableMetricData{resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.24.0, attributes={deployment.envi ..... instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.micrometer-1.5, version=null, schemaUrl=null, attributes={}}, ....

but, in the otel-lgtm Grafana, if I query that metric, I don't see a instrumentation scope in the raw attributes of the metric:

application_ready_time_seconds{instance="stevec-ide", job="my-app", main_application_class="my.package.MyApplication"}

[zeitlinger]

micrometer metrics are disabled by default in the agent. What are your settings?

[stevesea]

That micrometer metric was just an example. OTel metrics similarly lack the instrumentation scope if I query prometheus in the otel-lgtm container:

http_server_request_duration_seconds_count{http_request_method="GET", http_response_status_code="200", http_route="/health", instance="stevec-ide", job="my-app", network_protocol_version="1.1", url_scheme="http"}

But, the micrometer metric is illustrative -- when micrometer metric export is enabled, there are two jvm.memory.used metrics. One from the OTel agent, and a different one from Micrometer.

Without the instrumentation scope, I can't filter for only the metric I want in my charts.
If I hit the agent's prom endpoint, it shows the two metrics (different scope, different attributes):

jvm_memory_used_bytes{area="heap",id="G1 Eden Space",otel_scope_name="io.opentelemetry.micrometer-1.5"} 5.4525952E7
jvm_memory_used_bytes{area="heap",id="G1 Old Gen",otel_scope_name="io.opentelemetry.micrometer-1.5"} 4.5054496E7
jvm_memory_used_bytes{area="heap",id="G1 Survivor Space",otel_scope_name="io.opentelemetry.micrometer-1.5"} 6695296.0
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-nmethods'",otel_scope_name="io.opentelemetry.micrometer-1.5"} 1607808.0
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'",otel_scope_name="io.opentelemetry.micrometer-1.5"} 4836096.0
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'profiled nmethods'",otel_scope_name="io.opentelemetry.micrometer-1.5"} 2.0279552E7
jvm_memory_used_bytes{area="nonheap",id="Compressed Class Space",otel_scope_name="io.opentelemetry.micrometer-1.5"} 1.0436352E7
jvm_memory_used_bytes{area="nonheap",id="Metaspace",otel_scope_name="io.opentelemetry.micrometer-1.5"} 8.2084464E7
jvm_memory_used_bytes{jvm_memory_pool_name="CodeHeap 'non-nmethods'",jvm_memory_type="non_heap",otel_scope_name="io.opentelemetry.runtime-telemetry-java8",otel_scope_version="2.10.0-alpha"} 1607808.0
jvm_memory_used_bytes{jvm_memory_pool_name="CodeHeap 'non-profiled nmethods'",jvm_memory_type="non_heap",otel_scope_name="io.opentelemetry.runtime-telemetry-java8",otel_scope_version="2.10.0-alpha"} 4836096.0
jvm_memory_used_bytes{jvm_memory_pool_name="CodeHeap 'profiled nmethods'",jvm_memory_type="non_heap",otel_scope_name="io.opentelemetry.runtime-telemetry-java8",otel_scope_version="2.10.0-alpha"} 2.0288896E7
jvm_memory_used_bytes{jvm_memory_pool_name="Compressed Class Space",jvm_memory_type="non_heap",otel_scope_name="io.opentelemetry.runtime-telemetry-java17",otel_scope_version="2.10.0-alpha"} 1.0074736E7
jvm_memory_used_bytes{jvm_memory_pool_name="Compressed Class Space",jvm_memory_type="non_heap",otel_scope_name="io.opentelemetry.runtime-telemetry-java8",otel_scope_version="2.10.0-alpha"} 1.0436352E7
jvm_memory_used_bytes{jvm_memory_pool_name="G1 Eden Space",jvm_memory_type="heap",otel_scope_name="io.opentelemetry.runtime-telemetry-java17",otel_scope_version="2.10.0-alpha"} 8.8080384E7
jvm_memory_used_bytes{jvm_memory_pool_name="G1 Eden Space",jvm_memory_type="heap",otel_scope_name="io.opentelemetry.runtime-telemetry-java8",otel_scope_version="2.10.0-alpha"} 5.4525952E7
jvm_memory_used_bytes{jvm_memory_pool_name="G1 Old Gen",jvm_memory_type="heap",otel_scope_name="io.opentelemetry.runtime-telemetry-java8",otel_scope_version="2.10.0-alpha"} 4.5054496E7
jvm_memory_used_bytes{jvm_memory_pool_name="G1 Survivor Space",jvm_memory_type="heap",otel_scope_name="io.opentelemetry.runtime-telemetry-java17",otel_scope_version="2.10.0-alpha"} 1.0824592E7
jvm_memory_used_bytes{jvm_memory_pool_name="G1 Survivor Space",jvm_memory_type="heap",otel_scope_name="io.opentelemetry.runtime-telemetry-java8",otel_scope_version="2.10.0-alpha"} 6695296.0
jvm_memory_used_bytes{jvm_memory_pool_name="Metaspace",jvm_memory_type="non_heap",otel_scope_name="io.opentelemetry.runtime-telemetry-java17",otel_scope_version="2.10.0-alpha"} 7.9056336E7
jvm_memory_used_bytes{jvm_memory_pool_name="Metaspace",jvm_memory_type="non_heap",otel_scope_name="io.opentelemetry.runtime-telemetry-java8",otel_scope_version="2.10.0-alpha"} 8.2085112E7

Here's the configuration I'm using.

otel-lgtm is running locally via docker run
docker run -p 8000:3000 -p 4317:4317 -p 4318:4318 --rm -ti grafana/otel-lgtm

in the file pointed to by OTEL_JAVAAGENT_CONFIGURATION_FILE :

otel.logs.exporter=none
otel.metrics.exporter=none
otel.traces.exporter=none

otel.resource.disabled.keys=process.command_line,process.command_args,process.command,process.pid

# For New Relic
# * use delta counters/histograms rather than cumulative.
# * use exponential bucket histograms rather than the default explicit buckets
otel.exporter.otlp.metrics.temporality.preference=delta
otel.exporter.otlp.metrics.default.histogram.aggregation=BASE2_EXPONENTIAL_BUCKET_HISTOGRAM

otel.instrumentation.micrometer.enabled=true

otel.instrumentation.runtime-telemetry-java17.enable-all=true

in my IDE, my run configuration specifies the following env vars to export to the local otel-lgtm container:

OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION=EXPLICIT_BUCKET_HISTOGRAM
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative
OTEL_JAVAAGENT_CONFIGURATION_FILE=src/main/resources/otel.properties
OTEL_LOGS_EXPORTER=otlp
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp,prometheus
OTEL_METRIC_EXPORT_INTERVAL=5000
OTEL_RESOURCE_ATTRIBUTES=service.instance.id=stevec-ide,deployment.environment=stevec-local
OTEL_SERVICE_NAME=my-app

[zeitlinger]

Thanks for explaining the use case.

The only solution I know is to check for the existence of other attributes - jvm_memory_type in this case.

BTW, I now realize that scope is not supposed to be part of the attributes - as it's usually not needed.

Resource attributes can be promoted though: https://github.com/prometheus/prometheus/blob/main/config/config.go#L1442