adding custom metrics in openlit

[wolfgangsmdt]

Hello,

I would like to add a custom metrics to openlit user interface.
For example in the screenshot below where I have two yello arrows I would like to add TTFT (Time To First Token) and TBT (Time Between Token).

I know you use opentelemetry to get the metrics and save them after in clickhouse data base. But I do not know how and which format and structure.

So I need your help if you could explain to me more in details how we can manipulate opentelemetry and push our custom metrics to the data base and then show it on the user interface.

Thanks

Hello @patcher9 could you please help me with this or point me to the right persons.
Thank you

[patcher9]

So from the SDK side I think we can add TTFT and TBT (This should only be for cases when streaming responses). Would need this first in SDK as part of span attributes and then we can work on the UI with @AmanAgarwal041

[wolfgangsmdt]

Hello @patcher9 and @AmanAgarwal041

Do you have any news about this.

I have another questions. Please have a look below.

Q1

I am using a RAG in my application. Is there any way to benchmark and monitor every single part in my RAG pipeline.

Below is a photo which represent my RAG:

My goal is to have openlit showing everything with all the RAG components.

Q2

Is there any json file I can edit to add widgets in openlit front-end?
My goal is to add custom widgets is it possible without modifying all your front-end code with a simple json file similar to grafana?

If this is not possible is there any way I can use grafana to show all metrics and from clickhouse database instead?

Q3

How I can add custom metrics to clickhouse database and how I trigger open-telemetry so I can control when to trigger the collect of metrics and when not.

Can I start the collect manually? and how?

Thank you : )

[wolfgangsmdt]

Hello @patcher9

Thank you very much for your quick and clear reply.
I will be waiting for the new release : )

[patcher9]

Just an update, I have merged a few PR that have updated several instrumentation to include TTFT and TBT. I also updated them to align better with the community semantic conventions. Will be updating the others aswell within the next two days and create a release.

Sorry for the delay

[wolfgangsmdt]

Thank you @patcher9

Great news. I will wait for the new release.

Meanwhile I am trying to understand how Openlit saves the data to the database (column/rows) so I can parse it.

[patcher9]

The latest release now has the metrics you needed

[wolfgangsmdt]

Thank you. I will try it out on monday.

[wolfgangsmdt]

Hello @patcher9

I have a question, it is possible to use grafana localy instead of grafana cloud as mentioned here: https://github.com/openlit/openlit/blob/main/docs/latest/connections/grafanacloud.mdx#gpu-dashboard

I would like to know what should I do to get it work because apparently it uses prometheus instead of clickhouse.

So my goal is to use grafana (I will import the json) and clickhouse from openlit to show the metrics.

Thanks : )

[patcher9]

https://docs.openlit.io/latest/connections/prometheus-tempo

https://docs.openlit.io/latest/connections/prometheus-jaeger

if you are using prometheus as the metrics backend and jaeger/tempo as the tracing backend, then these two docs will give you a grafana dashboard. But if you are directly using clickhouse then dont have a dashboard built for you but should be very easy possible

You can connect clickhouse as a datasource in grafana and then build the dashboards. This has been on my mind but noone had asked till now so didnt work on it.

[wolfgangsmdt]

Hello @patcher9

Thank you very much for your reply.
I need your help to get the datasource from clickhouse.

when I connect to the database clickhouse I found this tables below:

So I used the otel_metrics_gauge but it seems that the metrics are not changing when I run inference. could you please help me to figure out where are the correct datasources metrics to use.

Thanks

[patcher9]

I would say to use the otel_traces table as that has most of the data

[wolfgangsmdt]

I will try it on monday. Thank you

[wolfgangsmdt]

Hello @patcher9 ,

Are you available on teams or discord so we can do a small call so it will be easier to communicate?