#zincobserve

Metrics Feature in ZO 0.4.0 & Telegraf Integration

TLDR Joe asked about using metrics feature in ZO 0.4.0 and Telegraf integration. Prabhat clarified its early implementation and mentioned improvements, including PromQL support and partitioning strategy for data storage.

1mo

Join the chat

Apr 21, 2023 (1 month ago)

Photo of md5-d9ca032e3941589aafa8433269974f96

Joe

07:00 PM

I can't seem to find info in the docs on which stream name to use for the metrics feature introduced in ZO 0.4.0,

Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5

Prabhat

11:43 PM

What you can do with metrics today:

• Ingest metrics usin prometheus remote write endpoint - https://domain.com/api/org_name/prometheus/write.
• Explore metrics using metrics explorer in the UI.
• Build dashboards using SQL or drag and drop in dashboards.

11:44

Prabhat

11:44 PM

It is super early for metrics. Will add docs soon.

Apr 23, 2023 (1 month ago)

Joe

10:30 AM

Ah okay, so you can't get metrics in from Telegraf then?

Prabhat

10:32 AM

You can use the standard telegraph stuff to send data as logs and can still run SQL queries on top of it, but can't store that as metrics yet. The idea is to make PromQL work on metrics streams.

Joe

10:32 AM

Ah okay

Prabhat

10:32 AM

but getting telegraf data as metrics streams is a good idea. We need to be able to specify that input is coming for metrics stream

Joe

10:33 AM

Good to hear it's being thought about 🙂

Prabhat

10:33 AM

I believe you are already using Telegraf. Would PromQL be helpful to you?

Joe

10:34 AM

I haven't used PromQL before, I am used to using Flux and InfluxQL. I'm sure it wouldn't be too difficult to learn PromQL though.

Prabhat

10:35 AM

You will still have the option to use SQL. PromQL makes life quite easy.

Joe

10:35 AM

Nice!

Prabhat

10:35 AM

but if you have to use SQL then it would not make any difference if it is a log stream or metrics stream

10:36

Prabhat

10:36 AM

What did you expect from metrics stream?

Joe

10:36 AM

Ah yep that makes sense. I have been using log streams for metrics for testing. I saw the metrics tab on the side and thought it would be a separate place where metrics are stored instead of mixing it with logs (although I have been using separate streams for this anyway)

Prabhat

10:41 AM

There are some differences in metrics stream and log streams. Metrics has way less data compared to logs and appropriate partioning needs to be done for metrics or the storage will become very inefficient. For logs we partition by hour and are able to get 4-5 MB file size which is ideal file size for storage on s3. For metrics, we get the file size per hour to be 32kb-300kb making it very inefficient. We are thinking about weekly partitioning for metrics data.

Joe

10:42 AM

Sounds good

Prabhat

10:44 AM

What is your data size looking like for metrics?

Joe

10:44 AM

Currently 5121.78 MB ingested, 149.89 MB compressed

Prabhat

10:45 AM

Are you using object store or local disk? What is the approx size of each parquet file ?

10:46

Prabhat

10:46 AM

also is the systems metrics data or some other metrics data?

Joe

10:47 AM

Local disk, and around 300K per parquet. It's systems metrics data.

Prabhat

10:48 AM

you will be fine using local disk. If you choose to move to object storage this is slightly inefficient and you would want bigger files.

Joe

10:48 AM

Ah okay, good to know

Apr 25, 2023 (1 month ago)

Prabhat

02:02 AM

Joe I noticed that telegraf supports prometheus remote write. So I upgraded the documentation to use it - https://docs.zinc.dev/docs/ingestion/telegraf/

02:03

Prabhat

02:03 AM

Using this you can now ingest data from telegraf as metrics and use metrics explorer to see charts like this:

02:03

Prabhat

02:03 AM

May 03, 2023 (1 month ago)

Joe

10:46 AM

Oh excellent! Thanks 🙂

ZincObserve

ZincSearch is the simplest and easiest search system to get up and running. An open source easy-to-use search engine to solves your observability needs | Knowledge Base powered by Struct.AI

Generate Knowledge

Similar Threads

Monitoring and Metrics Setup with ZO, Vector, and Docker Compose

Gaby inquired about dashboard integration and setting up metrics with ZO. Prabhat explained using PromQL and provided a configuration example for a single-node setup with Vector.

Feedback on Implementing Prometheus Query Interface for ZincObserve

Prabhat shares progress on implementing a prometheus query interface for ZincObserve and asks for feedback on functionalities. Eric emphasizes the need for a unified query language across metrics, traces, and logs, discussing potential benefits and drawbacks of SQL and PromQL.

1mo

Metrics Page Support and Improving Multi-Host Filtering

Gaby wants better support for multiple sources in Metrics with filtering options. Prabhat suggests using charts with host filters. They discuss possible improvements, such as adding filtering controls, limiting capabilities, and better organization methods for metrics across hosts.

ZincObserve Release and Grafana Dashboard

Prabhat announced the new ZincObserve release and shared Grafana dashboard details. Sebastian suggested improvements to the dashboard for easier datasource customization.

Telegraf Integration with ZincObserve and Token Authentication Fix

Joe shared a config to add Telegraf data to ZO and reported an issue with token authentication. Hengfei acknowledged the issue and provided a fixed version. Joe confirmed the fix works, and it was added to the docs.

1mo