#zincobserve

Monitoring and Metrics Setup with ZO, Vector, and Docker Compose

TLDR Gaby inquired about dashboard integration and setting up metrics with ZO. Prabhat explained using PromQL and provided a configuration example for a single-node setup with Vector.

Join the chat

May 25, 2023 (6 days ago)

Photo of md5-540a8e08ce1c199c4efaeb0388742259

Gaby

01:39 PM

It would be really useful if ZO came bundled with this dashboard from this issue by default under a "Monitoring" section.

https://github.com/zinclabs/zincobserve/issues/801

Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5

Prabhat

01:42 PM

PromQL support will be out in the coming release. Bundling charts or picking from a git repo or any other repo should be follow on items.

01:42

Prabhat

01:42 PM

These charts are built using PromQL

Gaby

01:43 PM

Interesting, i have to read about PromQL

01:43

Gaby

01:43 PM

Does that mean it requires having a separate grafana/prometheus instance? Or is it built into ZO?

Prabhat

01:45 PM

So ZO exposes its metrics using prometheus endpoints. Try visiting http://localhost:5080/metrics on your laptop or change the domain to appropriate installation and you will see all the metrics of ZO in prometheus format.

01:45

Prabhat

01:45 PM

You don't need grafana

01:46

Prabhat

01:46 PM

You do however need prometheus or otel collector to scrape these metrics and feed them back in ZincObserve for now.

Gaby

01:47 PM

Ohhh, might be worth adding in the docs how to do that with Vector, since I'm pretty sure it can do that easily. Users can then just copy paste the toml into their vector instance

01:47

Gaby

01:47 PM

😂 if i figured something out, i will share it

Prabhat

01:48 PM

Actually vector can scrape prometheus metrics

01:49

Prabhat

01:49 PM

Do you have a multi node ZincObserve setup?

01:49

Prabhat

01:49 PM

if its a single node setup then then its as simple as

[sources.my_source_id]
type = "prometheus_scrape"
endpoints = [ "" ]

01:50

Prabhat

01:50 PM

for source

01:50

Prabhat

01:50 PM

and for target

Gaby

01:50 PM

Yeah i only have 1 ZO instance, and 1 Vector

01:51

Gaby

01:51 PM

Awesome, i will give that a try once the new version is out :-)

Prabhat

01:51 PM

[sinks.my_sink_id]
type = "prometheus_remote_write"
inputs = [ "my_source_id" ]
endpoint = ""

01:51

Prabhat

01:51 PM

thats it and you are done

01:52

Prabhat

01:52 PM

Are you running it in kubernetes?

Gaby

01:52 PM

Plain Docker

01:52

Gaby

01:52 PM

Well Docker Compose 😂

Prabhat

01:52 PM

why docker compose. Its a single container

Gaby

01:53 PM

Both ZO and Vector are on the same host

01:53

Gaby

01:53 PM

Also NGINX for https

Prabhat

01:53 PM

oh, you have your homelab fully setup

Gaby

01:54 PM

But vector can publish to ZO via http, within the docker network without the https overhead

01:54

Gaby

01:54 PM

Yeah, this is a home lab setup

01:54

Gaby

01:54 PM

At work we only testing ZO, the prod metrics are in ELK still

Prabhat

01:55 PM

How many nodes of ES do you have at work?

Gaby

01:56 PM

I think 6-8, one vm for kibana, one vm for logstash

01:57

Gaby

01:57 PM

We are mirroring data to ZO. And only 1 VM of ZO can handle the same amount 😂

01:57

Gaby

01:57 PM

Around 2-4million logs/hr

Prabhat

01:58 PM

that is crazy. We have more performance improvements coming soon.

Gaby

01:58 PM

Yeah the low mem usage, is insane

01:59

Gaby

01:59 PM

ELK eats ram like its candy

Prabhat

01:59 PM

Yeah, JVM is like a monster.

ZincObserve

ZincSearch is the simplest and easiest search system to get up and running. An open source easy-to-use search engine to solves your observability needs | Knowledge Base powered by Struct.AI

Generate Knowledge

Similar Threads

Metrics Feature in ZO 0.4.0 & Telegraf Integration

Joe asked about using metrics feature in ZO 0.4.0 and Telegraf integration. Prabhat clarified its early implementation and mentioned improvements, including PromQL support and partitioning strategy for data storage.

1mo

ZincObserve Release and Grafana Dashboard

Prabhat announced the new ZincObserve release and shared Grafana dashboard details. Sebastian suggested improvements to the dashboard for easier datasource customization.

Metrics Page Support and Improving Multi-Host Filtering

Gaby wants better support for multiple sources in Metrics with filtering options. Prabhat suggests using charts with host filters. They discuss possible improvements, such as adding filtering controls, limiting capabilities, and better organization methods for metrics across hosts.

Feedback on Implementing Prometheus Query Interface for ZincObserve

Prabhat shares progress on implementing a prometheus query interface for ZincObserve and asks for feedback on functionalities. Eric emphasizes the need for a unified query language across metrics, traces, and logs, discussing potential benefits and drawbacks of SQL and PromQL.

1mo

Configuring Grafana with ZincObserve and zPlane

Phillip encountered an issue connecting Grafana to ZincObserve. Prabhat suggested using zPlane and provided documentation and deployment instructions. After resolving some errors, Phillip was able to run queries in Grafana.

1mo