#zincobserve

Monitoring and Metrics Setup with ZO, Vector, and Docker Compose

TLDR Gaby inquired about dashboard integration and setting up metrics with ZO. Prabhat explained using PromQL and provided a configuration example for a single-node setup with Vector.

Powered by Struct AI
👍 2
ğŸŽ‰ 1
➕ 1
😂 1
🙌 1
37
6d
Solved
Join the chat
May 25, 2023 (6 days ago)
Gaby
Photo of md5-540a8e08ce1c199c4efaeb0388742259
Gaby
01:39 PM
It would be really useful if ZO came bundled with this dashboard from this issue by default under a "Monitoring" section.

https://github.com/zinclabs/zincobserve/issues/801
➕ 1
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
01:42 PM
PromQL support will be out in the coming release. Bundling charts or picking from a git repo or any other repo should be follow on items.
01:42
Prabhat
01:42 PM
These charts are built using PromQL
Gaby
Photo of md5-540a8e08ce1c199c4efaeb0388742259
Gaby
01:43 PM
Interesting, i have to read about PromQL
01:43
Gaby
01:43 PM
Does that mean it requires having a separate grafana/prometheus instance? Or is it built into ZO?
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
01:45 PM
So ZO exposes its metrics using prometheus endpoints. Try visiting http://localhost:5080/metrics on your laptop or change the domain to appropriate installation and you will see all the metrics of ZO in prometheus format.
01:45
Prabhat
01:45 PM
You don't need grafana
01:46
Prabhat
01:46 PM
You do however need prometheus or otel collector to scrape these metrics and feed them back in ZincObserve for now.
Gaby
Photo of md5-540a8e08ce1c199c4efaeb0388742259
Gaby
01:47 PM
Ohhh, might be worth adding in the docs how to do that with Vector, since I'm pretty sure it can do that easily. Users can then just copy paste the toml into their vector instance
01:47
Gaby
01:47 PM
😂 if i figured something out, i will share it
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
01:48 PM
Actually vector can scrape prometheus metrics
01:49
Prabhat
01:49 PM
Do you have a multi node ZincObserve setup?
01:49
Prabhat
01:49 PM
if its a single node setup then then its as simple as
[sources.my_source_id]
type = "prometheus_scrape"
endpoints = [ "" ]
🙌 1
01:50
Prabhat
01:50 PM
for source
01:50
Prabhat
01:50 PM
and for target
Gaby
Photo of md5-540a8e08ce1c199c4efaeb0388742259
Gaby
01:50 PM
Yeah i only have 1 ZO instance, and 1 Vector
01:51
Gaby
01:51 PM
Awesome, i will give that a try once the new version is out :-)
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
01:51 PM
[sinks.my_sink_id]
type = "prometheus_remote_write"
inputs = [ "my_source_id" ]
endpoint = ""
👍 1
01:51
Prabhat
01:51 PM
thats it and you are done
01:52
Prabhat
01:52 PM
Are you running it in kubernetes?
Gaby
Photo of md5-540a8e08ce1c199c4efaeb0388742259
Gaby
01:52 PM
Plain Docker
01:52
Gaby
01:52 PM
Well Docker Compose 😂
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
01:52 PM
why docker compose. Its a single container
Gaby
Photo of md5-540a8e08ce1c199c4efaeb0388742259
Gaby
01:53 PM
Both ZO and Vector are on the same host
01:53
Gaby
01:53 PM
Also NGINX for https
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
01:53 PM
oh, you have your homelab fully setup
Gaby
Photo of md5-540a8e08ce1c199c4efaeb0388742259
Gaby
01:54 PM
But vector can publish to ZO via http, within the docker network without the https overhead
👍 1
01:54
Gaby
01:54 PM
Yeah, this is a home lab setup
01:54
Gaby
01:54 PM
At work we only testing ZO, the prod metrics are in ELK still
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
01:55 PM
How many nodes of ES do you have at work?
Gaby
Photo of md5-540a8e08ce1c199c4efaeb0388742259
Gaby
01:56 PM
I think 6-8, one vm for kibana, one vm for logstash
01:57
Gaby
01:57 PM
We are mirroring data to ZO. And only 1 VM of ZO can handle the same amount 😂
ğŸŽ‰ 1
01:57
Gaby
01:57 PM
Around 2-4million logs/hr
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
01:58 PM
that is crazy. We have more performance improvements coming soon.
Gaby
Photo of md5-540a8e08ce1c199c4efaeb0388742259
Gaby
01:58 PM
Yeah the low mem usage, is insane
01:59
Gaby
01:59 PM
ELK eats ram like its candy
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
01:59 PM
Yeah, JVM is like a monster.
😂 1