Vector Crashes with Timeouts during ZO Ingestion
TLDR Gaby experienced Vector crashes and timeouts during ZO ingestion. Prabhat suggested disabling alerts, and Gaby found better performance without alerts and by adjusting batch and buffer settings.
Powered by Struct AI
2 58
1w
Solved
May 20, 2023 (1 week ago)
Gaby
Gaby
02:02 PMI'm testing sending 100million events to ZO from Vector but Vector crashes with timeouts talking to ZO. The WebUI also becomes very unresponsive. It might be worth adding Batch and Buffer settings to the Vector config created by ZO UI.
Hengfei
Hengfei
02:04 PMAny logs about ZO, like error or crashed? and your event is logs or metrics?
Gaby
Gaby
02:06 PM0 errors on ZO side, just Vector crashing with a timeout talking to ZO. I will write a setup this weekend to replicate this and share it here
Hengfei
Hengfei
02:07 PMokay, 100 million logs, what the data size in ZO at the end?
02:08
Hengfei
02:08 PMand what is your node resource, cpu and memory
Gaby
Gaby
02:08 PMCompressed around ~800MB, ingested around 50GB
02:08
Gaby
02:08 PMIntel 12core, with 48GB of ram
02:09
Gaby
02:09 PMI had several instances where ram usage went to 100% during ingestion
Prabhat
Prabhat
02:10 PMIn how much time did you send all this data?
02:11
Prabhat
02:11 PME.g. Over a period of 10 minutes
Gaby
Gaby
02:12 PMProb 20-30 mins, it was pushing data as fast as vector could. It would crash and restart and keep pushing data
Prabhat
Prabhat
02:12 PMAlso, did you have any ingest functions?
Gaby
Gaby
02:13 PMI did a test sending 100GB at once using, and that never succeeded
02:13
Gaby
02:13 PM0 functions, only 1 Alert
Hengfei
Hengfei
02:13 PMPrabhat only 50GB data it should use so many memory.
Prabhat
Prabhat
02:13 PMRealtime alert?
Gaby
Gaby
02:13 PMYes
Hengfei
Hengfei
02:14 PMGaby How do you generate logs, What is the generate speed?
Gaby
Gaby
02:15 PMA mix of a few hosts with verbose syslog + vector demo logs
Prabhat
Prabhat
02:17 PMShould not use this much memory. We have added way too many things in the ingest path. We have anyways planned to review it starting next week.
Gaby
Gaby
02:17 PMYou can use multiple instances of this to generate a ton of logs:
https://vector.dev/docs/reference/configuration/sources/demo_logs/
https://vector.dev/docs/reference/configuration/sources/demo_logs/
Prabhat
Prabhat
02:18 PMYeah, have used it once
Gaby
Gaby
02:18 PMJust set interval to 0.0 and add 5-10 instances of it
02:18
Gaby
02:18 PMAnd 1 Sink to ZO
Hengfei
Hengfei
02:19 PMDo you run it in k8s?
Gaby
Gaby
02:20 PMJust plain Docker
Prabhat
Prabhat
02:58 PMAlso, where are you running it? Aws, gcp?
Gaby
Gaby
02:58 PMLocal server
02:58 1
Gaby
02:58 PMNo cloud
1May 21, 2023 (1 week ago)
Prabhat
Prabhat
12:36 AMFrom the calculations for compute perspective in this case (Let's make some reasonable assumptions doing this) we have 10 cpu cores (Let's leave 2 cores for other work being done) ingesting 50 GB in 20 minutes. This gives us ~4 MB/Sec/Core ingestion speed together with alerts. Our last tests showed us that we could do 10-15 MB/Sec/core when we did not have alerts. Could alerts have reduced the performance by 50%?
12:36
Prabhat
12:36 AMGaby Is it possible for you to try disabling alert while doing this ingestion?
12:37
Prabhat
12:37 AMI think it might be the alerts that is causing the extra CPU utilization.
12:37
Prabhat
12:37 AMWe will still need to investigate memory utilization.
12:41
Prabhat
12:41 AMGaby Was vector running on the same machine generating test data?
12:52
Prabhat
12:52 AMTo get a better perspective on ZincObserve ingestion performance, we need to have vector running on a different machine.
12:57
Prabhat
12:57 AMIf ZincObserve was getting only 6 CPUs instead of 10 and also had alerts then, we may be still in green and performance is not really bad. But let's get some details before confirming one way or other.
04:06
Prabhat
04:06 AMat 4 MB/sec/core we should be able to ingest 4 TB on this machine in 24 hours at current situation and 10 TB if we can get to 10MB/sec/core
04:25
Prabhat
04:25 AMLet's try to smooth out the issues.
Gaby
Gaby
05:40 AMYes, Vector is running on the same machine
05:41
Gaby
05:41 AMIt makes sense the realtime alerts to be causing the issue if that Query has to run on each batch of events ingested
May 22, 2023 (1 week ago)
Prabhat
Prabhat
02:48 AMYeah, alerts can be expensive.
May 23, 2023 (1 week ago)
Gaby
Gaby
03:24 AMI turned off alerts in one of my ZO instances, and performance seems way better. Another thing I did was set the Batch in Vector to 15k events, and the Buffer to 5 million with a timeout of 300secs. Instead of alerts what I did was a simple dashboard with the same query that can be checked once in a while. I may do hourly scheduled alerts in the future.
03:25
Gaby
03:25 AMWithout the batch setting, you can run into a case where Vector gets for example 100MB and tries to send all 100MB at once to ZO. Given that ZO has a json size limit this will fail. I have increased my limit to 2x the default number
Hengfei
Hengfei
03:27 AMThanks let us know your progress
Ashish
Ashish
11:26 AMHi Gaby
11:27
Ashish
11:27 AMI tested with 7 vector instances running on single c6i.2xlarge
11:27
Ashish
11:27 AM11:28
Ashish
11:28 AMI have 2 ingestor running on two separate c6i.2xlarge
11:28
Ashish
11:28 AMingestors havent restarted or killed bacause of load
11:29
Ashish
11:29 AMhowever vector has restrted quite a few times
11:29
Ashish
11:29 AMwith OOM
Gaby
Gaby
12:00 PMI had that issue happen too, Vector started restarting every once in a while
1May 24, 2023 (1 week ago)
Hengfei
Hengfei
07:05 AMGaby What is you config for demo logs? what is the log format set?
Gaby
Gaby
12:23 PMHengfei
[sources.generate_syslog]
type = "demo_logs"
format = "syslog"
count = 100
[transforms.remap_syslog]
inputs = [ "generate_syslog"]
type = "remap"
source = '''
structured = parse_syslog!(.message)
. = merge(., structured)
'''12:25
Gaby
12:25 PMChange the count to a higher number, and add the same source multiple times with diff names
12:26
Gaby
12:26 PMAnd set
interval = 0.0Hengfei
Hengfei
12:27 PMokay, as Ashish test, ZincObserve has no problem, the first problem is when we create 10 vector instances, the vector cost too much memory.
Gaby
Gaby
12:39 PMFair enough, i havent seen the issue since using Batch and Buffer with the HTTP sink of Vector
ZincObserve
ZincSearch is the simplest and easiest search system to get up and running. An open source easy-to-use search engine to solves your observability needs | Knowledge Base powered by Struct.AI
Similar Threads
Troubleshooting High CPU and Memory Usage in Zinc Service
Zygimantas experienced high CPU and memory usage in zinc service while querying large datasets. Hengfei offered suggestions to optimize and test for local disk and S3 usage. Gaby and Prabhat discussed SIMD tag performance.
5 68
1w
Monitoring and Metrics Setup with ZO, Vector, and Docker Compose
Gaby inquired about dashboard integration and setting up metrics with ZO. Prabhat explained using PromQL and provided a configuration example for a single-node setup with Vector.
6 37
1w
Solved
ZO Kubernetes Issues: Query Log Error and Adding Users
Sa had issues with querying logs in ZO on Kubernetes and creating users. Hengfei provided solutions for both problems, including updating with a dev version and setting memory cache values.
4 100
1w
Solved
Improving CPU Utilization During Bulk Ingestion in Zinc
Ayush experiences high CPU usage during bulk ingestion in Zinc and seeks optimization. Hengfei recommends adjusting the flush WAL to index interval. Ayush doesn't see significant improvement, requests sync API, but none is available.
21
2mo
Zinc Vector Ingestor Bug and UI for Tracing
Kaan reported a bug in the configuration for vector ingestor. Prabhat resolved it and discussed the addition of a tracing UI in the future.
2 5
2mo
Solved