ZO Features and Comparison with Other Open Source Projects
TLDR Gaby inquires about ZO's features as compared to other projects like Vector, tantivy, and OpenDAL. Prabhat explains their reasoning, focusing on ease-of-use, performance, storage costs, and avoiding lock-in to specific technologies. Both wish luck to each other's projects.
May 22, 2023 (1 week ago)
• Syslog Server instead of Vector
• Custom Search instead of tantivy
• Custom storage writing code instead of OpenDAL
We are not really reinventing the wheel and have used most of the existing open source stuff as base on top of which we are building. We also want to make things super easy. Let's take each of the points that you mentioned.
• Syslog server instead of vector - Network admins don't always setup multiple servers. Having a single syslog server that can accept data from all the network devices directly without an intermediary makes this process super easy.
• Custom search instead of tantivy - tantivy does full text indexing and has its use cases while we do brute force search. Some of the people will benefit from full text indexing. It's a case of tradeoff around run time performance vs storage cost. tantivy provides faster searches at the expense of ~13x higher storage cost. We believe that 13x lower storage cost makes a lot of difference. It also has a custom query language. You will have to learn it and get tied to it. We use standard SQL. Nothing to learn here if you already know SQL and plenty of guidance around it is available. We instead use apache arrow datafusion. Not re-implementing stuff from scratch here. Apache arrow datafusion has much larger community and things will only improve and at a much faster rate than tantivy. Also parquet storage makes this super easy for people to use any other tools if they ever want to to analyze instead of getting locked-in.
• Quickwit - It existed when I started building ZincSearch and evaluated it. Had it worked I would not have built ZincSearch and ZincObserve. Here is a blog that I wrote when I first built zincsearch - https://prabhatsharma.in/blog/in-search-of-a-search-engine-beyond-elasticsearch-introducing-zinc/ . Think about it, quickwit existed much earlier than ZincObserve and you knew about it, and yet you could never use it. You can't use it even today.
• Databend is more of analytics and while the underlying technology is very similar, use cases are different and this high focus on the use case is what makes things appealing for the end user. At the end of the day almost every platform is a data store. Put data in and get data out. Specific tailoring to the use case is what makes a product appealing for the specific set of users.
• We use https://github.com/apache/arrow-rs/tree/master/object_store instead of opendal . Very similar.
• VRL for functions.
• Quasar and VueJS for frontend.
• Plotly for graphing.
Hope this is helpful.
May 23, 2023 (1 week ago)
Monitoring and Metrics Setup with ZO, Vector, and Docker Compose
Gaby inquired about dashboard integration and setting up metrics with ZO. Prabhat explained using PromQL and provided a configuration example for a single-node setup with Vector.
Integrating Zincsearch/Zincobserve API into Custom GUI
Vinod wants to integrate zincsearch/zincobserve into an existing application with a custom GUI. Hengfei and Prabhat guided the use of the API and moving away from Elasticsearch.
Ingesting Data into ZincObserve with FileBeat
Jasper inquired about ingesting data into ZincObserve using FileBeat. Prabhat suggested using fluentbit, vector, or fluentd with additional compatibility provided by zbridge, available for commercial use.