ZO Features and Comparison with Other Open Source Projects

I'm curious. I noticed several things ZO does are already done by other projects with a bigger enterprise footprint. What's the reasoning for re-implementing those features in ZO instead of relying/putting together a tool that uses those other enterprise ready open source projects?

Examples:
• Syslog Server instead of Vector
• Custom Search instead of tantivy
• Custom storage writing code instead of OpenDAL

We have seen all of these apps/libraries and are familiar with them.
We are not really reinventing the wheel and have used most of the existing open source stuff as base on top of which we are building. We also want to make things super easy. Let's take each of the points that you mentioned.
• Syslog server instead of vector - Network admins don't always setup multiple servers. Having a single syslog server that can accept data from all the network devices directly without an intermediary makes this process super easy.
• Custom search instead of tantivy - tantivy does full text indexing and has its use cases while we do brute force search. Some of the people will benefit from full text indexing. It's a case of tradeoff around run time performance vs storage cost. tantivy provides faster searches at the expense of ~13x higher storage cost. We believe that 13x lower storage cost makes a lot of difference. It also has a custom query language. You will have to learn it and get tied to it. We use standard SQL. Nothing to learn here if you already know SQL and plenty of guidance around it is available. We instead use apache arrow datafusion. Not re-implementing stuff from scratch here. Apache arrow datafusion has much larger community and things will only improve and at a much faster rate than tantivy. Also parquet storage makes this super easy for people to use any other tools if they ever want to to analyze instead of getting locked-in.
• Quickwit - It existed when I started building ZincSearch and evaluated it. Had it worked I would not have built ZincSearch and ZincObserve. Here is a blog that I wrote when I first built zincsearch - https://prabhatsharma.in/blog/in-search-of-a-search-engine-beyond-elasticsearch-introducing-zinc/ . Think about it, quickwit existed much earlier than ZincObserve and you knew about it, and yet you could never use it. You can't use it even today.
• Databend is more of analytics and while the underlying technology is very similar, use cases are different and this high focus on the use case is what makes things appealing for the end user. At the end of the day almost every platform is a data store. Put data in and get data out. Specific tailoring to the use case is what makes a product appealing for the specific set of users.
• We use https://github.com/apache/arrow-rs/tree/master/object_store instead of opendal . Very similar.
• VRL for functions.
• Quasar and VueJS for frontend.
• Plotly for graphing.
Hope this is helpful.

Thanks for the response! Totally valid points, just wanted to hear your opinion on some of them. Quickwit is doing an official release soon, so we shall see what it brings to the table

While we are working towards building the best observability platform in the whole world, the world is big enough to accommodate multiple product with different value prop for different users. We wish good luck to quickwit folks.