You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "jtuglu1 (via GitHub)" <gi...@apache.org> on 2026/02/20 18:13:58 UTC

[I] Druid Modernization Ideas (druid)

jtuglu1 opened a new issue, #19039:
URL: https://github.com/apache/druid/issues/19039

   ### Motivation
   
   I'm making a running list of Druid modernization ideas that I see other engines in the space have adopted/are already adopting. Please feel free to comment/add your own ideas.
   
   1. Query Latency/Speed
   
   Without hw-native implementation of physical operators, Druid lags behind other engines that implement query processing codepaths using SIMD/pipelining instructions and other native code speed-ups. Another factor to this conversation is avoiding garbage collection in high allocation/spilling scenarios. Given JDK 22's support for FFI, I wonder if it makes sense to consider adding support for these accelerations in Rust and plugging them into the existing Druid query processing path. The realtime streaming path could also potentially benefit from these changes where things like GC spikes can sink your p99 ingest throughput/increase query latencies.
   
   This kind of split between parsing/planning and processing/execution is already being adopted by initiatives like https://github.com/facebookincubator/velox and https://github.com/StarRocks/starrocks.
   
   2. Data ETL in/out of Druid
   
   The Druid segment format, while hyper-optimized for workloads within Druid, serves little value to external ETL services (Spark, etc.) and data manipulation libraries that practitioners are familiar with (pandas, arrow, etc.). I think it would be a good idea to add Apache Arrow reader/writer support for Druid segments (that would allow any 3p system that speaks Arrow to integrate with Druid). It would also open up a path for switching the internal data transfer path (peon/historical -> broker -> router -> client) to use Arrow (instead of json) as well which could potentially speed up queries significantly.
   
   3. CBO for MSQe
   
   Currently the querying setup is split between native engine and MSQE. As MSQE matures, I believe the plan is to deprecate the native engine. To be competitive with other engines like Starrocks, etc. who have CBO/statistics-based planning for queries, I think it would be a good idea to add this to MSQ (this would involve tracking things like query/datasource-level statistics, etc.).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Druid Modernization Ideas (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on issue #19039:
URL: https://github.com/apache/druid/issues/19039#issuecomment-3983456838

   I have always felt the need for the __Internal metrics ingestion__ too.
   
   I ended up doing something rudimentary via `CliEventCollector` in embedded tests where Druid services send their metrics over HTTP to an "event collector" (a new type of Druid node used in embedded tests only), which then keeps the metrics in memory but doesn't persist them.
   
   One way to implement this for production could probably build on the idea of the event collector:
   - Add a new (realtime) task type, say `index_metrics_internal`
   - The task registers an HTTP endpoint and also registers a service, say `metrics_collector`.
   - Druid services use a wrapper over the existing `HttpPostEmitter`.
     - This wrapper first discovers the task based on the service
     - Then posts metrics to this endpoint
   - Task collects metrics in a queue and creates segments for say a `__metrics` datasource from the incoming stream/queue of metrics
   
   The advantage of using a realtime task is:
   - Don't need new sys tables to query this data.
   - Support all kinds of queries that Druid currently supports (sys tables do not support all types of queries currently)
   - All the benefit of realtime tasks, like auto-discover schema, etc.
   
   Con: Takes up a task slot


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Druid Modernization Ideas (druid)

Posted by "maytasm (via GitHub)" <gi...@apache.org>.
maytasm commented on issue #19039:
URL: https://github.com/apache/druid/issues/19039#issuecomment-3937529163

   @jtuglu1 Thanks for these great ideas. I think each one deserves its own issue, since they’re each fairly substantial on their own. This will keep the conversations clearly separated and make it easier to track progress (e.g., linking PRs to issues).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Druid Modernization Ideas (druid)

Posted by "Shekharrajak (via GitHub)" <gi...@apache.org>.
Shekharrajak commented on issue #19039:
URL: https://github.com/apache/druid/issues/19039#issuecomment-4412032950

   When I started using datafusion-comet for iceberg pipeline, it got me thinking: how can we have vectorized query acceleration for druid using datafusion https://github.com/apache/datafusion.
   
   If anyone have idea and flow in mind then we can start evaluating - integration of DataFusion's Arrow-native capabilities to speed up the data flow between Druid components (e.g. cold storage , hot storage segments / nodes )


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org