[GitHub] [druid] jasonk000 commented on issue #6827: DruidSchema and SystemSchema should avoid creating temporary collections of the cardinality of all segments

[GitBox <gi...@apache.org>]

jasonk000 commented on issue #6827:
URL: https://github.com/apache/druid/issues/6827#issuecomment-839362141


   👋 I'd like to tackle this, and I have some questions on implementation approach that you'd recommend. We are seeing an opportunity to improve the segments UI responsiveness by resolving this.
   
   My specific questions are (1) would you prefer a first-pr to unlock concurrency and a second-for query engine, or both combined, and more importantly (2) some guidance on how you'd prefer the calcite integration to be approached.
   
   On (2) Calcite integration: the current `scan()` implementation fully materializes all data before filtering and discarding much of it. I can see two approaches to improve this, and alternatives currently in use in the Druid codebase:
   
   1. SystemSchema currently uses the more basic `ScannableTable` implementation (this is suboptimal).
   2. The core Druid sql stack uses the more primitive building blocks with rules generation
   
   I am less familiar with Calcite but it seems from reading that:
   
   - A good middle ground would be to [implement the bulk of logic via AbstractQueryableTable](https://javadoc.io/static/org.apache.calcite/calcite-core/1.26.0/org/apache/calcite/adapter/java/AbstractQueryableTable.html)
   - The alternative implementation would be to break out a custom SegmentTable and build a parallel convention separate and optimization chain to the existing bindable convention. But, this requires building out a set of rules and transforms etc which would ultimately only be used to push down as much as possible into one loop.
   
   Corrections and suggestions to improve my understanding of Calcite most definitely welcome!
   
   --
   
   Extra background
   
   What I notice on our implementation is that there is some opportunity to improve response time for segment queries. Specifically, I can see that the `scan()` implementation fully materializes the data before handing it back to Calcite, at which point Calcite then performs filter, sort, limit. I expect it to be significantly faster to push down the query implementation details to the collection level, so that the filter and sort happen at an earlier stage in processing.
   
   To make this more concrete - if you have a query such as `select ... from sys.segments order by start desc limit 25` to sort and then fetch the first 25 rows, with thousands-millions of underlying segments, the `joda DateTime...toString()` occurs for all of them, even when not required for filter/sort, and then the majority is shortly thereafter discarded. It would be more efficient if sort & filter happened prior to materializing all data and passing it on and I expect it to improve the responsiveness of the segment UI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org