[GitHub] [incubator-druid] justinborromeo opened a new issue #7370: Add SQL support for time-ordered scans

[GitBox <gi...@apache.org>]

justinborromeo opened a new issue #7370: Add SQL support for time-ordered scans
URL: https://github.com/apache/incubator-druid/issues/7370
 
 
   ### Motivation
   
   See https://github.com/apache/incubator-druid/issues/6088 for original idea.  PR #7133 is close to completion and the next step is to add SQL support for time-ordered scans.  This would eliminate the need for using select queries in SQL planning since the only thing select is good for is time-ordering results.  Updating to use scan would improve memory performance.
   
   ### Proposed changes
   
   The SQL planning in `DruidQuery` will be changed so that Scan is used if ordering by __time is specified.  After that, Select will be essentially obsolete and will be removed from SQL planning altogether.
   
   The user interface won't change.
   
   ### Rationale
   
   I think removing select queries from the SQL planner completely is the best choice since its design isn't great memory-wise.  Although this means that time-ordered SELECT queries that fall outside of the configurable scan time-ordering limits (default 100K rows or 30 segments per time chunk) will fail, these limits can be tuned based on machine specs to a point where the query will succeed.  Furthermore, if the query is big enough to cause memory issues with scan, using a select will be even worse.
   
   ### Operational impact
   
   No impact to overall cluster operation.  Existing select queries might start failing if they're outside of the configurable row or segments per time chunk limits.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org