[GitHub] [druid] paul-rogers commented on pull request #11989: Add segment merged result cache for broker

[GitBox <gi...@apache.org>]

paul-rogers commented on pull request #11989:
URL: https://github.com/apache/druid/pull/11989#issuecomment-982054041


   @yuanlihan, cool feature! Here is one question you might want to explain: what is that use case that will be improved by this change?
   
   The feature would seem to optimize queries that hit exactly the same data every time. How common is this? It might solve a use case that you have, say, 100 dashboards, each of which will issue the same query. One of them runs the real query, 99 fetch from the cache.
   
   Here's a similar use case I've seen many times, but would *not* benefit from a post-merge cache: a dashboard wants to show the "last 10 minutes" of data, and we have minute-grain cubes (segments). Every query shifts the time range by one minute which will add a bit more new data, and exclude a bit of old data. With this design, we cache the post-merge data, so each shift in the time window will result in a new result set to cache.
   
   Can this feature instead cache pre-merge results? That way, we cache each minute slice once. The merge is still needed to gather up the slices required by the query. But, that merge is in-memory and should be pretty fast. The result would be, in this example, caching 1/10 the amount of data compared to caching post-merge data. And, less load on the historicals since we don't hit them each time the query time range shifts forward one minute.
   
   In short, please explain your use case a bit more so we an understand the goal of this enhancement.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org