You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/06/29 12:04:40 UTC

[GitHub] [druid] nishantmonu51 opened a new pull request #10093: Add groupBy limitSpec to queryCache key

nishantmonu51 opened a new pull request #10093:
URL: https://github.com/apache/druid/pull/10093


   ### Description
   
   This PR Fixes a bug with groupBy query cache key. The current cache key for groupby queries does not contain limitSpec info. So when a user queries with a given limit lets say 10 and then increase the limit to 100, he would still get the cached results with older limit. 
   
   Note: This only shows up if you have enabled caching for groupby queries, which is disabled by default. 
   
   This PR has:
   - [*] been self-reviewed.
   - [*] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [*] been tested in a test Druid cluster.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson merged pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
jihoonson merged pull request #10093:
URL: https://github.com/apache/druid/pull/10093


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] nishantmonu51 commented on pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
nishantmonu51 commented on pull request #10093:
URL: https://github.com/apache/druid/pull/10093#issuecomment-654673472


   @jihoonson : Added a check for isLimitPushDown in the cache key generation in current patch. 
   Let me know your thoughts on changing the pushdown behaviour to push null limitSpec to underlying broker layers in a separate follow up patch and cleanup the limit push down checks.  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #10093:
URL: https://github.com/apache/druid/pull/10093#issuecomment-655024790


   > @jihoonson: I can add a check for limitPushDown when generating cache key.
   > But wouldn't it be better for the underlying layers to not worry about whether limitSpec was pushed or not every time ?
   > The broker logic if it decides to not push down the limit, shouldn't it send a query with a null limitSpec to the underlying layers ? Or is there any reason for the broker to still push down limit spec ?
   
   @nishantmonu51 ah, I'm not sure exactly why the broker always sends the limitSpec, guessing it's for simplicity. Sending limitSpec only when pushing down limitSpec seems reasonable, although I don't mind much with the current implementation. I guess it's up to you.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #10093:
URL: https://github.com/apache/druid/pull/10093#issuecomment-654502620


   > @jihoonson : Yes, its an issue when limitSpec if pushed down, other thing to note, if you push down the limit once and then try to execute a query without pushing down the limit or remove the limit itself, then also it will be problematic as the cache will have the results with limit and the results will be served from cache.
   
   Thanks for the confirm. The symptom sounds reasonable since we don't include the limitSpec in the cache key at all. However, should the limitSpec be included only when `GroupByQuery.isApplyLimitPushDown() = true` then? When the limitSpec is processed in the post process step, the cache should be reusable for other queries if they are same except limitSpec similar to `HavingSpec`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson edited a comment on pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
jihoonson edited a comment on pull request #10093:
URL: https://github.com/apache/druid/pull/10093#issuecomment-654502620


   > @jihoonson : Yes, its an issue when limitSpec if pushed down, other thing to note, if you push down the limit once and then try to execute a query without pushing down the limit or remove the limit itself, then also it will be problematic as the cache will have the results with limit and the results will be served from cache.
   
   Thanks for the confirm. The symptom sounds reasonable since we currently don't include the limitSpec in the cache key at all. However, should the limitSpec be included only when `GroupByQuery.isApplyLimitPushDown() = true` then? When the limitSpec is processed in the post process step, the cache should be reusable for other queries if they are same except limitSpec similar to `HavingSpec`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] nishantmonu51 commented on pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
nishantmonu51 commented on pull request #10093:
URL: https://github.com/apache/druid/pull/10093#issuecomment-653539435


   @jihoonson : Yes, its an issue when limitSpec if pushed down, other thing to note, if you push down the limit once and then try to execute a query without pushing down the limit or remove the limit itself, then also it will be problematic as the cache will have the results with limit and the results will be served from cache. 
   
   Updated the PR description about the default value.  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson edited a comment on pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
jihoonson edited a comment on pull request #10093:
URL: https://github.com/apache/druid/pull/10093#issuecomment-654502620


   > @jihoonson : Yes, its an issue when limitSpec if pushed down, other thing to note, if you push down the limit once and then try to execute a query without pushing down the limit or remove the limit itself, then also it will be problematic as the cache will have the results with limit and the results will be served from cache.
   
   Thanks for the confirm. The symptom sounds reasonable since we currently don't include the limitSpec in the cache key at all. However, should the limitSpec be included only when `GroupByQuery.isApplyLimitPushDown() = true` then? When the limitSpec is processed in the post process step, the cache should be reusable for other queries if they are same except limitSpec, similar to `HavingSpec`. Or, am I missing something?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] nishantmonu51 commented on a change in pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
nishantmonu51 commented on a change in pull request #10093:
URL: https://github.com/apache/druid/pull/10093#discussion_r453640286



##########
File path: processing/src/test/java/org/apache/druid/query/groupby/GroupByQueryQueryToolChestTest.java
##########
@@ -929,6 +930,64 @@ private void doTestCacheStrategy(final ValueType valueType, final Object dimValu
     Assert.assertEquals(typeAdjustedResult2, fromResultCacheResult);
   }
 
+  @Test
+  public void testQueryCacheKeyWithLimitSpec()
+  {
+    final GroupByQuery query1 = GroupByQuery

Review comment:
       added test for this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] nishantmonu51 commented on a change in pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
nishantmonu51 commented on a change in pull request #10093:
URL: https://github.com/apache/druid/pull/10093#discussion_r450006711



##########
File path: processing/src/main/java/org/apache/druid/query/groupby/GroupByQueryQueryToolChest.java
##########
@@ -524,6 +524,7 @@ public boolean isCacheable(GroupByQuery query, boolean willMergeRunners)
             .appendCacheables(query.getAggregatorSpecs())
             .appendCacheables(query.getDimensions())
             .appendCacheable(query.getVirtualColumns())
+            .appendCacheable(query.getLimitSpec())

Review comment:
       havingSpec is used in post-processing step only, so it should not be required here 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson edited a comment on pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
jihoonson edited a comment on pull request #10093:
URL: https://github.com/apache/druid/pull/10093#issuecomment-655024790


   > @jihoonson: I can add a check for limitPushDown when generating cache key.
   > But wouldn't it be better for the underlying layers to not worry about whether limitSpec was pushed or not every time ?
   > The broker logic if it decides to not push down the limit, shouldn't it send a query with a null limitSpec to the underlying layers ? Or is there any reason for the broker to still push down limit spec ?
   
   @nishantmonu51 ah, I'm not sure exactly why the broker always sends the limitSpec, guessing it's for simplicity. Sending limitSpec only when pushing down limitSpec seems reasonable, although I don't mind much with the current implementation. I guess it's up to you. I can review the follow-up PR if you are going to do it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on a change in pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10093:
URL: https://github.com/apache/druid/pull/10093#discussion_r452385096



##########
File path: processing/src/test/java/org/apache/druid/query/groupby/GroupByQueryQueryToolChestTest.java
##########
@@ -929,6 +930,64 @@ private void doTestCacheStrategy(final ValueType valueType, final Object dimValu
     Assert.assertEquals(typeAdjustedResult2, fromResultCacheResult);
   }
 
+  @Test
+  public void testQueryCacheKeyWithLimitSpec()
+  {
+    final GroupByQuery query1 = GroupByQuery

Review comment:
       Oh sorry, I forgot to approve this PR. This comment is not a blocker if you want to do in a follow-up.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #10093:
URL: https://github.com/apache/druid/pull/10093#issuecomment-651283537


   Hi @nishantmonu51, does this bug happen only when `GroupByQuery.isApplyLimitPushDown()` = true? AFAIT, when `isApplyLimitPushDown()` = false, the `limitSpec` is applied after merging results from the cache (`GroupByQuery.postProcess()` is called after `GroupByStrategy.mergeResults()`) and thus the limitSpec shouldn't be a part of cache key.
   
   BTW, the cache has been enabled for groupBy by default since 0.13 (https://github.com/apache/druid/pull/5108).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] nishantmonu51 commented on pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
nishantmonu51 commented on pull request #10093:
URL: https://github.com/apache/druid/pull/10093#issuecomment-654636315


   @jihoonson: I can add a check for limitPushDown when generating cache key. 
   But wouldn't it be better for the underlying layers to not worry about whether limitSpec was pushed or not every time ? 
   The broker logic if it decides to not push down the limit, shouldn't it send a query with a null limitSpec to the underlying layers ? Or is there any reason for the broker to still push down limit spec ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson edited a comment on pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
jihoonson edited a comment on pull request #10093:
URL: https://github.com/apache/druid/pull/10093#issuecomment-654502620


   > @jihoonson : Yes, its an issue when limitSpec if pushed down, other thing to note, if you push down the limit once and then try to execute a query without pushing down the limit or remove the limit itself, then also it will be problematic as the cache will have the results with limit and the results will be served from cache.
   
   Thanks for the confirm. The symptom sounds reasonable since we currently don't include the limitSpec in the cache key at all. However, should the limitSpec be included only when `GroupByQuery.isApplyLimitPushDown() = true` then? When the limitSpec is processed in the post process step, the cache should be reusable for other queries if they are same except limitSpec, similar to `HavingSpec`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on a change in pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10093:
URL: https://github.com/apache/druid/pull/10093#discussion_r451039806



##########
File path: processing/src/test/java/org/apache/druid/query/groupby/GroupByQueryQueryToolChestTest.java
##########
@@ -929,6 +930,64 @@ private void doTestCacheStrategy(final ValueType valueType, final Object dimValu
     Assert.assertEquals(typeAdjustedResult2, fromResultCacheResult);
   }
 
+  @Test
+  public void testQueryCacheKeyWithLimitSpec()
+  {
+    final GroupByQuery query1 = GroupByQuery

Review comment:
       Would you please add another query which has the same spec and `CTX_KEY_APPLY_LIMIT_PUSH_DOWN` of true in the context, and check the equality of the cache key of the query and `query1`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] suneet-s commented on a change in pull request #10093: Add groupBy limitSpec to queryCache key

Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #10093:
URL: https://github.com/apache/druid/pull/10093#discussion_r447137245



##########
File path: processing/src/main/java/org/apache/druid/query/groupby/GroupByQueryQueryToolChest.java
##########
@@ -524,6 +524,7 @@ public boolean isCacheable(GroupByQuery query, boolean willMergeRunners)
             .appendCacheables(query.getAggregatorSpecs())
             .appendCacheables(query.getDimensions())
             .appendCacheable(query.getVirtualColumns())
+            .appendCacheable(query.getLimitSpec())

Review comment:
       Does this also need to consider the `havingSpec` similar to the ResultLevelCacheKey below?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org