You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/12/29 10:22:12 UTC

[GitHub] [druid] kaijianding opened a new pull request #10714: use actual dataInterval in cache key

kaijianding opened a new pull request #10714:
URL: https://github.com/apache/druid/pull/10714


   ### Description
   
   Currently, the interval used in CachingQueryRunner by CacheUtil.computeSegmentCacheKey() is the intersection of query interval and Segment interval from segment identifier.
   It's better to use the intersection of query interval and the actual dataInterval of the segment(the min/max data time in the segment, not the interval in the segment identifier).
   
   For example, when the segment granularity is DAY and there are many partitions built by realtime tasks :
   datasource_name1_2020-12-28T00:00:00.000Z_2020-12-29T00:00:00.000Z_2020-12-28T00:00:00.505Z: 
   datasource_name1_2020-12-28T00:00:00.000Z_2020-12-29T00:00:00.000Z_2020-12-28T00:00:00.505Z_1: 
   datasource_name1_2020-12-28T00:00:00.000Z_2020-12-29T00:00:00.000Z_2020-12-28T00:00:00.505Z_2: 
   
   The actual min/max time in these segments are:
   00:09:00 ~ 00:37:22
   01:11:34 ~ 01:45:43
   02:04:22 ~ 02:54:12
   
   Currently, the first query with query interval 00:00:00/02:14:19 populates a cache with interval 00:00:00/02:14:19 in cache key.
   the next query with query interval 00:00:00/02:15:19 can't reuse the cache populated by the first query due to the '00:00:00/02:14:19' used in the cache key.
   
   For this case, the interval used in cache key should be changed to 00:09:00/00:37:22, 01:11:34/01:45:43, 02:04:22/02:14:19
   
   This change will not break any contract, because of `Interval actualInterval = interval.overlap(dataInterval);` in `makeCursors()` in `IncrementalIndexStorageAdapter` and `QueryableIndexStorageAdapter`
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
      - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [x] been tested in a test Druid cluster.
   
   <hr>
   
   ##### Key changed/added classes in this PR
    * `CachingQueryRunner`
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a change in pull request #10714: use actual dataInterval in cache key

Posted by GitBox <gi...@apache.org>.
gianm commented on a change in pull request #10714:
URL: https://github.com/apache/druid/pull/10714#discussion_r553755667



##########
File path: server/src/main/java/org/apache/druid/segment/realtime/appenderator/SinkQuerySegmentWalker.java
##########
@@ -230,10 +232,15 @@ public SinkQuerySegmentWalker(
                       // 1) Only use caching if data is immutable
                       // 2) Hydrants are not the same between replicas, make sure cache is local
                       if (hydrantDefinitelySwapped && cache.isLocal()) {
+                        StorageAdapter storageAdapter = segmentAndCloseable.lhs.asStorageAdapter();
+                        long segmentMinTime = storageAdapter.getMinTime().getMillis();
+                        long segmentMaxTime = storageAdapter.getMaxTime().getMillis();
+                        Interval actualDataInterval = Intervals.utc(segmentMinTime, segmentMaxTime);

Review comment:
       I think there's an off by one here: `storageAdapter.getMaxTime()` is the highest timestamp in the segment, but the end of `actualDataInterval` is exclusive, and so it doesn't include that highest timestamp. I think doing `segmentMaxTime.plus(1)` will fix it. There's a similar issue in ServerManager.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm merged pull request #10714: use actual dataInterval in cache key

Posted by GitBox <gi...@apache.org>.
gianm merged pull request #10714:
URL: https://github.com/apache/druid/pull/10714


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] kaijianding commented on a change in pull request #10714: use actual dataInterval in cache key

Posted by GitBox <gi...@apache.org>.
kaijianding commented on a change in pull request #10714:
URL: https://github.com/apache/druid/pull/10714#discussion_r556362731



##########
File path: server/src/main/java/org/apache/druid/segment/realtime/appenderator/SinkQuerySegmentWalker.java
##########
@@ -230,10 +232,15 @@ public SinkQuerySegmentWalker(
                       // 1) Only use caching if data is immutable
                       // 2) Hydrants are not the same between replicas, make sure cache is local
                       if (hydrantDefinitelySwapped && cache.isLocal()) {
+                        StorageAdapter storageAdapter = segmentAndCloseable.lhs.asStorageAdapter();
+                        long segmentMinTime = storageAdapter.getMinTime().getMillis();
+                        long segmentMaxTime = storageAdapter.getMaxTime().getMillis();
+                        Interval actualDataInterval = Intervals.utc(segmentMinTime, segmentMaxTime);

Review comment:
       yes, you are right, will fix it




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] pjain1 commented on pull request #10714: use actual dataInterval in cache key

Posted by GitBox <gi...@apache.org>.
pjain1 commented on pull request #10714:
URL: https://github.com/apache/druid/pull/10714#issuecomment-950261845


   This introduced a bug - https://github.com/apache/druid/issues/11836


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org