You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "klsince (via GitHub)" <gi...@apache.org> on 2024/03/21 23:20:19 UTC

[PR] add SegmentContext to collect validDocIds bitmaps for many segments together [pinot]

klsince opened a new pull request, #12694:
URL: https://github.com/apache/pinot/pull/12694

   This PR adds SegmentContext used to collect segment related context before query execution. E.g. this makes it easier to collect validDocIds for many segments together, so that we can add the support for consistent table view for upsert table easier later on.
   
   This PR didn't change current functionalities but moved the logic of getting validDocId bitmaps upto one place, i.e. `tableDataMgr.getSegmentContexts(selectedSegments)`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] add SegmentContext to collect validDocIds bitmaps for many segments together [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince merged PR #12694:
URL: https://github.com/apache/pinot/pull/12694


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] add SegmentContext to collect validDocIds bitmaps for many segments together [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #12694:
URL: https://github.com/apache/pinot/pull/12694#issuecomment-2014067551

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12694?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `49.67742%` with `78 lines` in your changes are missing coverage. Please review.
   > Project coverage is 45.94%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`3248a42`)](https://app.codecov.io/gh/apache/pinot/pull/12694?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 154 commits behind head on master.
   
   | [Files](https://app.codecov.io/gh/apache/pinot/pull/12694?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...cal/upsert/BasePartitionUpsertMetadataManager.java](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91cHNlcnQvQmFzZVBhcnRpdGlvblVwc2VydE1ldGFkYXRhTWFuYWdlci5qYXZh) | 0.00% | [32 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...ava/org/apache/pinot/core/plan/FilterPlanNode.java](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0ZpbHRlclBsYW5Ob2RlLmphdmE=) | 52.94% | [4 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...a/org/apache/pinot/segment/spi/SegmentContext.java](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL1NlZ21lbnRDb250ZXh0LmphdmE=) | 0.00% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...pinot/core/plan/maker/InstancePlanMakerImplV2.java](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL21ha2VyL0luc3RhbmNlUGxhbk1ha2VySW1wbFYyLmphdmE=) | 66.66% | [2 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...va/org/apache/pinot/core/plan/maker/PlanMaker.java](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL21ha2VyL1BsYW5NYWtlci5qYXZh) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...psert/ConcurrentMapTableUpsertMetadataManager.java](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91cHNlcnQvQ29uY3VycmVudE1hcFRhYmxlVXBzZXJ0TWV0YWRhdGFNYW5hZ2VyLmphdmE=) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...rg/apache/pinot/core/plan/AggregationPlanNode.java](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uUGxhbk5vZGUuamF2YQ==) | 75.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...a/org/apache/pinot/core/plan/DistinctPlanNode.java](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0Rpc3RpbmN0UGxhbk5vZGUuamF2YQ==) | 60.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...a/org/apache/pinot/core/plan/DocIdSetPlanNode.java](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0RvY0lkU2V0UGxhbk5vZGUuamF2YQ==) | 66.66% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...va/org/apache/pinot/core/plan/ProjectPlanNode.java](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL1Byb2plY3RQbGFuTm9kZS5qYXZh) | 75.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | ... and [8 more](https://app.codecov.io/gh/apache/pinot/pull/12694?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #12694       +/-   ##
   =============================================
   - Coverage     61.75%   45.94%   -15.81%     
   - Complexity      207      908      +701     
   =============================================
     Files          2436     1854      -582     
     Lines        133233    97876    -35357     
     Branches      20636    15797     -4839     
   =============================================
   - Hits          82274    44970    -37304     
   - Misses        44911    49580     +4669     
   + Partials       6048     3326     -2722     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12694/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12694/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/12694/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12694/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12694/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12694/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12694/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.94% <49.67%> (-15.68%)` | :arrow_down: |
   | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12694/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12694/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.94% <49.67%> (+18.21%)` | :arrow_up: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/12694/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.94% <49.67%> (-15.81%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/12694/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.94% <49.67%> (-15.81%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12694/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.94% <49.67%> (-0.95%)` | :arrow_down: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12694/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12694?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] add SegmentContext to collect validDocIds bitmaps for many segments together [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #12694:
URL: https://github.com/apache/pinot/pull/12694#discussion_r1554386583


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/TableUpsertMetadataManager.java:
##########
@@ -50,4 +53,6 @@ public interface TableUpsertMetadataManager extends Closeable {
    * @return A {@code Map} where keys are partition id and values are count of primary keys for that specific partition
    */
   Map<Integer, Long> getPartitionToPrimaryKeyCount();
+
+  List<SegmentContext> getSegmentContexts(List<IndexSegment> selectedSegments);

Review Comment:
   good point



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] add SegmentContext to collect validDocIds bitmaps for many segments together [pinot]

Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on code in PR #12694:
URL: https://github.com/apache/pinot/pull/12694#discussion_r1536170709


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/ConcurrentMapTableUpsertMetadataManager.java:
##########
@@ -55,6 +58,14 @@ public Map<Integer, Long> getPartitionToPrimaryKeyCount() {
     return partitionToPrimaryKeyCount;
   }
 
+  @Override
+  public Map<IndexSegment, SegmentContext> getSegmentContexts(List<IndexSegment> selectedSegments) {
+    Map<IndexSegment, SegmentContext> segmentContexts = new HashMap<>();
+    _partitionMetadataManagerMap.forEach(
+        (k, v) -> v.getUpsertView(selectedSegments, segmentContexts));
+    return segmentContexts;
+  }

Review Comment:
   pretty neat! 



##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -256,8 +257,9 @@ private InstanceResponseBlock executeInternal(ServerQueryRequest queryRequest, E
 
     InstanceResponseBlock instanceResponse = null;
     try {
-      instanceResponse = executeInternal(indexSegments, queryContext, timerContext, executorService, streamer,
-          queryRequest.isEnableStreaming());
+      instanceResponse =
+          executeInternal(tableDataManager, indexSegments, queryContext, timerContext, executorService, streamer,
+              queryRequest.isEnableStreaming());

Review Comment:
   nit: can we pass List<SegmentContext> itself instead of tableDataManager?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] add SegmentContext to collect validDocIds bitmaps for many segments together [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #12694:
URL: https://github.com/apache/pinot/pull/12694#discussion_r1536258715


##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -256,8 +257,9 @@ private InstanceResponseBlock executeInternal(ServerQueryRequest queryRequest, E
 
     InstanceResponseBlock instanceResponse = null;
     try {
-      instanceResponse = executeInternal(indexSegments, queryContext, timerContext, executorService, streamer,
-          queryRequest.isEnableStreaming());
+      instanceResponse =
+          executeInternal(tableDataManager, indexSegments, queryContext, timerContext, executorService, streamer,
+              queryRequest.isEnableStreaming());

Review Comment:
   inside executeInternal() the list of indexSegments is pruned firstly, so passing tableDataManager into executeInternal() to call the getSegmentContexts() method after pruning just for the remaining indexSegments.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] add SegmentContext to collect validDocIds bitmaps for many segments together [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12694:
URL: https://github.com/apache/pinot/pull/12694#discussion_r1556437699


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -433,6 +435,14 @@ public Map<String, SegmentErrorInfo> getSegmentErrors() {
     }
   }
 
+  @Override
+  public List<SegmentContext> getSegmentContexts(List<IndexSegment> selectedSegments,
+      Map<String, String> queryOptions) {
+    List<SegmentContext> segmentContexts = new ArrayList<>();

Review Comment:
   (minor) 
   ```suggestion
       List<SegmentContext> segmentContexts = new ArrayList<>(selectedSegments.size());
   ```



##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java:
##########
@@ -300,6 +303,17 @@ public void onConsumingToOnline(String segmentNameStr) {
     _ingestionDelayTracker.markPartitionForVerification(segmentName.getPartitionGroupId());
   }
 
+  @Override
+  public List<SegmentContext> getSegmentContexts(List<IndexSegment> selectedSegments,
+      Map<String, String> queryOptions) {
+    List<SegmentContext> segmentContexts = new ArrayList<>();

Review Comment:
   (minor)
   ```suggestion
       List<SegmentContext> segmentContexts = new ArrayList<>(selectedSegments.size());
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] add SegmentContext to collect validDocIds bitmaps for many segments together [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12694:
URL: https://github.com/apache/pinot/pull/12694#discussion_r1554332240


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/SegmentContext.java:
##########
@@ -0,0 +1,58 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi;
+
+import org.apache.pinot.segment.spi.index.mutable.ThreadSafeMutableRoaringBitmap;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
+
+
+public class SegmentContext {
+  private final IndexSegment _indexSegment;
+  private MutableRoaringBitmap _queryableDocIdsSnapshot = null;
+
+  public SegmentContext(IndexSegment indexSegment) {
+    _indexSegment = indexSegment;
+  }
+
+  public IndexSegment getIndexSegment() {
+    return _indexSegment;
+  }
+
+  public MutableRoaringBitmap getQueryableDocIdsSnapshot() {
+    return _queryableDocIdsSnapshot;
+  }
+
+  public void setQueryableDocIdsSnapshot(MutableRoaringBitmap queryableDocIdsSnapshot) {
+    _queryableDocIdsSnapshot = queryableDocIdsSnapshot;
+  }
+
+  public static MutableRoaringBitmap getQueryableDocIdsSnapshotFromSegment(IndexSegment segment) {

Review Comment:
   This method should be maintained outside of the context. This is upsert specific logic, so it should be managed within the upsert metadata manager



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/TableUpsertMetadataManager.java:
##########
@@ -50,4 +53,6 @@ public interface TableUpsertMetadataManager extends Closeable {
    * @return A {@code Map} where keys are partition id and values are count of primary keys for that specific partition
    */
   Map<Integer, Long> getPartitionToPrimaryKeyCount();
+
+  List<SegmentContext> getSegmentContexts(List<IndexSegment> selectedSegments);

Review Comment:
   Should we pass in `List<SegmentContext>` and return `List<SegmentContext>`? Currently upsert is the only component decorating segment, but we might add more in the future



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org