You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2023/09/13 19:15:15 UTC

[GitHub] [pinot] Jackie-Jiang opened a new pull request, #11585: Enhance SegmentPartitionMetadataManager to handle new segment

Jackie-Jiang opened a new pull request, #11585:
URL: https://github.com/apache/pinot/pull/11585

   Similar to #10466, we need to be more graceful for new segment (newly pushed or new consuming segment) because it is common that they don't have all replicas available in external view.
   This PR adds the new segment into the partition info only when all the replicas are available


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang merged pull request #11585: Enhance SegmentPartitionMetadataManager to handle new segment

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #11585:
URL: https://github.com/apache/pinot/pull/11585


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #11585: Enhance SegmentPartitionMetadataManager to handle new segment

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11585:
URL: https://github.com/apache/pinot/pull/11585#issuecomment-1718256574

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11585?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#11585](https://app.codecov.io/gh/apache/pinot/pull/11585?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (dee7410) into [master](https://app.codecov.io/gh/apache/pinot/commit/011c072196bc7959155a34f4ae9877abd676669f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (011c072) will **decrease** coverage by `0.01%`.
   > The diff coverage is `63.33%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #11585      +/-   ##
   ============================================
   - Coverage     63.04%   63.04%   -0.01%     
     Complexity     1108     1108              
   ============================================
     Files          2326     2326              
     Lines        124882   124926      +44     
     Branches      19065    19075      +10     
   ============================================
   + Hits          78731    78757      +26     
   - Misses        40544    40564      +20     
   + Partials       5607     5605       -2     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/11585/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/11585/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (ø)` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/11585/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (ø)` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/11585/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/11585/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.00% <63.33%> (-0.01%)` | :arrow_down: |
   | [java-17](https://app.codecov.io/gh/apache/pinot/pull/11585/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.89% <63.33%> (+<0.01%)` | :arrow_up: |
   | [java-20](https://app.codecov.io/gh/apache/pinot/pull/11585/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.91% <63.33%> (+<0.01%)` | :arrow_up: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/11585/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.04% <63.33%> (-0.01%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/11585/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.03% <63.33%> (-0.01%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11585/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `67.44% <100.00%> (ø)` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11585/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.50% <61.66%> (+0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files Changed](https://app.codecov.io/gh/apache/pinot/pull/11585?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...mentpartition/SegmentPartitionMetadataManager.java](https://app.codecov.io/gh/apache/pinot/pull/11585?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50cGFydGl0aW9uL1NlZ21lbnRQYXJ0aXRpb25NZXRhZGF0YU1hbmFnZXIuamF2YQ==) | `73.61% <62.71%> (-2.39%)` | :arrow_down: |
   | [.../apache/pinot/core/routing/TablePartitionInfo.java](https://app.codecov.io/gh/apache/pinot/pull/11585?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yb3V0aW5nL1RhYmxlUGFydGl0aW9uSW5mby5qYXZh) | `88.88% <100.00%> (ø)` | |
   
   ... and [8 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11585/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11585: Enhance SegmentPartitionMetadataManager to handle new segment

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11585:
URL: https://github.com/apache/pinot/pull/11585#discussion_r1325042227


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpartition/SegmentPartitionMetadataManager.java:
##########
@@ -123,16 +141,23 @@ private List<String> getOnlineServers(ExternalView externalView, String segment)
 
   private void computeTablePartitionInfo() {
     PartitionInfo[] partitionInfoMap = new PartitionInfo[_numPartitions];
-    Set<String> segmentsWithInvalidPartition = new HashSet<>();
+    List<String> segmentsWithInvalidPartition = new ArrayList<>();
+    List<Map.Entry<String, SegmentInfo>> newSegmentInfoEntries = new ArrayList<>();
+    long currentTimeMs = System.currentTimeMillis();
     for (Map.Entry<String, SegmentInfo> entry : _segmentInfoMap.entrySet()) {
       String segment = entry.getKey();
       SegmentInfo segmentInfo = entry.getValue();
       int partitionId = segmentInfo._partitionId;
-      List<String> onlineServers = segmentInfo._onlineServers;
       if (partitionId == INVALID_PARTITION_ID) {
         segmentsWithInvalidPartition.add(segment);
         continue;
       }
+      // Process new segments in the end
+      if (InstanceSelector.isNewSegment(segmentInfo._pushTimeMs, currentTimeMs)) {

Review Comment:
   New segment should have valid partition id. Segment ZK metadata won't be changed for new segment within the life cycle



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] walterddr commented on a diff in pull request #11585: Enhance SegmentPartitionMetadataManager to handle new segment

Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #11585:
URL: https://github.com/apache/pinot/pull/11585#discussion_r1325034420


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpartition/SegmentPartitionMetadataManager.java:
##########
@@ -151,8 +176,48 @@ private void computeTablePartitionInfo() {
         LOGGER.warn("Found {} segments: {} with invalid partition from table: {}", numSegmentsWithInvalidPartition,
             segmentsWithInvalidPartition, _tableNameWithType);
       } else {
-        LOGGER.warn("Found {} segments: {} with invalid partition from table: {}", numSegmentsWithInvalidPartition,
-            segmentsWithInvalidPartition, _tableNameWithType);
+        LOGGER.warn("Found {} segments: {}... with invalid partition from table: {}", numSegmentsWithInvalidPartition,
+            segmentsWithInvalidPartition.subList(0, 10), _tableNameWithType);
+      }
+    }
+    // Process new segments
+    if (!newSegmentInfoEntries.isEmpty()) {
+      List<String> excludedNewSegments = new ArrayList<>();
+      for (Map.Entry<String, SegmentInfo> entry : newSegmentInfoEntries) {
+        String segment = entry.getKey();
+        SegmentInfo segmentInfo = entry.getValue();
+        int partitionId = segmentInfo._partitionId;
+        List<String> onlineServers = segmentInfo._onlineServers;
+        PartitionInfo partitionInfo = partitionInfoMap[partitionId];
+        if (partitionInfo == null) {
+          // If the new segment is the first segment of a partition, treat it as regular segment
+          Set<String> fullyReplicatedServers = new HashSet<>(onlineServers);
+          List<String> segments = new ArrayList<>();
+          segments.add(segment);
+          partitionInfo = new PartitionInfo(fullyReplicatedServers, segments);
+          partitionInfoMap[partitionId] = partitionInfo;
+        } else {
+          // If the new segment is not the first segment of a partition, add it only if it won't reduce the fully
+          // replicated servers. It is common that a new segment (newly pushed, or a new consuming segment) doesn't have
+          // all the replicas available yet, and we want to exclude it from the partition info until all the replicas
+          // are available.
+          //noinspection SlowListContainsAll

Review Comment:
   this means any request with partitioned computation will miss a segment until it is fully replicated? what if a commit time of a consuming segment is on average 1hr and in that case:
   1. we wont be able to access consuming segment data until 1hr later when it commits + fully replicated to all replicas
   2. we will be able to access consuming segment as long as all replica servers starts consuming (but results will be inconsistent depending which partition replica we dispatch the request to
   
   which of the above is true?



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpartition/SegmentPartitionMetadataManager.java:
##########
@@ -123,16 +141,23 @@ private List<String> getOnlineServers(ExternalView externalView, String segment)
 
   private void computeTablePartitionInfo() {
     PartitionInfo[] partitionInfoMap = new PartitionInfo[_numPartitions];
-    Set<String> segmentsWithInvalidPartition = new HashSet<>();
+    List<String> segmentsWithInvalidPartition = new ArrayList<>();
+    List<Map.Entry<String, SegmentInfo>> newSegmentInfoEntries = new ArrayList<>();
+    long currentTimeMs = System.currentTimeMillis();
     for (Map.Entry<String, SegmentInfo> entry : _segmentInfoMap.entrySet()) {
       String segment = entry.getKey();
       SegmentInfo segmentInfo = entry.getValue();
       int partitionId = segmentInfo._partitionId;
-      List<String> onlineServers = segmentInfo._onlineServers;
       if (partitionId == INVALID_PARTITION_ID) {
         segmentsWithInvalidPartition.add(segment);
         continue;
       }
+      // Process new segments in the end
+      if (InstanceSelector.isNewSegment(segmentInfo._pushTimeMs, currentTimeMs)) {

Review Comment:
   is it possible that a new segment is with invalid partitionID? should we move this above the invalid id check?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11585: Enhance SegmentPartitionMetadataManager to handle new segment

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11585:
URL: https://github.com/apache/pinot/pull/11585#discussion_r1325041098


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpartition/SegmentPartitionMetadataManager.java:
##########
@@ -151,8 +176,48 @@ private void computeTablePartitionInfo() {
         LOGGER.warn("Found {} segments: {} with invalid partition from table: {}", numSegmentsWithInvalidPartition,
             segmentsWithInvalidPartition, _tableNameWithType);
       } else {
-        LOGGER.warn("Found {} segments: {} with invalid partition from table: {}", numSegmentsWithInvalidPartition,
-            segmentsWithInvalidPartition, _tableNameWithType);
+        LOGGER.warn("Found {} segments: {}... with invalid partition from table: {}", numSegmentsWithInvalidPartition,
+            segmentsWithInvalidPartition.subList(0, 10), _tableNameWithType);
+      }
+    }
+    // Process new segments
+    if (!newSegmentInfoEntries.isEmpty()) {
+      List<String> excludedNewSegments = new ArrayList<>();
+      for (Map.Entry<String, SegmentInfo> entry : newSegmentInfoEntries) {
+        String segment = entry.getKey();
+        SegmentInfo segmentInfo = entry.getValue();
+        int partitionId = segmentInfo._partitionId;
+        List<String> onlineServers = segmentInfo._onlineServers;
+        PartitionInfo partitionInfo = partitionInfoMap[partitionId];
+        if (partitionInfo == null) {
+          // If the new segment is the first segment of a partition, treat it as regular segment
+          Set<String> fullyReplicatedServers = new HashSet<>(onlineServers);
+          List<String> segments = new ArrayList<>();
+          segments.add(segment);
+          partitionInfo = new PartitionInfo(fullyReplicatedServers, segments);
+          partitionInfoMap[partitionId] = partitionInfo;
+        } else {
+          // If the new segment is not the first segment of a partition, add it only if it won't reduce the fully
+          // replicated servers. It is common that a new segment (newly pushed, or a new consuming segment) doesn't have
+          // all the replicas available yet, and we want to exclude it from the partition info until all the replicas
+          // are available.
+          //noinspection SlowListContainsAll

Review Comment:
   Segment is count as new for up to 5 minutes after pushed/created, so committing segment won't be counted as new segment. New consuming segment is created after the previous consuming segment is committed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org