You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "deepthi912 (via GitHub)" <gi...@apache.org> on 2025/03/04 06:45:29 UTC

[PR] apache#15162 - Bug Fix Segments going into BAD state for Dedup Tables with metadataTTL [pinot]

deepthi912 opened a new pull request, #15178:
URL: https://github.com/apache/pinot/pull/15178

   **Context:**
   https://apache-pinot.slack.com/archives/C011C9JHN7R/p1740757158048619
   Noticed segments going into BAD state when metadataTTL is set for dedup tables and Replication > 1
   
   `[RealtimeSegmentDataManager_leaderboard_entries__0__1__20250228T1528Z] [leaderboard_entries__0__1__20250228T1528Z] Caught exception in consumer thread after stop() is invoked: java.lang.RuntimeException: Caught exception while removing segment: leaderboard_entries__0__1__20250228T1528Z of table: leaderboard_entries_REALTIME from ConcurrentMapPartitionDedupMetadataManager, ignoring the exception`
   
   **Root Cause:**
   segment.getSegmentMetadata().getColumnMetadataMap() is null when offloading oldSegmentManager for which segmentMetadata is like below:
   
   `{"segmentName":"githubEvents__1__5__20250303T1928Z","schemaName":"githubEvents","crc":-9223372036854775808,"creationTimeMillis":1741030112009,"creationTimeReadable":"2025-03-03T19:28:32:009 UTC","timeColumn":null,"timeUnit":null,"timeGranularitySec":null,"startTimeMillis":null,"startTimeReadable":null,"endTimeMillis":null,"endTimeReadable":null,"segmentVersion":null,"creatorName":null,"totalDocs":0,"custom":{},"startOffset":null,"endOffset":null} `
   
   **Before Fix:**
   ![image](https://github.com/user-attachments/assets/51cc61a7-51c4-4ab3-921d-7db1b43a165e)
   
   **After Fix:**
   ![image](https://github.com/user-attachments/assets/081a3d8e-a89d-4a06-86f3-ee2449c44cae)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] apache#15162 - Bug Fix Segments going into BAD state for Dedup Tables with metadataTTL [pinot]

Posted by "deepthi912 (via GitHub)" <gi...@apache.org>.
deepthi912 commented on code in PR #15178:
URL: https://github.com/apache/pinot/pull/15178#discussion_r1982241163


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/BasePartitionDedupMetadataManager.java:
##########
@@ -283,8 +283,13 @@ public void removeSegment(IndexSegment segment) {
       return;
     }
     try {
-      if (skipSegmentOutOfTTL(segment, false)) {
-        return;
+      // segment.getSegmentMetadata().getColumnMetadataMap() will be null when offloading a temporary segment data
+      // manager created in BaseTableDataManager.addSegment which caused replicas to go into BAD state when
+      // metadataTTL is set
+      if (segment.getSegmentMetadata() != null && segment.getSegmentMetadata().getColumnMetadataMap() != null) {

Review Comment:
   Should we eliminate the private final Logger _segmentLogger; all together from this class? I think the e.String() is logging the message right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] apache#15162 - Bug Fix Segments going into BAD state for Dedup Tables with metadataTTL [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #15178:
URL: https://github.com/apache/pinot/pull/15178#issuecomment-2696469308

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15178?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `0%` with `3 lines` in your changes missing coverage. Please review.
   > Project coverage is 63.37%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`a8f2f87`)](https://app.codecov.io/gh/apache/pinot/commit/a8f2f879aeb488f8f7c067d87b94f35a814035f7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 1790 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/15178?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...local/dedup/BasePartitionDedupMetadataManager.java](https://app.codecov.io/gh/apache/pinot/pull/15178?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fdedup%2FBasePartitionDedupMetadataManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9kZWR1cC9CYXNlUGFydGl0aW9uRGVkdXBNZXRhZGF0YU1hbmFnZXIuamF2YQ==) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15178?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #15178      +/-   ##
   ============================================
   + Coverage     61.75%   63.37%   +1.61%     
   - Complexity      207     1363    +1156     
   ============================================
     Files          2436     2759     +323     
     Lines        133233   155114   +21881     
     Branches      20636    23905    +3269     
   ============================================
   + Hits          82274    98298   +16024     
   - Misses        44911    49384    +4473     
   - Partials       6048     7432    +1384     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/15178/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/15178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/15178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/15178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/15178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/15178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/15178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.37% <0.00%> (+1.74%)` | :arrow_up: |
   | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/15178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.35% <0.00%> (+1.61%)` | :arrow_up: |
   | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/15178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.35% <0.00%> (+35.62%)` | :arrow_up: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/15178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.37% <0.00%> (+1.61%)` | :arrow_up: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/15178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.37% <0.00%> (+1.62%)` | :arrow_up: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/15178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.99% <0.00%> (+9.10%)` | :arrow_up: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/15178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.95% <0.00%> (+6.22%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/15178?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] apache#15162 - Bug Fix Segments going into BAD state for Dedup Tables with metadataTTL [pinot]

Posted by "deepthi912 (via GitHub)" <gi...@apache.org>.
deepthi912 commented on code in PR #15178:
URL: https://github.com/apache/pinot/pull/15178#discussion_r1982295648


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -892,8 +892,13 @@ public void run() {
         }
       } catch (Throwable e) {
         if (_shouldStop) {
-          _segmentLogger.info("Caught exception in consumer thread after stop() is invoked: {}, ignoring the exception",
-              e.toString());
+          if (_segmentLogger.isDebugEnabled()) {
+            e.printStackTrace();

Review Comment:
   Sorry, got misunderstood with printing stack trace. Updated!



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/BasePartitionDedupMetadataManager.java:
##########
@@ -283,8 +283,13 @@ public void removeSegment(IndexSegment segment) {
       return;
     }
     try {
-      if (skipSegmentOutOfTTL(segment, false)) {
-        return;
+      // segment.getSegmentMetadata().getColumnMetadataMap() will be null when offloading a temporary segment data
+      // manager created in BaseTableDataManager.addSegment which caused replicas to go into BAD state when
+      // metadataTTL is set
+      if (segment.getSegmentMetadata() != null && segment.getSegmentMetadata().getColumnMetadataMap() != null) {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] apache#15162 - Bug Fix Segments going into BAD state for Dedup Tables with metadataTTL [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #15178:
URL: https://github.com/apache/pinot/pull/15178#discussion_r1982270429


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -892,8 +892,13 @@ public void run() {
         }
       } catch (Throwable e) {
         if (_shouldStop) {
-          _segmentLogger.info("Caught exception in consumer thread after stop() is invoked: {}, ignoring the exception",
-              e.toString());
+          if (_segmentLogger.isDebugEnabled()) {
+            e.printStackTrace();

Review Comment:
   use _segmentLogger.debug(...) as printStackTrace is writing to stdout/err



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] apache#15162 - Bug Fix Segments going into BAD state for Dedup Tables with metadataTTL [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #15178:
URL: https://github.com/apache/pinot/pull/15178#discussion_r1982131547


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/BasePartitionDedupMetadataManager.java:
##########
@@ -283,8 +283,13 @@ public void removeSegment(IndexSegment segment) {
       return;
     }
     try {
-      if (skipSegmentOutOfTTL(segment, false)) {
-        return;
+      // segment.getSegmentMetadata().getColumnMetadataMap() will be null when offloading a temporary segment data
+      // manager created in BaseTableDataManager.addSegment which caused replicas to go into BAD state when
+      // metadataTTL is set
+      if (segment.getSegmentMetadata() != null && segment.getSegmentMetadata().getColumnMetadataMap() != null) {

Review Comment:
   as we figured out that MutableSegment returns null for getColumnMetadataMap(), as it doesn't have column metadata map, we may consider to extend getMaxDedupTime() method instead of here, so caller of getMaxDedupTime() can leverage the check as well.
   ```
   protected double getMaxDedupTime(IndexSegment segment) {
       if (segment instanceof MutableSegment) {
         // MutableSegment doesn't have columnMetadataMap to get its max dedup time, so returning the largest value seen so far to process this segment as within TTL to be conservative
         return _largestSeenTime.get();
       }
       return ((Number) segment.getSegmentMetadata().getColumnMetadataMap().get(_dedupTimeColumn)
           .getMaxValue()).doubleValue();
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] apache#15162 - Bug Fix Segments going into BAD state for Dedup Tables with metadataTTL [pinot]

Posted by "deepthi912 (via GitHub)" <gi...@apache.org>.
deepthi912 commented on code in PR #15178:
URL: https://github.com/apache/pinot/pull/15178#discussion_r1982241163


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/BasePartitionDedupMetadataManager.java:
##########
@@ -283,8 +283,13 @@ public void removeSegment(IndexSegment segment) {
       return;
     }
     try {
-      if (skipSegmentOutOfTTL(segment, false)) {
-        return;
+      // segment.getSegmentMetadata().getColumnMetadataMap() will be null when offloading a temporary segment data
+      // manager created in BaseTableDataManager.addSegment which caused replicas to go into BAD state when
+      // metadataTTL is set
+      if (segment.getSegmentMetadata() != null && segment.getSegmentMetadata().getColumnMetadataMap() != null) {

Review Comment:
   Should we eliminate the private final Logger _segmentLogger; all together from this class and use the regular way? I think the e.String() is logging the message right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] Bug Fix Segments going into BAD state for Dedup Tables using TTL and RF > 1 [pinot]

Posted by "deepthi912 (via GitHub)" <gi...@apache.org>.
deepthi912 commented on code in PR #15178:
URL: https://github.com/apache/pinot/pull/15178#discussion_r1983963123


##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/DedupIntegrationTest.java:
##########
@@ -59,8 +72,41 @@ public void setUp()
     addSchema(schema);
     TableConfig tableConfig = createDedupTableConfig(_avroFiles.get(0), "id", getNumKafkaPartitions());
     addTableConfig(tableConfig);
-
     waitForAllDocsLoaded(600_000L);
+    createDedupConfigsWithReplicas();
+  }
+
+  public void createDedupConfigsWithReplicas()
+      throws IOException {
+    Schema schemaWithReplicas = createSchema(getSchemaFileName());
+    schemaWithReplicas.setSchemaName(DEDUP_WITH_REPLICAS_SCHEMA);
+    addSchema(schemaWithReplicas);
+    TableConfig tableConfigWithReplication =
+        createDedupTableConfigWithReplication(_avroFiles.get(0), "id", getNumKafkaPartitions());
+    addTableConfig(tableConfigWithReplication);
+    waitForDocsLoaded(600_000L, true, DEDUP_TABLE_WITH_REPLICAS);
+  }
+
+  /**
+   * Creates a new Dedup enabled table config with replication=2 and metadatTTL=30
+   */
+  protected TableConfig createDedupTableConfigWithReplication(File sampleAvroFile, String primaryKeyColumn,

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] Bug Fix Segments going into BAD state for Dedup Tables using TTL and RF > 1 [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince merged PR #15178:
URL: https://github.com/apache/pinot/pull/15178


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] apache#15162 - Bug Fix Segments going into BAD state for Dedup Tables with metadataTTL [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #15178:
URL: https://github.com/apache/pinot/pull/15178#discussion_r1982207064


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/BasePartitionDedupMetadataManager.java:
##########
@@ -283,8 +283,13 @@ public void removeSegment(IndexSegment segment) {
       return;
     }
     try {
-      if (skipSegmentOutOfTTL(segment, false)) {
-        return;
+      // segment.getSegmentMetadata().getColumnMetadataMap() will be null when offloading a temporary segment data
+      // manager created in BaseTableDataManager.addSegment which caused replicas to go into BAD state when
+      // metadataTTL is set
+      if (segment.getSegmentMetadata() != null && segment.getSegmentMetadata().getColumnMetadataMap() != null) {

Review Comment:
   Also, it may help debug such issues in future to log more about the exceptions at this place in RealtimeSegmentDataManager: L895.
   ```      } catch (Throwable e) {
           if (_shouldStop) {
             _segmentLogger.info("Caught exception in consumer thread after stop() is invoked: {}, ignoring the exception",
                 e.toString());
   ...
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] Bug Fix Segments going into BAD state for Dedup Tables using TTL and RF > 1 [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #15178:
URL: https://github.com/apache/pinot/pull/15178#discussion_r1983808352


##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/DedupIntegrationTest.java:
##########
@@ -59,8 +72,41 @@ public void setUp()
     addSchema(schema);
     TableConfig tableConfig = createDedupTableConfig(_avroFiles.get(0), "id", getNumKafkaPartitions());
     addTableConfig(tableConfig);
-
     waitForAllDocsLoaded(600_000L);
+    createDedupConfigsWithReplicas();
+  }
+
+  public void createDedupConfigsWithReplicas()
+      throws IOException {
+    Schema schemaWithReplicas = createSchema(getSchemaFileName());
+    schemaWithReplicas.setSchemaName(DEDUP_WITH_REPLICAS_SCHEMA);
+    addSchema(schemaWithReplicas);
+    TableConfig tableConfigWithReplication =
+        createDedupTableConfigWithReplication(_avroFiles.get(0), "id", getNumKafkaPartitions());
+    addTableConfig(tableConfigWithReplication);
+    waitForDocsLoaded(600_000L, true, DEDUP_TABLE_WITH_REPLICAS);
+  }
+
+  /**
+   * Creates a new Dedup enabled table config with replication=2 and metadatTTL=30
+   */
+  protected TableConfig createDedupTableConfigWithReplication(File sampleAvroFile, String primaryKeyColumn,

Review Comment:
   nit: ...WithReplicas to be consistent



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] apache#15162 - Bug Fix Segments going into BAD state for Dedup Tables with metadataTTL [pinot]

Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #15178:
URL: https://github.com/apache/pinot/pull/15178#discussion_r1980138473


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/BasePartitionDedupMetadataManager.java:
##########
@@ -283,8 +283,13 @@ public void removeSegment(IndexSegment segment) {
       return;
     }
     try {
-      if (skipSegmentOutOfTTL(segment, false)) {
-        return;
+      // segment.getSegmentMetadata().getColumnMetadataMap() will be null when offloading a temporary segment data
+      // manager created in BaseTableDataManager.addSegment which caused replicas to go into BAD state when
+      // metadataTTL is set
+      if (segment.getSegmentMetadata() != null && segment.getSegmentMetadata().getColumnMetadataMap() != null) {

Review Comment:
   I'd suggest to move the extra checks in to skipSegmentOutOfTTL method



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [PR] apache#15162 - Bug Fix Segments going into BAD state for Dedup Tables with metadataTTL [pinot]

Posted by "deepthi912 (via GitHub)" <gi...@apache.org>.
deepthi912 commented on code in PR #15178:
URL: https://github.com/apache/pinot/pull/15178#discussion_r1982241163


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/BasePartitionDedupMetadataManager.java:
##########
@@ -283,8 +283,13 @@ public void removeSegment(IndexSegment segment) {
       return;
     }
     try {
-      if (skipSegmentOutOfTTL(segment, false)) {
-        return;
+      // segment.getSegmentMetadata().getColumnMetadataMap() will be null when offloading a temporary segment data
+      // manager created in BaseTableDataManager.addSegment which caused replicas to go into BAD state when
+      // metadataTTL is set
+      if (segment.getSegmentMetadata() != null && segment.getSegmentMetadata().getColumnMetadataMap() != null) {

Review Comment:
   Should we eliminate the private final Logger _segmentLogger; all together from this class and use the regular way? I think the e.String() is logging the message right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org