You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "noob-se7en (via GitHub)" <gi...@apache.org> on 2025/11/20 08:44:16 UTC

[PR] Adds Disaster Recover Mode as Controller/Cluster config [pinot]

noob-se7en opened a new pull request, #17243:
URL: https://github.com/apache/pinot/pull/17243

   Instructions:
   1. The PR has to be tagged with at least one of the following labels (*):
      1. `feature`
      2. `bugfix`
      3. `performance`
      4. `ui`
      5. `backward-incompat`
      6. `release-notes` (**)
   2. Remove these instructions before publishing the PR.
    
   (*) Other labels to consider:
   - `testing`
   - `dependencies`
   - `docker`
   - `kubernetes`
   - `observability`
   - `security`
   - `code-style`
   - `extension-point`
   - `refactor`
   - `cleanup`
   
   (**) Use `release-notes` label for scenarios like:
   - New configuration options
   - Deprecation of configurations
   - Signature changes to public methods/interfaces
   - New plugins added or old plugins removed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Adds Disaster Recover Mode as Controller/Cluster config [pinot]

Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on code in PR #17243:
URL: https://github.com/apache/pinot/pull/17243#discussion_r2586149349


##########
pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java:
##########
@@ -1129,6 +1132,17 @@ public boolean isAutoResetErrorSegmentsOnValidationEnabled() {
     return getProperty(ControllerPeriodicTasksConf.AUTO_RESET_ERROR_SEGMENTS_VALIDATION, true);
   }
 
+  public DisasterRecoveryMode getDisasterRecoveryMode() {
+    return getDisasterRecoveryMode(getProperty(ControllerPeriodicTasksConf.DISASTER_RECOVERY_MODE_CONFIG_KEY));
+  }
+
+  public static DisasterRecoveryMode getDisasterRecoveryMode(@Nullable String disasterRecoveryModeString) {

Review Comment:
   yeah, right. Thanks fixed it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Adds Disaster Recover Mode as Controller/Cluster config [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #17243:
URL: https://github.com/apache/pinot/pull/17243#discussion_r2566647006


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -2751,6 +2751,13 @@ public void repairSegmentsInErrorStateForPauselessConsumption(TableConfig tableC
         ControllerGauge.PAUSELESS_SEGMENTS_IN_UNRECOVERABLE_ERROR_COUNT, segmentsInUnRecoverableState);
   }
 
+  public boolean shouldRepairErrorSegmentsForPartialUpsertOrDedup(@Nullable DisasterRecoveryMode disasterRecoveryMode) {
+    if (disasterRecoveryMode == null) {

Review Comment:
   This check is redundant



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Adds Disaster Recover Mode as Controller/Cluster config [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #17243:
URL: https://github.com/apache/pinot/pull/17243#issuecomment-3556856030

   ### :x: 2 Tests Failed:
   | Tests completed | Failed | Passed | Skipped |
   |---|---|---|---|
   | 7166 | 2 | 7164 | 0 |
   <details><summary>View the top 2 failed test(s) by shortest run time</summary>
   
   > <pre><code class="language-python">org.apache.pinot.controller.helix.core.rebalance.DataLossRiskAssessorTest::testDataLossRiskAssessorPeerDownloadEnabledPauselessEnabledCommittingSegment</code></pre>
   > <details><summary>Stack Traces | 0.01s run time</summary>
   > 
   > > <pre><code class="language-python">expected [false] but found [true]</code></pre>
   > 
   > </details>
   
   
   > <pre><code class="language-python">org.apache.pinot.controller.helix.core.rebalance.DataLossRiskAssessorTest::testDataLossRiskAssessorPeerDownloadEnabledPauselessEnabledCommittingSegment</code></pre>
   > <details><summary>Stack Traces | 0.01s run time</summary>
   > 
   > > <pre><code class="language-python">expected [false] but found [true]</code></pre>
   > 
   > </details>
   
   </details>
   
   To view more test analytics, go to the [Test Analytics Dashboard](https://app.codecov.io/gh/apache/pinot/tests/noob-se7en%3Adr_config)
   <sub>📋 Got 3 mins? [Take this short survey](https://forms.gle/22i53Qa1CySZjA6c7) to help us improve Test Analytics.</sub>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Adds Disaster Recover Mode as Controller/Cluster config [pinot]

Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish merged PR #17243:
URL: https://github.com/apache/pinot/pull/17243


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Adds Disaster Recover Mode as Controller/Cluster config [pinot]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17243:
URL: https://github.com/apache/pinot/pull/17243#discussion_r2553369017


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -275,15 +279,17 @@ private boolean shouldRunSegmentValidation(Properties periodicTaskProperties, lo
   }
 
   private boolean shouldRepairErrorSegmentsForPartialUpsertOrDedup(Properties periodicTaskProperties) {
-    return Optional.ofNullable(periodicTaskProperties.getProperty(REPAIR_ERROR_SEGMENTS_FOR_PARTIAL_UPSERT_OR_DEDUP))
-        .map(value -> {
-          try {
-            return Boolean.parseBoolean(value);
-          } catch (Exception e) {
-            return false;
-          }
-        })
-        .orElse(false);
+    String property = periodicTaskProperties.getProperty(REPAIR_ERROR_SEGMENTS_FOR_PARTIAL_UPSERT_OR_DEDUP);
+
+    if (property == null) {
+      return _llcRealtimeSegmentManager.shouldRepairErrorSegmentsForPartialUpsertOrDedup(_disasterRecoveryMode);
+    }
+
+    try {
+      return Boolean.parseBoolean(property);
+    } catch (Exception e) {

Review Comment:
   The catch block silently swallows all exceptions and returns `false` without logging. This makes debugging configuration issues difficult. Consider logging the exception with appropriate context about the invalid property value before returning false.
   ```suggestion
       } catch (Exception e) {
         LOGGER.warn("Failed to parse property '{}' for '{}'. Returning false.", property, REPAIR_ERROR_SEGMENTS_FOR_PARTIAL_UPSERT_OR_DEDUP, e);
   ```



##########
pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java:
##########
@@ -1129,6 +1132,17 @@ public boolean isAutoResetErrorSegmentsOnValidationEnabled() {
     return getProperty(ControllerPeriodicTasksConf.AUTO_RESET_ERROR_SEGMENTS_VALIDATION, true);
   }
 
+  public DisasterRecoveryMode getDisasterRecoveryMode() {
+    return getDisasterRecoveryMode(getProperty(ControllerPeriodicTasksConf.DISASTER_RECOVERY_MODE_CONFIG_KEY));
+  }
+
+  public static DisasterRecoveryMode getDisasterRecoveryMode(@Nullable String disasterRecoveryModeString) {
+    if (disasterRecoveryModeString == null) {
+      return null;
+    }
+    return DisasterRecoveryMode.valueOf(disasterRecoveryModeString);

Review Comment:
   The `valueOf()` call will throw an `IllegalArgumentException` if an invalid disaster recovery mode string is provided, but the exception message won't include helpful context about valid values or where the configuration came from. Consider wrapping this in a try-catch that provides a more descriptive error message listing valid `DisasterRecoveryMode` values.
   ```suggestion
       try {
         return DisasterRecoveryMode.valueOf(disasterRecoveryModeString);
       } catch (IllegalArgumentException e) {
         String validModes = Arrays.stream(DisasterRecoveryMode.values())
             .map(Enum::name)
             .collect(Collectors.joining(", "));
         throw new IllegalArgumentException(
             "Invalid disaster recovery mode: '" + disasterRecoveryModeString
                 + "'. Valid values are: [" + validModes + "]."
                 + " (from config key: '" + ControllerPeriodicTasksConf.DISASTER_RECOVERY_MODE_CONFIG_KEY + "')", e);
       }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Adds Disaster Recover Mode as Controller/Cluster config [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #17243:
URL: https://github.com/apache/pinot/pull/17243#discussion_r2566646524


##########
pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java:
##########
@@ -1129,6 +1132,17 @@ public boolean isAutoResetErrorSegmentsOnValidationEnabled() {
     return getProperty(ControllerPeriodicTasksConf.AUTO_RESET_ERROR_SEGMENTS_VALIDATION, true);
   }
 
+  public DisasterRecoveryMode getDisasterRecoveryMode() {
+    return getDisasterRecoveryMode(getProperty(ControllerPeriodicTasksConf.DISASTER_RECOVERY_MODE_CONFIG_KEY));
+  }
+
+  public static DisasterRecoveryMode getDisasterRecoveryMode(@Nullable String disasterRecoveryModeString) {

Review Comment:
   Should it return `DEFAULT` when mode is not explicitly set?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org