You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "krishan1390 (via GitHub)" <gi...@apache.org> on 2026/05/26 12:34:17 UTC

[PR] Controller cleanups: SegmentDeletionManager fallback, LLC filter, retention strategy extraction, mock util [pinot]

krishan1390 opened a new pull request, #18582:
URL: https://github.com/apache/pinot/pull/18582

   ## Summary
   Four small, independent controller-side improvements grouped into one PR. Each lands as its own commit so reviewers can read them concern-by-concern and each is independently revertable. **Please do not squash on merge** — preserve per-commit granularity.
   
   1. **`SegmentDeletionManager`: fall back to recursive remove when batch ZK remove leaves a non-empty znode.** The batch ZK remove API is non-recursive, so it silently fails to delete segment znodes that have accumulated children. On batch-remove failure, fall back to the single-path `_propertyStore.remove(...)` API, which handles non-empty nodes by deleting recursively. Skip when the znode is already absent.
   
   2. **`PinotHelixResourceManager#getLastLLCCompletedSegments`: skip non-LLC-named segments.** A realtime table can contain uploaded (non-LLC-named) segments in `DONE` state alongside the LLC ones. `LLCSegmentName.of(name)` returns `null` for those, causing an NPE on the next field access. Filter them out. Includes a regression test (`PinotHelixResourceManagerLastLLCSegmentsTest`) that reproduces the NPE without the fix.
   
   3. **Extract retention strategy construction into `TableConfigRetentionUtils`.** Pull the `TimeRetentionStrategy` construction (with its parse / invalid-config handling) out of `RetentionManager` into a static helper. Returning `null` for absent or malformed config preserves the existing "skip retention" behavior, and the warn-log shape is preserved (single message for both null/empty and parse-error paths).
   
   4. **Add a partition-aware `SegmentMetadataMockUtils.mockSegmentMetadata` overload.** Additive test helper that returns a mock backed by a `MurmurPartitionFunction`. Purely test-ergonomics.
   
   ## Test plan
   - [x] `./mvnw -pl pinot-controller -am test -Dtest=SegmentDeletionManagerTest,RetentionManagerTest,PinotHelixResourceManagerLastLLCSegmentsTest`
   - [x] `./mvnw spotless:apply checkstyle:check license:check -pl pinot-controller`
   - [ ] Manual: drop a non-LLC `DONE` segment into a realtime table and confirm retention no longer NPEs.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Controller cleanups: SegmentDeletionManager fallback, LLC filter, retention strategy extraction, mock util [pinot]

Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #18582:
URL: https://github.com/apache/pinot/pull/18582#discussion_r3316896115


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/TableConfigRetentionUtils.java:
##########
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.retention;
+
+import java.util.Locale;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
+import org.apache.pinot.controller.helix.core.retention.strategy.RetentionStrategy;
+import org.apache.pinot.controller.helix.core.retention.strategy.TimeRetentionStrategy;
+import org.apache.pinot.spi.config.table.SegmentsValidationAndRetentionConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Utility methods for deriving retention-related objects from a {@link TableConfig}.
+public class TableConfigRetentionUtils {
+  private static final Logger LOGGER = LoggerFactory.getLogger(TableConfigRetentionUtils.class);
+
+  private TableConfigRetentionUtils() {
+  }
+
+  /// Builds a {@link RetentionStrategy} from {@code tableConfig}, or returns {@code null} when the
+  /// retention config is absent, empty, or malformed. A null return means no retention is configured
+  /// and no segment should be treated as purgeable.
+  ///
+  /// @param useCreationTimeFallback when true, the strategy falls back to segment creation time
+  ///                                when segment end time is unavailable

Review Comment:
   @Jackie-Jiang asked to add new comments in markdown style



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Controller cleanups: SegmentDeletionManager fallback, LLC filter, retention strategy extraction, mock util [pinot]

Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps merged PR #18582:
URL: https://github.com/apache/pinot/pull/18582


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Controller cleanups: SegmentDeletionManager fallback, LLC filter, retention strategy extraction, mock util [pinot]

Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on code in PR #18582:
URL: https://github.com/apache/pinot/pull/18582#discussion_r3316224866


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/TableConfigRetentionUtils.java:
##########
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.retention;
+
+import java.util.Locale;
+import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
+import org.apache.pinot.controller.helix.core.retention.strategy.RetentionStrategy;
+import org.apache.pinot.controller.helix.core.retention.strategy.TimeRetentionStrategy;
+import org.apache.pinot.spi.config.table.SegmentsValidationAndRetentionConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/// Utility methods for deriving retention-related objects from a {@link TableConfig}.
+public class TableConfigRetentionUtils {
+  private static final Logger LOGGER = LoggerFactory.getLogger(TableConfigRetentionUtils.class);
+
+  private TableConfigRetentionUtils() {
+  }
+
+  /// Builds a {@link RetentionStrategy} from {@code tableConfig}, or returns {@code null} when the
+  /// retention config is absent, empty, or malformed. A null return means no retention is configured
+  /// and no segment should be treated as purgeable.
+  ///
+  /// @param useCreationTimeFallback when true, the strategy falls back to segment creation time
+  ///                                when segment end time is unavailable

Review Comment:
   nit: Let's follow existing convention for javadoc comments here with /** */.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Controller cleanups: SegmentDeletionManager fallback, LLC filter, retention strategy extraction, mock util [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18582:
URL: https://github.com/apache/pinot/pull/18582#issuecomment-4544760705

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18582?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 56.77%. Comparing base ([`0a8ac02`](https://app.codecov.io/gh/apache/pinot/commit/0a8ac02c8a062ced0f8542f5d0351b0ab2e31db4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`28efa97`](https://app.codecov.io/gh/apache/pinot/commit/28efa97b5121d31333ecccb2940ce5316da3c51e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   > :exclamation:  There is a different number of reports uploaded between BASE (0a8ac02) and HEAD (28efa97). Click for more details.
   > 
   > <details><summary>HEAD has 16 uploads less than BASE</summary>
   >
   >| Flag | BASE (0a8ac02) | HEAD (28efa97) |
   >|------|------|------|
   >|java-21|5|1|
   >|unittests|2|1|
   >|temurin|5|1|
   >|unittests2|1|0|
   >|integration|3|0|
   >|integration1|1|0|
   >|custom-integration1|1|0|
   >|integration2|1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #18582      +/-   ##
   ============================================
   - Coverage     64.27%   56.77%   -7.50%     
   + Complexity     1137        1    -1136     
   ============================================
     Files          3335     2564     -771     
     Lines        205898   148952   -56946     
     Branches      32129    24099    -8030     
   ============================================
   - Hits         132333    84562   -47771     
   + Misses        62931    57203    -5728     
   + Partials      10634     7187    -3447     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/18582/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18582/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/18582/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/18582/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/18582/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/18582/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.77% <ø> (-7.50%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/18582/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.77% <ø> (-7.50%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/18582/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.77% <ø> (-7.50%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18582/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.77% <ø> (+0.01%)` | :arrow_up: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18582/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18582?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org