You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "noob-se7en (via GitHub)" <gi...@apache.org> on 2025/11/20 06:12:29 UTC

[PR] Enable deep store upload retry for Pauseless [pinot]

noob-se7en opened a new pull request, #17241:
URL: https://github.com/apache/pinot/pull/17241

   Instructions:
   1. The PR has to be tagged with at least one of the following labels (*):
      1. `feature`
      2. `bugfix`
      3. `performance`
      4. `ui`
      5. `backward-incompat`
      6. `release-notes` (**)
   2. Remove these instructions before publishing the PR.
    
   (*) Other labels to consider:
   - `testing`
   - `dependencies`
   - `docker`
   - `kubernetes`
   - `observability`
   - `security`
   - `code-style`
   - `extension-point`
   - `refactor`
   - `cleanup`
   
   (**) Use `release-notes` label for scenarios like:
   - New configuration options
   - Deprecation of configurations
   - Signature changes to public methods/interfaces
   - New plugins added or old plugins removed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enable deep store upload retry for Pauseless by default [pinot]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17241:
URL: https://github.com/apache/pinot/pull/17241#discussion_r2553369232


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -233,17 +233,26 @@ public PinotLLCRealtimeSegmentManager(PinotHelixResourceManager helixResourceMan
             helixResourceManager);
     _flushThresholdUpdateManager = new FlushThresholdUpdateManager();
     _isDeepStoreLLCSegmentUploadRetryEnabled = controllerConf.isDeepStoreRetryUploadLLCSegmentEnabled();
+    if (_isDeepStoreLLCSegmentUploadRetryEnabled != null && _isDeepStoreLLCSegmentUploadRetryEnabled) {
+      enableDeepStoreSegmentUploadRetry(controllerConf.getDeepStoreRetryUploadParallelism());
+    }
     _isTmpSegmentAsyncDeletionEnabled = controllerConf.isTmpSegmentAsyncDeletionEnabled();
     _deepstoreUploadRetryTimeoutMs = controllerConf.getDeepStoreRetryUploadTimeoutMs();
-    _fileUploadDownloadClient = _isDeepStoreLLCSegmentUploadRetryEnabled ? initFileUploadDownloadClient() : null;
-    _deepStoreUploadExecutor = _isDeepStoreLLCSegmentUploadRetryEnabled ? Executors.newFixedThreadPool(
-        controllerConf.getDeepStoreRetryUploadParallelism()) : null;
-    _deepStoreUploadExecutorPendingSegments =
-        _isDeepStoreLLCSegmentUploadRetryEnabled ? ConcurrentHashMap.newKeySet() : null;
   }
 
-  public boolean isDeepStoreLLCSegmentUploadRetryEnabled() {
-    return _isDeepStoreLLCSegmentUploadRetryEnabled;
+  private void enableDeepStoreSegmentUploadRetry(int deepStoreRetryUploadParallelism) {
+    _fileUploadDownloadClient = initFileUploadDownloadClient();
+    _deepStoreUploadExecutor = Executors.newFixedThreadPool(deepStoreRetryUploadParallelism);
+    _deepStoreUploadExecutorPendingSegments = ConcurrentHashMap.newKeySet();
+  }
+
+  public boolean isDeepStoreLLCSegmentUploadRetryEnabled(boolean pauselessEnabled) {
+    if (_isDeepStoreLLCSegmentUploadRetryEnabled == null && pauselessEnabled) {
+      _isDeepStoreLLCSegmentUploadRetryEnabled = true;
+      enableDeepStoreSegmentUploadRetry(
+          ControllerConf.ControllerPeriodicTasksConf.DEFAULT_DEEP_STORE_RETRY_UPLOAD_PARALLELISM);
+    }
+    return _isDeepStoreLLCSegmentUploadRetryEnabled != null && _isDeepStoreLLCSegmentUploadRetryEnabled;

Review Comment:
   The lazy initialization in `isDeepStoreLLCSegmentUploadRetryEnabled()` is not thread-safe. Multiple threads could simultaneously check `_isDeepStoreLLCSegmentUploadRetryEnabled == null`, resulting in multiple calls to `enableDeepStoreSegmentUploadRetry()` and resource leaks (multiple thread pools and HTTP clients). Use `synchronized` or implement double-checked locking with `volatile` to ensure thread safety.



##########
pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManagerTest.java:
##########
@@ -1201,12 +1201,18 @@ public void testUploadToSegmentStore()
 
     // init fake PinotLLCRealtimeSegmentManager
     ControllerConf controllerConfig = new ControllerConf();
-    controllerConfig.setProperty(ControllerConf.ControllerPeriodicTasksConf.ENABLE_DEEP_STORE_RETRY_UPLOAD_LLC_SEGMENT,
-        true);
+    boolean pauselessEnabled = false;
+    if (new Random().nextDouble() > 0.5) {
+      controllerConfig.setProperty(
+          ControllerConf.ControllerPeriodicTasksConf.ENABLE_DEEP_STORE_RETRY_UPLOAD_LLC_SEGMENT,
+          true);
+    } else {
+      pauselessEnabled = true;
+    }

Review Comment:
   Using randomization in unit tests makes them non-deterministic and harder to debug when they fail. The test should explicitly cover both code paths with separate test methods or parameterized tests to ensure consistent and reproducible behavior.



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -209,13 +209,13 @@ public class PinotLLCRealtimeSegmentManager {
   private final ControllerMetrics _controllerMetrics;
   private final MetadataEventNotifierFactory _metadataEventNotifierFactory;
   private final FlushThresholdUpdateManager _flushThresholdUpdateManager;
-  private final boolean _isDeepStoreLLCSegmentUploadRetryEnabled;
+  private Boolean _isDeepStoreLLCSegmentUploadRetryEnabled;

Review Comment:
   [nitpick] Using a nullable `Boolean` to represent a tri-state (uninitialized/true/false) makes the logic harder to follow. Consider using an explicit enum (e.g., `UNINITIALIZED`, `ENABLED`, `DISABLED`) to improve code clarity and avoid potential confusion with null checks.



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -209,13 +209,13 @@ public class PinotLLCRealtimeSegmentManager {
   private final ControllerMetrics _controllerMetrics;
   private final MetadataEventNotifierFactory _metadataEventNotifierFactory;
   private final FlushThresholdUpdateManager _flushThresholdUpdateManager;
-  private final boolean _isDeepStoreLLCSegmentUploadRetryEnabled;
+  private Boolean _isDeepStoreLLCSegmentUploadRetryEnabled;
   private final boolean _isTmpSegmentAsyncDeletionEnabled;
   private final int _deepstoreUploadRetryTimeoutMs;
-  private final FileUploadDownloadClient _fileUploadDownloadClient;
+  private FileUploadDownloadClient _fileUploadDownloadClient;
   private final AtomicInteger _numCompletingSegments = new AtomicInteger(0);
-  private final ExecutorService _deepStoreUploadExecutor;
-  private final Set<String> _deepStoreUploadExecutorPendingSegments;
+  private ExecutorService _deepStoreUploadExecutor;
+  private Set<String> _deepStoreUploadExecutorPendingSegments;

Review Comment:
   These fields are modified after object construction without synchronization. If multiple threads access these fields during or after lazy initialization, it could lead to visibility issues. Mark these fields as `volatile` or ensure all access is properly synchronized.
   ```suggestion
     private volatile Set<String> _deepStoreUploadExecutorPendingSegments;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enable deep store upload retry for Pauseless by default [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #17241:
URL: https://github.com/apache/pinot/pull/17241


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enable deep store upload retry for Pauseless [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #17241:
URL: https://github.com/apache/pinot/pull/17241#issuecomment-3556350765

   ### :x: 2 Tests Failed:
   | Tests completed | Failed | Passed | Skipped |
   |---|---|---|---|
   | 5804 | 2 | 5802 | 0 |
   <details><summary>View the top 2 failed test(s) by shortest run time</summary>
   
   > <pre><code class="language-python">org.apache.pinot.controller.api.PinotTableRestletResourceTest::testTableTasksCleanupWithActiveTasks</code></pre>
   > <details><summary>Stack Traces | 0.269s run time</summary>
   > 
   > > <pre><code class="language-python">org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 500 (Internal Server Error) with reason: &amp;quot;Failed to delete job: TaskQueue_SegmentGenerationAndPushTask_Task_SegmentGenerationAndPushTask_8a6bd9ac-0e09-41ab-9712-402b782f90db_1763623164555 from queue: TaskQueue_SegmentGenerationAndPushTask&amp;quot; while sending request: .../tasks/task/Task_SegmentGenerationAndPushTask_8a6bd9ac-0e09-41ab-9712-402b782f90db_1763623164555?forceDelete=true to controller: runnervmg1sw1.gsy2v5g5vcjefpjhwj5tqngg5b.ex.internal.cloudapp.net, version: Unknown</code></pre>
   > 
   > </details>
   
   
   > <pre><code class="language-python">org.apache.pinot.controller.api.PinotTableRestletResourceTest::testTableTasksCleanupWithActiveTasks</code></pre>
   > <details><summary>Stack Traces | 0.394s run time</summary>
   > 
   > > <pre><code class="language-python">org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 500 (Internal Server Error) with reason: &quot;Failed to delete job: TaskQueue_SegmentGenerationAndPushTask_Task_SegmentGenerationAndPushTask_8a6bd9ac-0e09-41ab-9712-402b782f90db_1763623164555 from queue: TaskQueue_SegmentGenerationAndPushTask&quot; while sending request: .../tasks/task/Task_SegmentGenerationAndPushTask_8a6bd9ac-0e09-41ab-9712-402b782f90db_1763623164555?forceDelete=true to controller: runnervmg1sw1.gsy2v5g5vcjefpjhwj5tqngg5b.ex.internal.cloudapp.net, version: Unknown</code></pre>
   > 
   > </details>
   
   </details>
   
   To view more test analytics, go to the [Test Analytics Dashboard](https://app.codecov.io/gh/apache/pinot/tests/noob-se7en%3Aenable_deepstore_retry)
   <sub>📋 Got 3 mins? [Take this short survey](https://forms.gle/22i53Qa1CySZjA6c7) to help us improve Test Analytics.</sub>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enable deep store upload retry for Pauseless by default [pinot]

Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on code in PR #17241:
URL: https://github.com/apache/pinot/pull/17241#discussion_r2586227166


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -233,17 +233,26 @@ public PinotLLCRealtimeSegmentManager(PinotHelixResourceManager helixResourceMan
             helixResourceManager);
     _flushThresholdUpdateManager = new FlushThresholdUpdateManager();
     _isDeepStoreLLCSegmentUploadRetryEnabled = controllerConf.isDeepStoreRetryUploadLLCSegmentEnabled();
+    if (_isDeepStoreLLCSegmentUploadRetryEnabled != null && _isDeepStoreLLCSegmentUploadRetryEnabled) {
+      enableDeepStoreSegmentUploadRetry(controllerConf.getDeepStoreRetryUploadParallelism());
+    }
     _isTmpSegmentAsyncDeletionEnabled = controllerConf.isTmpSegmentAsyncDeletionEnabled();
     _deepstoreUploadRetryTimeoutMs = controllerConf.getDeepStoreRetryUploadTimeoutMs();
-    _fileUploadDownloadClient = _isDeepStoreLLCSegmentUploadRetryEnabled ? initFileUploadDownloadClient() : null;
-    _deepStoreUploadExecutor = _isDeepStoreLLCSegmentUploadRetryEnabled ? Executors.newFixedThreadPool(
-        controllerConf.getDeepStoreRetryUploadParallelism()) : null;
-    _deepStoreUploadExecutorPendingSegments =
-        _isDeepStoreLLCSegmentUploadRetryEnabled ? ConcurrentHashMap.newKeySet() : null;
   }
 
-  public boolean isDeepStoreLLCSegmentUploadRetryEnabled() {
-    return _isDeepStoreLLCSegmentUploadRetryEnabled;
+  private void enableDeepStoreSegmentUploadRetry(int deepStoreRetryUploadParallelism) {
+    _fileUploadDownloadClient = initFileUploadDownloadClient();
+    _deepStoreUploadExecutor = Executors.newFixedThreadPool(deepStoreRetryUploadParallelism);
+    _deepStoreUploadExecutorPendingSegments = ConcurrentHashMap.newKeySet();
+  }
+
+  public boolean isDeepStoreLLCSegmentUploadRetryEnabled(boolean pauselessEnabled) {

Review Comment:
   Hmm right. Also this method name is misleading. 
   Will address this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Enable deep store upload retry for Pauseless by default [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #17241:
URL: https://github.com/apache/pinot/pull/17241#discussion_r2558041513


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -233,17 +233,26 @@ public PinotLLCRealtimeSegmentManager(PinotHelixResourceManager helixResourceMan
             helixResourceManager);
     _flushThresholdUpdateManager = new FlushThresholdUpdateManager();
     _isDeepStoreLLCSegmentUploadRetryEnabled = controllerConf.isDeepStoreRetryUploadLLCSegmentEnabled();
+    if (_isDeepStoreLLCSegmentUploadRetryEnabled != null && _isDeepStoreLLCSegmentUploadRetryEnabled) {
+      enableDeepStoreSegmentUploadRetry(controllerConf.getDeepStoreRetryUploadParallelism());
+    }
     _isTmpSegmentAsyncDeletionEnabled = controllerConf.isTmpSegmentAsyncDeletionEnabled();
     _deepstoreUploadRetryTimeoutMs = controllerConf.getDeepStoreRetryUploadTimeoutMs();
-    _fileUploadDownloadClient = _isDeepStoreLLCSegmentUploadRetryEnabled ? initFileUploadDownloadClient() : null;
-    _deepStoreUploadExecutor = _isDeepStoreLLCSegmentUploadRetryEnabled ? Executors.newFixedThreadPool(
-        controllerConf.getDeepStoreRetryUploadParallelism()) : null;
-    _deepStoreUploadExecutorPendingSegments =
-        _isDeepStoreLLCSegmentUploadRetryEnabled ? ConcurrentHashMap.newKeySet() : null;
   }
 
-  public boolean isDeepStoreLLCSegmentUploadRetryEnabled() {
-    return _isDeepStoreLLCSegmentUploadRetryEnabled;
+  private void enableDeepStoreSegmentUploadRetry(int deepStoreRetryUploadParallelism) {
+    _fileUploadDownloadClient = initFileUploadDownloadClient();
+    _deepStoreUploadExecutor = Executors.newFixedThreadPool(deepStoreRetryUploadParallelism);
+    _deepStoreUploadExecutorPendingSegments = ConcurrentHashMap.newKeySet();
+  }
+
+  public boolean isDeepStoreLLCSegmentUploadRetryEnabled(boolean pauselessEnabled) {

Review Comment:
   This is anti-pattern. If we want to have this always enabled for pauseless, we should skip calling this method for pauseless table (always create the executor), instead of setting it as enabled which will also change the behavior of non-pauseless table



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org