You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "shounakmk219 (via GitHub)" <gi...@apache.org> on 2024/07/11 06:33:30 UTC

[PR] Storage Quota imposition on Realtime tables [pinot]

shounakmk219 opened a new pull request, #13584:
URL: https://github.com/apache/pinot/pull/13584

   # Description
   
   This PR allows pinot to impose storage quota restrictions on realtime tables.
   To block the ingestion on realtime tables below consideration are kept in mind
   
   1. Replicas should be consistent.
   2. Ingestion should auto resume upon quota increase or freed up storage.
   3. Easy observability into the blocked ingestion with relevant info. 
   4. Query results should be consistent
   
   Also to keep it simple the ingestion is blocked only when storage quotas are breached
   
   # Blocking ingestion
   
   - The storage quota will be imposed only upon successful consuming segment completion.
   - `SegmentCompletionManager` will first calculate whether the table is within quota.
   - If the table is breaching quota, `IS_QUOTA_EXCEEDED` flag will be set on the table IS
   - This flag is checked during the new consuming segment creation
   - If it’s `true` then new consuming segments will not be created
   - Ongoing segment commit will be unaffected.
   
   ## What if the `IS_QUOTA_EXCEEDED` flag is set to false manually on zk directly?
   
   Setting the `IS_QUOTA_EXCEEDED` flag to `false` alone will not create the new consuming segments. New consuming segments can be created by:
   
   1. Calling the `/resumeConsumption` API on controller
   2. Running `RealtimeSegmentValidationManager` job with setting `recreateDeletedConsumingSegment` as true
   
   The resume consumption API internally depends on the `RealtimeSegmentValidationManager` task itself to handle the consuming segment creation. Hence to cover both the ways we only need to handle the quota exceed check at `RealtimeSegmentValidationManager`.
   `RealtimeSegmentValidationManager` will again check for the storage quota and set the `IS_QUOTA_EXCEEDED` flag to the right value.
   
   # Resuming ingestion
   
   Once a table exceeds the storage quota, we may need to resume the consumption in below cases:
   
   1. Storage quota is extended for the table
   2. Storage frees up upon older segments being deleted (by retention manager)
   
   To automate the resume consumption flow we will depend on the `RealtimeSegmentValidationManager` job itself. 
   In case the user does not want to wait for the job to run, they can manually trigger the job from the controller API or use the resume consumption endpoint itself.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1756734820


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/StorageQuotaChecker.java:
##########
@@ -43,14 +44,19 @@ public class StorageQuotaChecker {
   private final ControllerMetrics _controllerMetrics;
   private final LeadControllerManager _leadControllerManager;
   private final PinotHelixResourceManager _pinotHelixResourceManager;
+  private final ControllerConf _controllerConf;
+  private final int _timeout;
 
   public StorageQuotaChecker(TableSizeReader tableSizeReader,
       ControllerMetrics controllerMetrics, LeadControllerManager leadControllerManager,
-      PinotHelixResourceManager pinotHelixResourceManager) {
+      PinotHelixResourceManager pinotHelixResourceManager, ControllerConf controllerConf) {
     _tableSizeReader = tableSizeReader;
     _controllerMetrics = controllerMetrics;
     _leadControllerManager = leadControllerManager;
     _pinotHelixResourceManager = pinotHelixResourceManager;
+    _controllerConf = controllerConf;

Review Comment:
   Makes sense, will make the change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1763607852


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -108,8 +112,45 @@ protected void processTable(String tableNameWithType, Context context) {
     if (context._runSegmentLevelValidation) {
       runSegmentLevelValidation(tableConfig, streamConfig);
     }
-    _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
-        context._recreateDeletedConsumingSegment, context._offsetCriteria);
+
+    if (shouldEnsureConsuming(tableNameWithType, context)) {
+      _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
+          context._recreateDeletedConsumingSegment, context._offsetCriteria);
+    }
+  }
+
+  private boolean shouldEnsureConsuming(String tableNameWithType, Context context) {
+    // Keeps the table paused/unpaused based pause validations.
+    // Skips updating the pause state if table is paused by admin
+    PauseState pauseState = computePauseState(tableNameWithType);
+    if (!pauseState.isPaused()) {
+      boolean unPausedUponStorageWithinQuota =
+        pauseState.getReasonCode().equals(PauseState.ReasonCode.STORAGE_QUOTA_EXCEEDED);
+      if (unPausedUponStorageWithinQuota) {
+        // recreate consuming segments if table is resumed upon the table storage getting within quota limit
+        context._recreateDeletedConsumingSegment = true;

Review Comment:
   No problem, I'll raise a PR to remove the flag



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1682260924


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1696,6 +1705,20 @@ private IdealState updatePauseStatusInIdealState(String tableNameWithType, boole
     return updatedIdealState;
   }
 
+  public boolean updateStorageQuotaExceededInIdealState(String tableNameWithType, boolean quotaExceeded) {
+    IdealState is = getIdealState(tableNameWithType);
+    if (is.getRecord().getBooleanField(IS_QUOTA_EXCEEDED, false) != quotaExceeded) {
+      HelixHelper.updateIdealState(_helixManager, tableNameWithType, idealState -> {
+        ZNRecord znRecord = idealState.getRecord();
+        znRecord.setSimpleField(IS_QUOTA_EXCEEDED, Boolean.valueOf(quotaExceeded).toString());
+        return new IdealState(znRecord);
+      }, RetryPolicies.noDelayRetryPolicy(1));

Review Comment:
   No strong reason to not retry. Was keeping it inline with the table pause IS update. 
   Right now in case of failure the next segment commit or validation job, whichever runs first, will update the IS.
   Let me know if you have any magic number for retry in mind, maybe 3?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1759248976


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -108,8 +112,36 @@ protected void processTable(String tableNameWithType, Context context) {
     if (context._runSegmentLevelValidation) {
       runSegmentLevelValidation(tableConfig, streamConfig);
     }
-    _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
-        context._recreateDeletedConsumingSegment, context._offsetCriteria);
+
+    // Keeps the table paused/unpaused based on storage quota validation.
+    // Skips updating the pause state if table is paused by admin
+    PauseState pauseState = computePauseState(tableNameWithType);
+
+    if (!pauseState.isPaused()) {
+      boolean unPausedUponStorageWithinQuota =
+          pauseState.getReasonCode().equals(PauseState.ReasonCode.STORAGE_QUOTA_EXCEEDED);
+      _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
+          context._recreateDeletedConsumingSegment || unPausedUponStorageWithinQuota, context._offsetCriteria);

Review Comment:
   Leave it as it is. That variable is not needed to evaluate `shouldRestartConsumption()` (actually, a better name could be `shouldEnsureConsuming()`).  If the condition evaluates to true, then call it the same way as before.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1761941816


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -108,8 +112,45 @@ protected void processTable(String tableNameWithType, Context context) {
     if (context._runSegmentLevelValidation) {
       runSegmentLevelValidation(tableConfig, streamConfig);
     }
-    _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
-        context._recreateDeletedConsumingSegment, context._offsetCriteria);
+
+    if (shouldEnsureConsuming(tableNameWithType, context)) {
+      _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
+          context._recreateDeletedConsumingSegment, context._offsetCriteria);
+    }
+  }
+
+  private boolean shouldEnsureConsuming(String tableNameWithType, Context context) {
+    // Keeps the table paused/unpaused based pause validations.
+    // Skips updating the pause state if table is paused by admin
+    PauseState pauseState = computePauseState(tableNameWithType);
+    if (!pauseState.isPaused()) {
+      boolean unPausedUponStorageWithinQuota =
+        pauseState.getReasonCode().equals(PauseState.ReasonCode.STORAGE_QUOTA_EXCEEDED);
+      if (unPausedUponStorageWithinQuota) {
+        // recreate consuming segments if table is resumed upon the table storage getting within quota limit
+        context._recreateDeletedConsumingSegment = true;

Review Comment:
   ```suggestion
           return true;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1682278381


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -108,8 +110,20 @@ protected void processTable(String tableNameWithType, Context context) {
     if (context._runSegmentLevelValidation) {
       runSegmentLevelValidation(tableConfig, streamConfig);
     }
+
+    // IS is updated in below cases
+    // Case 1 -> Table is exceeding storage quota but IS still has "isQuotaExceeded" as false
+    // This will help prevent consuming segment creation through this task upon manual zk changes to
+    // "isQuotaExceeded" in IS.
+    // Case 2 -> Table is within quota limits now but IS still has "isQuotaExceeded" as true
+    // This will help resume the table consumption once the quota is available due to either quota increase or
+    // segment deletion.
+    // In which case we need to pass "recreateDeletedConsumingSegment" as true to "ensureAllPartitionsConsuming" below.
+    boolean idealStateUpdated = _llcRealtimeSegmentManager.updateStorageQuotaExceededInIdealState(tableNameWithType,
+        _storageQuotaChecker.isTableStorageQuotaExceeded(tableConfig));
+
     _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
-        context._recreateDeletedConsumingSegment, context._offsetCriteria);
+        context._recreateDeletedConsumingSegment || idealStateUpdated, context._offsetCriteria);

Review Comment:
   That's right. This will effectively recreate the consuming segment only in case 2. 
   Even though `idealStateUpdated` is set in case 1 as well, eventually it will skip the consuming segment creation as in case 1 the `IS_QUOTA_EXCEEDED` flag is set which will effectively skip creating consuming segment.
   But then I need to ensure the `updateStorageQuotaExceededInIdealState` should return false incase the IS update fails as in that case `IS_QUOTA_EXCEEDED` flag is not set which will end up recreating the consuming segment. Will add that check as well. 
   Thanks for pointing it out!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2222200098

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13584?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `0%` with `60 lines` in your changes missing coverage. Please review.
   > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`ad67d9d`)](https://app.codecov.io/gh/apache/pinot/commit/ad67d9d342c32232a975ca1bd75b177727d3c3d9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 742 commits behind head on master.
   
   | [Files](https://app.codecov.io/gh/apache/pinot/pull/13584?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...not/controller/validation/StorageQuotaChecker.java](https://app.codecov.io/gh/apache/pinot/pull/13584?src=pr&el=tree&filepath=pinot-controller%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcontroller%2Fvalidation%2FStorageQuotaChecker.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL1N0b3JhZ2VRdW90YUNoZWNrZXIuamF2YQ==) | 0.00% | [31 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../core/realtime/PinotLLCRealtimeSegmentManager.java](https://app.codecov.io/gh/apache/pinot/pull/13584?src=pr&el=tree&filepath=pinot-controller%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcontroller%2Fhelix%2Fcore%2Frealtime%2FPinotLLCRealtimeSegmentManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90TExDUmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh) | 0.00% | [18 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../helix/core/realtime/SegmentCompletionManager.java](https://app.codecov.io/gh/apache/pinot/pull/13584?src=pr&el=tree&filepath=pinot-controller%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcontroller%2Fhelix%2Fcore%2Frealtime%2FSegmentCompletionManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1NlZ21lbnRDb21wbGV0aW9uTWFuYWdlci5qYXZh) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...r/validation/RealtimeSegmentValidationManager.java](https://app.codecov.io/gh/apache/pinot/pull/13584?src=pr&el=tree&filepath=pinot-controller%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcontroller%2Fvalidation%2FRealtimeSegmentValidationManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL1JlYWx0aW1lU2VnbWVudFZhbGlkYXRpb25NYW5hZ2VyLmphdmE=) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...apache/pinot/controller/BaseControllerStarter.java](https://app.codecov.io/gh/apache/pinot/pull/13584?src=pr&el=tree&filepath=pinot-controller%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcontroller%2FBaseControllerStarter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9CYXNlQ29udHJvbGxlclN0YXJ0ZXIuamF2YQ==) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../controller/api/upload/SegmentValidationUtils.java](https://app.codecov.io/gh/apache/pinot/pull/13584?src=pr&el=tree&filepath=pinot-controller%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcontroller%2Fapi%2Fupload%2FSegmentValidationUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvdXBsb2FkL1NlZ21lbnRWYWxpZGF0aW9uVXRpbHMuamF2YQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   > :exclamation:  There is a different number of reports uploaded between BASE (59551e4) and HEAD (ad67d9d). Click for more details.
   > 
   > <details><summary>HEAD has 48 uploads less than BASE</summary>
   >
   >| Flag | BASE (59551e4) | HEAD (ad67d9d) |
   >|------|------|------|
   >|integration|7|2|
   >|integration2|3|2|
   >|temurin|12|2|
   >|java-21|7|2|
   >|skip-bytebuffers-true|3|1|
   >|skip-bytebuffers-false|7|1|
   >|unittests|5|0|
   >|unittests1|2|0|
   >|java-11|5|0|
   >|unittests2|3|0|
   >|integration1|2|0|
   >|custom-integration1|2|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #13584       +/-   ##
   =============================================
   - Coverage     61.75%    0.00%   -61.76%     
   =============================================
     Files          2436     2483       +47     
     Lines        133233   137337     +4104     
     Branches      20636    21320      +684     
   =============================================
   - Hits          82274        0    -82274     
   - Misses        44911   137337    +92426     
   + Partials       6048        0     -6048     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/13584/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/13584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/13584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-0.01%)` | :arrow_down: |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/13584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/13584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/13584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/13584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.63%)` | :arrow_down: |
   | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/13584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.75%)` | :arrow_down: |
   | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/13584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-27.73%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/13584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.76%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/13584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/13584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/13584/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/13584?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2274108292

   Hey @mcvsubbu @sajjad-moradi , it is ok if the table storage spills over the quota by few segments before stopping the consumption and that said we can keep the quota validation + IS update logic at one place in segment validation manager. 
   Also as per the suggestion of introducing `pauseStatus`, a single check  on `pauseStatus` will cover both the pause reasons, eliminating the need of adding any extra checks during segment completion.
   
   > Can you state why you need the quota to be checked during segment completion?
   
   There is no strict storage quota imposition requirement, but also didn’t saw any major concern in putting the check during segment completion
   
   > remove latest segments (and adjust consumption offsets, of course) if the quota is exceeded
   
   We don’t want to do this as this will affect the query results (moving back in time).
   
   If it turns out that segment validation manager job frequency is not working for the storage quota imposition we can introduce a separate job when needed.
   Let me know if this sounds good.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2252520066

   @sajjad-moradi can you please review the PR?
   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "sajjad-moradi (via GitHub)" <gi...@apache.org>.
sajjad-moradi commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2273986235

   @Jackie-Jiang any thoughts here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1692531490


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1696,6 +1705,32 @@ private IdealState updatePauseStatusInIdealState(String tableNameWithType, boole
     return updatedIdealState;
   }
 
+  /**
+   * Updates the table IS property 'isQuotaExceeded' based on provided 'quotaExceeded'.
+   * Will be a no op in case the IS already has the same value.
+   * @param tableNameWithType table on which to update the IS
+   * @param quotaExceeded boolean indicating whether table has exceeded the quota limits
+   * @return true if the IS was successfully updated for the table. Returns false in case of no op or the update fails.
+   */
+  public boolean updateStorageQuotaExceededInIdealState(String tableNameWithType, boolean quotaExceeded) {
+    IdealState is = getIdealState(tableNameWithType);

Review Comment:
   Notes from discussion with @swaminathanmanish 
   Need to maintain the state for below reasons:
   
   The ideal state update happens only during the point where table storage exceeds the quota, making it a less frequent operation (not affected by segment commit rate)
   - All further segment commits will just undergo read operation (IS is already being fetched during segment commit so not additional read is made for quota exceed check).
   - All prior segment commits will only perform size calculation and no IS operation (reuse already available IS)
   
   Whereas in stateless approach we will keep querying all servers to get the table size for all the further segment commits which feels unnecessary.
   
   
   The resume consumption flow gets tricky with stateless as we don’t have a pointer on whether the consuming segments were not created due to quota breach or were manually deleted.
   - Right now the segment validation task does not recreate consuming segments. Unless we have a context on whether the quota was earlier breached or not, we would end up recreating the missing consuming segment at every run which is not the existing behaviour.
   - Also this job runs every hour so the IS update operation done by this job won’t be a concern.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1765124295


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -108,8 +112,45 @@ protected void processTable(String tableNameWithType, Context context) {
     if (context._runSegmentLevelValidation) {
       runSegmentLevelValidation(tableConfig, streamConfig);
     }
-    _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
-        context._recreateDeletedConsumingSegment, context._offsetCriteria);
+
+    if (shouldEnsureConsuming(tableNameWithType, context)) {
+      _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
+          context._recreateDeletedConsumingSegment, context._offsetCriteria);
+    }
+  }
+
+  private boolean shouldEnsureConsuming(String tableNameWithType, Context context) {
+    // Keeps the table paused/unpaused based pause validations.
+    // Skips updating the pause state if table is paused by admin
+    PauseState pauseState = computePauseState(tableNameWithType);
+    if (!pauseState.isPaused()) {
+      boolean unPausedUponStorageWithinQuota =
+        pauseState.getReasonCode().equals(PauseState.ReasonCode.STORAGE_QUOTA_EXCEEDED);
+      if (unPausedUponStorageWithinQuota) {
+        // recreate consuming segments if table is resumed upon the table storage getting within quota limit
+        context._recreateDeletedConsumingSegment = true;

Review Comment:
   https://github.com/apache/pinot/pull/14024



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1762202266


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -108,8 +112,45 @@ protected void processTable(String tableNameWithType, Context context) {
     if (context._runSegmentLevelValidation) {
       runSegmentLevelValidation(tableConfig, streamConfig);
     }
-    _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
-        context._recreateDeletedConsumingSegment, context._offsetCriteria);
+
+    if (shouldEnsureConsuming(tableNameWithType, context)) {
+      _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
+          context._recreateDeletedConsumingSegment, context._offsetCriteria);
+    }
+  }
+
+  private boolean shouldEnsureConsuming(String tableNameWithType, Context context) {
+    // Keeps the table paused/unpaused based pause validations.
+    // Skips updating the pause state if table is paused by admin
+    PauseState pauseState = computePauseState(tableNameWithType);
+    if (!pauseState.isPaused()) {
+      boolean unPausedUponStorageWithinQuota =
+        pauseState.getReasonCode().equals(PauseState.ReasonCode.STORAGE_QUOTA_EXCEEDED);
+      if (unPausedUponStorageWithinQuota) {
+        // recreate consuming segments if table is resumed upon the table storage getting within quota limit
+        context._recreateDeletedConsumingSegment = true;

Review Comment:
   Once the quota is increased on table or storage is freed due to retention, how will the consuming segments be recreated unless we set this flag?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2274055552

   > On a separate note, @mcvsubbu thinks it's better to have all the logic for hanlding storage quota (pausing/resuming) in one place, in "realtime segment validation manager" periodic job. Of course the benefit is that all the logic is in one place, so it's better in terms of maintainability. The downside of doing that is the the storage won't be checked till the next round of the periodic job. Maybe that's ok because normally consuming segments take hours to complete, and even checking quota upon segment completion is also delayed. WDYT?
   
   There is enough going on during the segment completion, that I would hate to add more logic there with more conditions. Can you state why you need the quota to be checked during segment completion?
   
   I am even fine adding logic in the background task to remove latest segments (and adjust consumption offsets, of course) if the quota is exceeded (it may matter in some cases that the excess quota will result in charges). On the other hand, if this is the case, the metric on current storage use should be monitored in order to flag such situations early and in advance


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "sajjad-moradi (via GitHub)" <gi...@apache.org>.
sajjad-moradi commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2240321105

   > Mostly good. Can we add a test to verify this?
   > 
   > @swaminathanmanish @sajjad-moradi please also take a look
   
   Will review next week.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1707957034


##########
pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerGauge.java:
##########
@@ -160,7 +160,8 @@ public enum ControllerGauge implements AbstractMetrics.Gauge {
 
   TABLE_DISABLED("tableDisabled", false),
 
-  TABLE_REBALANCE_IN_PROGRESS("tableRebalanceInProgress", false);
+  TABLE_REBALANCE_IN_PROGRESS("tableRebalanceInProgress", false),
+  TABLE_STORAGE_QUOTA_EXCEEDED("tableStorageQuotaExceeded", true);

Review Comment:
   We should not need a metric saying quota is exceeded. Instead, let us have a metric that emits the percent of quota used by the table. Alerts can be set in specific destinations on the value of this metric. Some installations can choose to set the alert at 80%, others at 90%, etc. depending on the action they want to take.
   
   The value can also be greater than 100%, btw.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "sajjad-moradi (via GitHub)" <gi...@apache.org>.
sajjad-moradi commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2266179376

   I just had a discussion with @mcvsubbu about this. Here's what we think:
   With the proposed solution in this PR, there will be two IS flags indicating if the table is paused: "isTablePaused" and "isQuotaExceeded". That doesn't sound right. Instead we can have a more detailed field called pauseStatus. Here's an example for the storage quota exceed case:
   ```json
   "pauseStatus": {
     "isPaused": true,
     "reasonCode": "STORAGE_QUOTA_EXCEEDED",
     "comment": "Current storage 2.3G exceeds quota 2.0G",
     "time": "2024-08-01T09:30:00Z"
   }
   ```
   And here is an example for a regular pause that's caused by an admin hitting the pause endpoint:
   ```json
   "pauseStatus": {
     "isPaused": true,
     "reasonCode": "ADMINISTRATIVE",
     "comment": "Need to change the underlying topic",
     "time": "2024-08-01T09:30:00Z"
   }
   ```
   This way in case in future there's another reason for pausing the table, there's no need to add yet another field to IS to pause the table. We can simply do that by adding another `reasonCode` which is an enum in the source code.
   Moreover, in case in future we want to extend pause functionality like adding support for partition level pausing, we can simply add more fields to pauseStatus.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "sajjad-moradi (via GitHub)" <gi...@apache.org>.
sajjad-moradi commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2273985656

   On a separate note, @mcvsubbu thinks it's better to have all the logic for hanlding storage quota (pausing/resuming) in one place, in "realtime segment validation manager" periodic job. 
   Of course the benefit is that all the logic is in one place, so it's better in terms of maintainability. The downside of doing that is the the storage won't be checked till the next round of the periodic job. Maybe that's ok because normally consuming segments take hours to complete, and even checking quota upon segment completion is also delayed. WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1757335291


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -108,8 +112,34 @@ protected void processTable(String tableNameWithType, Context context) {
     if (context._runSegmentLevelValidation) {
       runSegmentLevelValidation(tableConfig, streamConfig);
     }
-    _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
-        context._recreateDeletedConsumingSegment, context._offsetCriteria);
+
+    // Keeps the table paused/unpaused based on storage quota validation.
+    // Skips updating the pause state if table is paused by admin
+    PauseState pauseState = computePauseState(tableNameWithType);
+
+    if (!pauseState.isPaused()) {

Review Comment:
   I learned that the flag is not quite as I had imagined. Yes, it is a config, and if turned true, then it ends up invoking the periodic job with 'true' (which ends up handling a condition of someone accidentally or intentionally removing the consuming segments of a table).
   
   the flag is _also_ used in the "resume" part because resume happens via kicking off the periodic task.
   
   cc: @sajjad-moradi 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1685973069


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1696,6 +1705,32 @@ private IdealState updatePauseStatusInIdealState(String tableNameWithType, boole
     return updatedIdealState;
   }
 
+  /**
+   * Updates the table IS property 'isQuotaExceeded' based on provided 'quotaExceeded'.
+   * Will be a no op in case the IS already has the same value.
+   * @param tableNameWithType table on which to update the IS
+   * @param quotaExceeded boolean indicating whether table has exceeded the quota limits
+   * @return true if the IS was successfully updated for the table. Returns false in case of no op or the update fails.
+   */
+  public boolean updateStorageQuotaExceededInIdealState(String tableNameWithType, boolean quotaExceeded) {
+    IdealState is = getIdealState(tableNameWithType);
+    if (is.getRecord().getBooleanField(IS_QUOTA_EXCEEDED, false) != quotaExceeded) {
+      is = HelixHelper.updateIdealState(_helixManager, tableNameWithType, idealState -> {
+        ZNRecord znRecord = idealState.getRecord();
+        znRecord.setSimpleField(IS_QUOTA_EXCEEDED, Boolean.valueOf(quotaExceeded).toString());
+        return new IdealState(znRecord);
+      }, RetryPolicies.noDelayRetryPolicy(1));
+      if (is == null) {
+        LOGGER.error("Failed to set 'isQuotaExceeded' to {} in the Ideal State for table {}.", quotaExceeded,

Review Comment:
   This can mean that quota can be breached because we could not set the state ?



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1696,6 +1705,32 @@ private IdealState updatePauseStatusInIdealState(String tableNameWithType, boole
     return updatedIdealState;
   }
 
+  /**
+   * Updates the table IS property 'isQuotaExceeded' based on provided 'quotaExceeded'.
+   * Will be a no op in case the IS already has the same value.
+   * @param tableNameWithType table on which to update the IS
+   * @param quotaExceeded boolean indicating whether table has exceeded the quota limits
+   * @return true if the IS was successfully updated for the table. Returns false in case of no op or the update fails.
+   */
+  public boolean updateStorageQuotaExceededInIdealState(String tableNameWithType, boolean quotaExceeded) {
+    IdealState is = getIdealState(tableNameWithType);

Review Comment:
   Discussed with @shounakmk219 to explore stateless approach, where we always read table size, instead of persisting in Zk. 
   
   Ideal state update read-modify-write is expensive we will incur large number of writes when there are large number of segments. This state is also updated from RealtimeSegmentValidation thread on the controller. 
   
   cc @Jackie-Jiang 



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1696,6 +1705,20 @@ private IdealState updatePauseStatusInIdealState(String tableNameWithType, boole
     return updatedIdealState;
   }
 
+  public boolean updateStorageQuotaExceededInIdealState(String tableNameWithType, boolean quotaExceeded) {
+    IdealState is = getIdealState(tableNameWithType);
+    if (is.getRecord().getBooleanField(IS_QUOTA_EXCEEDED, false) != quotaExceeded) {
+      HelixHelper.updateIdealState(_helixManager, tableNameWithType, idealState -> {
+        ZNRecord znRecord = idealState.getRecord();
+        znRecord.setSimpleField(IS_QUOTA_EXCEEDED, Boolean.valueOf(quotaExceeded).toString());
+        return new IdealState(znRecord);
+      }, RetryPolicies.noDelayRetryPolicy(1));

Review Comment:
   If we do retry, we will have to move this check also inside the updater lambda right
     IdealState is = getIdealState(tableNameWithType);
       if (is.getRecord().getBooleanField(IS_QUOTA_EXCEEDED, false) != quotaExceeded) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2326191290

   Hey @mcvsubbu @sajjad-moradi I have updated the PR with the changes from PauseState PR and also removed the storage quota check from segment commit process and now only have it as part of the `RealtimeSegmentValidationManager`.
   Please take another look!
   Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps merged PR #13584:
URL: https://github.com/apache/pinot/pull/13584


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2268114649

   @sajjad-moradi @mcvsubbu , the `pauseStatus` field makes sense. Do you think we need to maintain the set of different pause reasons active on a table as well? Context will be lost during the reason being overridden/not recorded based on how we define the priority of these reasons. 
   
   - paused consumption should not be overridden by quota breach as in that case consumption will resume once quota gets honoured
   - quota breach can be overridden by paused consumption as anyways quota limit is checked every time before resuming consumption.
   
   Here the priority is well established so may not be a big concern.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "sajjad-moradi (via GitHub)" <gi...@apache.org>.
sajjad-moradi commented on PR #13584:
URL: https://github.com/apache/pinot/pull/13584#issuecomment-2273949793

   The priority you described makes sense. Basically storage quota should not be even be checked if the table is paused by an admin.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1682291845


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/StorageQuotaChecker.java:
##########
@@ -205,4 +213,48 @@ public QuotaCheckerResponse isSegmentStorageWithinQuota(TableConfig tableConfig,
       return failure(message);
     }
   }
+
+  public boolean isTableStorageQuotaExceeded(TableConfig tableConfig) {

Review Comment:
   Agree, will do.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1757102975


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -108,8 +112,34 @@ protected void processTable(String tableNameWithType, Context context) {
     if (context._runSegmentLevelValidation) {
       runSegmentLevelValidation(tableConfig, streamConfig);
     }
-    _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
-        context._recreateDeletedConsumingSegment, context._offsetCriteria);
+
+    // Keeps the table paused/unpaused based on storage quota validation.
+    // Skips updating the pause state if table is paused by admin
+    PauseState pauseState = computePauseState(tableNameWithType);
+
+    if (!pauseState.isPaused()) {

Review Comment:
   Sorry for the nit, but this reads like it is invoking the periodic consumption fix if the table is not paused. That is partially true, but another factor is that the table may have run out of quota.
   
   So, it is better to create a method like `shouldEnsurePartitionsConsuming()` from which we return a boolean instead of returning the PauseState.
   
   Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1758116054


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -108,8 +112,36 @@ protected void processTable(String tableNameWithType, Context context) {
     if (context._runSegmentLevelValidation) {
       runSegmentLevelValidation(tableConfig, streamConfig);
     }
-    _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
-        context._recreateDeletedConsumingSegment, context._offsetCriteria);
+
+    // Keeps the table paused/unpaused based on storage quota validation.
+    // Skips updating the pause state if table is paused by admin
+    PauseState pauseState = computePauseState(tableNameWithType);
+
+    if (!pauseState.isPaused()) {
+      boolean unPausedUponStorageWithinQuota =
+          pauseState.getReasonCode().equals(PauseState.ReasonCode.STORAGE_QUOTA_EXCEEDED);
+      _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
+          context._recreateDeletedConsumingSegment || unPausedUponStorageWithinQuota, context._offsetCriteria);

Review Comment:
   what about `_recreateDeletedConsumingSegment `?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1757275377


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -108,8 +112,34 @@ protected void processTable(String tableNameWithType, Context context) {
     if (context._runSegmentLevelValidation) {
       runSegmentLevelValidation(tableConfig, streamConfig);
     }
-    _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
-        context._recreateDeletedConsumingSegment, context._offsetCriteria);
+
+    // Keeps the table paused/unpaused based on storage quota validation.
+    // Skips updating the pause state if table is paused by admin
+    PauseState pauseState = computePauseState(tableNameWithType);
+
+    if (!pauseState.isPaused()) {

Review Comment:
   Had a chat offline, will handle the future of `_recreateDeletedConsumingSegment` flag in a separate PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1763445783


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -108,8 +112,45 @@ protected void processTable(String tableNameWithType, Context context) {
     if (context._runSegmentLevelValidation) {
       runSegmentLevelValidation(tableConfig, streamConfig);
     }
-    _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
-        context._recreateDeletedConsumingSegment, context._offsetCriteria);
+
+    if (shouldEnsureConsuming(tableNameWithType, context)) {
+      _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
+          context._recreateDeletedConsumingSegment, context._offsetCriteria);
+    }
+  }
+
+  private boolean shouldEnsureConsuming(String tableNameWithType, Context context) {
+    // Keeps the table paused/unpaused based pause validations.
+    // Skips updating the pause state if table is paused by admin
+    PauseState pauseState = computePauseState(tableNameWithType);
+    if (!pauseState.isPaused()) {
+      boolean unPausedUponStorageWithinQuota =
+        pauseState.getReasonCode().equals(PauseState.ReasonCode.STORAGE_QUOTA_EXCEEDED);
+      if (unPausedUponStorageWithinQuota) {
+        // recreate consuming segments if table is resumed upon the table storage getting within quota limit
+        context._recreateDeletedConsumingSegment = true;

Review Comment:
   Fair enough. I was thinking of the world when the flag is not there. Maybe it is good to remove the flag in this PR. Alternatively, you can commit this and either one of us can remove the flag.
   
   here is what I have for the flag removal:
   - change ensureConsuming() method to be in favor of the flag always being true
   - Remove the flag from the config and the map. Anyone who deletes the consuming segments by mistake will want to restart consumption, so it will work. Anyone who deliberately wants to discard consuming segments can do with the pause consumption operation command we have. If necessary, we can enhance the pause command to optionally discard current consuming segments.
   
   I will give a ship it, let me know if you want me to raise a PR to remove that flag.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1755282957


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/StorageQuotaChecker.java:
##########
@@ -43,14 +44,19 @@ public class StorageQuotaChecker {
   private final ControllerMetrics _controllerMetrics;
   private final LeadControllerManager _leadControllerManager;
   private final PinotHelixResourceManager _pinotHelixResourceManager;
+  private final ControllerConf _controllerConf;
+  private final int _timeout;
 
   public StorageQuotaChecker(TableSizeReader tableSizeReader,
       ControllerMetrics controllerMetrics, LeadControllerManager leadControllerManager,
-      PinotHelixResourceManager pinotHelixResourceManager) {
+      PinotHelixResourceManager pinotHelixResourceManager, ControllerConf controllerConf) {
     _tableSizeReader = tableSizeReader;
     _controllerMetrics = controllerMetrics;
     _leadControllerManager = leadControllerManager;
     _pinotHelixResourceManager = pinotHelixResourceManager;
+    _controllerConf = controllerConf;

Review Comment:
   Nit. I may just pick up whatever is needed from config and not keep the entire config object here. In this case, an `_isEnabled` boolean will suffice?



##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/StorageQuotaChecker.java:
##########
@@ -43,14 +44,19 @@ public class StorageQuotaChecker {
   private final ControllerMetrics _controllerMetrics;
   private final LeadControllerManager _leadControllerManager;
   private final PinotHelixResourceManager _pinotHelixResourceManager;
+  private final ControllerConf _controllerConf;
+  private final int _timeout;

Review Comment:
   I think it is useful to call it `_timeoutMs` since it improves readability.



##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -108,8 +112,32 @@ protected void processTable(String tableNameWithType, Context context) {
     if (context._runSegmentLevelValidation) {
       runSegmentLevelValidation(tableConfig, streamConfig);
     }
+
+    // Keeps the table paused/unpaused based on storage quota validation.
+    // Skips updating the pause state if table is paused by admin
+    PauseState pauseState = computePauseState(tableNameWithType);
+
     _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
-        context._recreateDeletedConsumingSegment, context._offsetCriteria);
+        context._recreateDeletedConsumingSegment || !pauseState.isPaused(), context._offsetCriteria);

Review Comment:
   Please change this to  _not_ call `ensureAllPartitionsConsuming` if the table is paused (or storage is exceeeded) rather than overloading the paused boolean with recreation of deleted segment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1757332293


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -108,8 +112,36 @@ protected void processTable(String tableNameWithType, Context context) {
     if (context._runSegmentLevelValidation) {
       runSegmentLevelValidation(tableConfig, streamConfig);
     }
-    _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
-        context._recreateDeletedConsumingSegment, context._offsetCriteria);
+
+    // Keeps the table paused/unpaused based on storage quota validation.
+    // Skips updating the pause state if table is paused by admin
+    PauseState pauseState = computePauseState(tableNameWithType);
+
+    if (!pauseState.isPaused()) {
+      boolean unPausedUponStorageWithinQuota =
+          pauseState.getReasonCode().equals(PauseState.ReasonCode.STORAGE_QUOTA_EXCEEDED);
+      _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
+          context._recreateDeletedConsumingSegment || unPausedUponStorageWithinQuota, context._offsetCriteria);

Review Comment:
   Let us re-factor this as follows:
   - Create a method called `shouldRestartConsumption()` that returns a boolean
   - Change the original call to `ensureConsuming` to be under an if check based on the return of this method
   - The method `shouldRestartConsumption()` returns false if either one of the conditions hold: (a) the table has been paused by admin (2) the table is out of quota
   
   Agree?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1686745580


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1696,6 +1705,32 @@ private IdealState updatePauseStatusInIdealState(String tableNameWithType, boole
     return updatedIdealState;
   }
 
+  /**
+   * Updates the table IS property 'isQuotaExceeded' based on provided 'quotaExceeded'.
+   * Will be a no op in case the IS already has the same value.
+   * @param tableNameWithType table on which to update the IS
+   * @param quotaExceeded boolean indicating whether table has exceeded the quota limits
+   * @return true if the IS was successfully updated for the table. Returns false in case of no op or the update fails.
+   */
+  public boolean updateStorageQuotaExceededInIdealState(String tableNameWithType, boolean quotaExceeded) {
+    IdealState is = getIdealState(tableNameWithType);
+    if (is.getRecord().getBooleanField(IS_QUOTA_EXCEEDED, false) != quotaExceeded) {
+      is = HelixHelper.updateIdealState(_helixManager, tableNameWithType, idealState -> {
+        ZNRecord znRecord = idealState.getRecord();
+        znRecord.setSimpleField(IS_QUOTA_EXCEEDED, Boolean.valueOf(quotaExceeded).toString());
+        return new IdealState(znRecord);
+      }, RetryPolicies.noDelayRetryPolicy(1));
+      if (is == null) {
+        LOGGER.error("Failed to set 'isQuotaExceeded' to {} in the Ideal State for table {}.", quotaExceeded,

Review Comment:
   That's right.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1692536389


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1696,6 +1705,32 @@ private IdealState updatePauseStatusInIdealState(String tableNameWithType, boole
     return updatedIdealState;
   }
 
+  /**
+   * Updates the table IS property 'isQuotaExceeded' based on provided 'quotaExceeded'.
+   * Will be a no op in case the IS already has the same value.
+   * @param tableNameWithType table on which to update the IS
+   * @param quotaExceeded boolean indicating whether table has exceeded the quota limits
+   * @return true if the IS was successfully updated for the table. Returns false in case of no op or the update fails.
+   */
+  public boolean updateStorageQuotaExceededInIdealState(String tableNameWithType, boolean quotaExceeded) {
+    IdealState is = getIdealState(tableNameWithType);

Review Comment:
   > Notes from discussion with @swaminathanmanish Need to maintain the state for below reasons:
   > 
   > The ideal state update happens only during the point where table storage exceeds the quota, making it a less frequent operation (not affected by segment commit rate)
   > 
   > * All further segment commits will just undergo read operation (IS is already being fetched during segment commit so not additional read is made for quota exceed check).
   > * All prior segment commits will only perform size calculation and no IS operation (reuse already available IS)
   > 
   > Whereas in stateless approach we will keep querying all servers to get the table size for all the further segment commits which feels unnecessary.
   > 
   > The resume consumption flow gets tricky with stateless as we don’t have a pointer on whether the consuming segments were not created due to quota breach or were manually deleted.
   > 
   > * Right now the segment validation task does not recreate consuming segments. Unless we have a context on whether the quota was earlier breached or not, we would end up recreating the missing consuming segment at every run which is not the existing behaviour.
   > * Also this job runs every hour so the IS update operation done by this job won’t be a concern.
   
   Thanks @shounakmk219. This makes sense. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "shounakmk219 (via GitHub)" <gi...@apache.org>.
shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1741836921


##########
pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerGauge.java:
##########
@@ -160,7 +160,8 @@ public enum ControllerGauge implements AbstractMetrics.Gauge {
 
   TABLE_DISABLED("tableDisabled", false),
 
-  TABLE_REBALANCE_IN_PROGRESS("tableRebalanceInProgress", false);
+  TABLE_REBALANCE_IN_PROGRESS("tableRebalanceInProgress", false),
+  TABLE_STORAGE_QUOTA_EXCEEDED("tableStorageQuotaExceeded", true);

Review Comment:
   I have removed the `tableStorageQuotaExceeded` gauge. Will let the downstream system figure out the logic based on the `TABLE_QUOTA` and `TABLE_TOTAL_SIZE_ON_SERVER` gauges.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Storage Quota imposition on Realtime tables [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1681811635


##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/StorageQuotaChecker.java:
##########
@@ -75,9 +80,12 @@ public static QuotaCheckerResponse failure(String msg) {
    * Returns whether the new added segment is within the storage quota.
    */
   public QuotaCheckerResponse isSegmentStorageWithinQuota(TableConfig tableConfig, String segmentName,
-      long segmentSizeInBytes, int timeoutMs)
+      long segmentSizeInBytes)
       throws InvalidConfigException {
-    Preconditions.checkArgument(timeoutMs > 0, "Timeout value must be > 0, input: %s", timeoutMs);
+    if (!_controllerConf.getEnableStorageQuotaCheck()) {
+      return success("Storage quota check is disabled, skipping the check");
+    }
+    Preconditions.checkArgument(_timeout > 0, "Timeout value must be > 0, input: %s", _timeout);

Review Comment:
   Move this check to the constructor?



##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/StorageQuotaChecker.java:
##########
@@ -205,4 +213,48 @@ public QuotaCheckerResponse isSegmentStorageWithinQuota(TableConfig tableConfig,
       return failure(message);
     }
   }
+
+  public boolean isTableStorageQuotaExceeded(TableConfig tableConfig) {

Review Comment:
   Should we also check if quota check is enabled?
   Consider extracting the common part for `isSegmentStorageWithinQuota()` and this method



##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1696,6 +1705,20 @@ private IdealState updatePauseStatusInIdealState(String tableNameWithType, boole
     return updatedIdealState;
   }
 
+  public boolean updateStorageQuotaExceededInIdealState(String tableNameWithType, boolean quotaExceeded) {
+    IdealState is = getIdealState(tableNameWithType);
+    if (is.getRecord().getBooleanField(IS_QUOTA_EXCEEDED, false) != quotaExceeded) {
+      HelixHelper.updateIdealState(_helixManager, tableNameWithType, idealState -> {
+        ZNRecord znRecord = idealState.getRecord();
+        znRecord.setSimpleField(IS_QUOTA_EXCEEDED, Boolean.valueOf(quotaExceeded).toString());
+        return new IdealState(znRecord);
+      }, RetryPolicies.noDelayRetryPolicy(1));

Review Comment:
   Why not allow retry here?



##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -108,8 +110,20 @@ protected void processTable(String tableNameWithType, Context context) {
     if (context._runSegmentLevelValidation) {
       runSegmentLevelValidation(tableConfig, streamConfig);
     }
+
+    // IS is updated in below cases
+    // Case 1 -> Table is exceeding storage quota but IS still has "isQuotaExceeded" as false
+    // This will help prevent consuming segment creation through this task upon manual zk changes to
+    // "isQuotaExceeded" in IS.
+    // Case 2 -> Table is within quota limits now but IS still has "isQuotaExceeded" as true
+    // This will help resume the table consumption once the quota is available due to either quota increase or
+    // segment deletion.
+    // In which case we need to pass "recreateDeletedConsumingSegment" as true to "ensureAllPartitionsConsuming" below.
+    boolean idealStateUpdated = _llcRealtimeSegmentManager.updateStorageQuotaExceededInIdealState(tableNameWithType,
+        _storageQuotaChecker.isTableStorageQuotaExceeded(tableConfig));
+
     _llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig, streamConfig,
-        context._recreateDeletedConsumingSegment, context._offsetCriteria);
+        context._recreateDeletedConsumingSegment || idealStateUpdated, context._offsetCriteria);

Review Comment:
   We should recreate deleted consuming segment only with case 2 above?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org