You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "rseetham (via GitHub)" <gi...@apache.org> on 2025/03/24 16:11:47 UTC

[PR] Allow controller to skip PinotFS setup test during boot [pinot]

rseetham opened a new pull request, #15358:
URL: https://github.com/apache/pinot/pull/15358

   This is to fix the issue #15127
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Allow controller to skip PinotFS setup test during boot [pinot]

Posted by "rseetham (via GitHub)" <gi...@apache.org>.
rseetham commented on code in PR #15358:
URL: https://github.com/apache/pinot/pull/15358#discussion_r2044895593


##########
pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java:
##########
@@ -72,6 +72,10 @@ public class ControllerConf extends PinotConfiguration {
   public static final String LEAD_CONTROLLER_RESOURCE_REBALANCE_STRATEGY = "controller.resource.rebalance.strategy";
   public static final String LEAD_CONTROLLER_RESOURCE_REBALANCE_DELAY_MS = "controller.resource.rebalance.delay_ms";
 
+  //boolean Check if dataDir is avaiable on boot?
+  public static final String SKIP_DATA_DIR_VALIDATION_ON_BOOT = "controller.skip_data_dir_validation_on_boot";

Review Comment:
   addressed all comments now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Allow controller to skip PinotFS setup test during boot [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #15358:
URL: https://github.com/apache/pinot/pull/15358#issuecomment-2748800153

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15358?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `14.28571%` with `6 lines` in your changes missing coverage. Please review.
   > Project coverage is 63.49%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`0186a9d`)](https://app.codecov.io/gh/apache/pinot/commit/0186a9dce9eb2bf8bc306d34e9af0bb141b9b197?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 1896 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/15358?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...ller/api/resources/ControllerFilePathProvider.java](https://app.codecov.io/gh/apache/pinot/pull/15358?src=pr&el=tree&filepath=pinot-controller%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcontroller%2Fapi%2Fresources%2FControllerFilePathProvider.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL0NvbnRyb2xsZXJGaWxlUGF0aFByb3ZpZGVyLmphdmE=) | 0.00% | [5 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15358?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #15358      +/-   ##
   ============================================
   + Coverage     61.75%   63.49%   +1.74%     
   - Complexity      207     1369    +1162     
   ============================================
     Files          2436     2784     +348     
     Lines        133233   157033   +23800     
     Branches      20636    24097    +3461     
   ============================================
   + Hits          82274    99707   +17433     
   - Misses        44911    49795    +4884     
   - Partials       6048     7531    +1483     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/15358/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/15358/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/15358/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/15358/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/15358/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/15358/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/15358/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.49% <14.28%> (+1.86%)` | :arrow_up: |
   | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/15358/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.48% <14.28%> (+1.73%)` | :arrow_up: |
   | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/15358/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.47% <14.28%> (+35.74%)` | :arrow_up: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/15358/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.49% <14.28%> (+1.74%)` | :arrow_up: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/15358/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.49% <14.28%> (+1.74%)` | :arrow_up: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/15358/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.98% <ø> (+9.08%)` | :arrow_up: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/15358/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.20% <14.28%> (+6.47%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/15358?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Allow controller to skip PinotFS setup test during boot [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #15358:
URL: https://github.com/apache/pinot/pull/15358#discussion_r2049554330


##########
pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java:
##########
@@ -72,6 +72,10 @@ public class ControllerConf extends PinotConfiguration {
   public static final String LEAD_CONTROLLER_RESOURCE_REBALANCE_STRATEGY = "controller.resource.rebalance.strategy";
   public static final String LEAD_CONTROLLER_RESOURCE_REBALANCE_DELAY_MS = "controller.resource.rebalance.delay_ms";
 
+  //boolean Check if dataDir is avaiable on boot?
+  public static final String CONTINUE_WITHOUT_DEEP_STORE = "controller.startup.continueWithoutDeepStore";
+

Review Comment:
   (minor) Remove one empty line



##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ControllerFilePathProvider.java:
##########
@@ -66,13 +66,23 @@ private ControllerFilePathProvider(ControllerConf controllerConf)
     try {
       _dataDirURI = URIUtils.getUri(dataDir);
       LOGGER.info("Data directory: {}", _dataDirURI);
-
-      PinotFS pinotFS = PinotFSFactory.create(_dataDirURI.getScheme());
-      if (pinotFS.exists(_dataDirURI)) {
-        Preconditions
-            .checkState(pinotFS.isDirectory(_dataDirURI), "Data directory: %s must be a directory", _dataDirURI);
-      } else {
-        Preconditions.checkState(pinotFS.mkdir(_dataDirURI), "Failed to create data directory: %s", _dataDirURI);
+      try {
+          PinotFS pinotFS = PinotFSFactory.create(_dataDirURI.getScheme());

Review Comment:
   (minor) Reformat the code with [Pinot Style](https://docs.pinot.apache.org/developers/developers-and-contributors/code-setup#set-up-ide)



##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ControllerFilePathProvider.java:
##########
@@ -66,13 +66,23 @@ private ControllerFilePathProvider(ControllerConf controllerConf)
     try {
       _dataDirURI = URIUtils.getUri(dataDir);
       LOGGER.info("Data directory: {}", _dataDirURI);
-
-      PinotFS pinotFS = PinotFSFactory.create(_dataDirURI.getScheme());
-      if (pinotFS.exists(_dataDirURI)) {
-        Preconditions
-            .checkState(pinotFS.isDirectory(_dataDirURI), "Data directory: %s must be a directory", _dataDirURI);
-      } else {
-        Preconditions.checkState(pinotFS.mkdir(_dataDirURI), "Failed to create data directory: %s", _dataDirURI);
+      try {
+          PinotFS pinotFS = PinotFSFactory.create(_dataDirURI.getScheme());
+          if (pinotFS.exists(_dataDirURI)) {
+            Preconditions

Review Comment:
   We can move the `controllerConf.continueWithoutDeepStore()` check under this branch



##########
pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java:
##########
@@ -538,6 +542,10 @@ public boolean isUpdateSegmentStateModel() {
     return getProperty(UPDATE_SEGMENT_STATE_MODEL, false);
   }
 
+  public boolean continueWithoutDeepStore() {

Review Comment:
   (minor)
   ```suggestion
     public boolean isContinueWithoutDeepStore() {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Allow controller to skip PinotFS setup test during boot [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #15358:
URL: https://github.com/apache/pinot/pull/15358#discussion_r2017443281


##########
pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java:
##########
@@ -72,6 +72,10 @@ public class ControllerConf extends PinotConfiguration {
   public static final String LEAD_CONTROLLER_RESOURCE_REBALANCE_STRATEGY = "controller.resource.rebalance.strategy";
   public static final String LEAD_CONTROLLER_RESOURCE_REBALANCE_DELAY_MS = "controller.resource.rebalance.delay_ms";
 
+  //boolean Check if dataDir is avaiable on boot?
+  public static final String SKIP_DATA_DIR_VALIDATION_ON_BOOT = "controller.skip_data_dir_validation_on_boot";

Review Comment:
   ```suggestion
     public static final String CONTINUE_WITHOUT_DEEP_STORE = "controller.startup.continueWithoutDeepStore";
   ```



##########
pinot-controller/package.json:
##########
@@ -0,0 +1,11 @@
+{

Review Comment:
   Is this related?



##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/ControllerFilePathProvider.java:
##########
@@ -66,13 +66,14 @@ private ControllerFilePathProvider(ControllerConf controllerConf)
     try {
       _dataDirURI = URIUtils.getUri(dataDir);
       LOGGER.info("Data directory: {}", _dataDirURI);
-
-      PinotFS pinotFS = PinotFSFactory.create(_dataDirURI.getScheme());
-      if (pinotFS.exists(_dataDirURI)) {
-        Preconditions
-            .checkState(pinotFS.isDirectory(_dataDirURI), "Data directory: %s must be a directory", _dataDirURI);
-      } else {
-        Preconditions.checkState(pinotFS.mkdir(_dataDirURI), "Failed to create data directory: %s", _dataDirURI);
+      if (controllerConf.skipDataDirValidationOnBoot()) {

Review Comment:
   Let's still try to create the data dir, but log a warning instead when this is set to true



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Allow controller to skip PinotFS setup test during boot [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #15358:
URL: https://github.com/apache/pinot/pull/15358


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org