You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "navina (via GitHub)" <gi...@apache.org> on 2023/02/01 22:53:55 UTC

[GitHub] [pinot] navina opened a new pull request, #10216: Default to segment upload to controller when server segment store uri is missing

navina opened a new pull request, #10216:
URL: https://github.com/apache/pinot/pull/10216

   * When testing server upload to deepstore, we noticed that segment store uri config is not mandatory for the server instance. The upload from server to deep store was failing due to this missing configuration. Moreover, the server logs did not contain any information about why the commit failed and the partition would just hang on the server. 
   This PR:
   * Improves on the error logging 
   * If segment store uri is missing, we will default to using segment upload to controller. 
   * Added some quick fixes to the `StartController` and `StartServer` commands to accept config overrides via command line
   
   
   Labels: `bugfix`
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] mcvsubbu commented on a diff in pull request #10216: Allow segment upload to deepstore only when server segment store uri is configured

Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on code in PR #10216:
URL: https://github.com/apache/pinot/pull/10216#discussion_r1098058410


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java:
##########
@@ -1000,6 +1000,8 @@ protected boolean commitSegment(String controllerVipUrl, boolean isSplitCommit)
     SegmentCompletionProtocol.Response commitResponse = commit(controllerVipUrl, isSplitCommit);
 
     if (!commitResponse.getStatus().equals(SegmentCompletionProtocol.ControllerResponseStatus.COMMIT_SUCCESS)) {
+      _segmentLogger.warn("Controller response was {} and not {}", commitResponse.getStatus(),

Review Comment:
   Please move this log to the place where the return value is detected as not being SUCCESS. We already have other logs for non-success cases in the layers below. 



##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/PinotFSSegmentUploader.java:
##########
@@ -54,6 +54,8 @@ public PinotFSSegmentUploader(String segmentStoreDirUri, int timeoutMillis) {
 
   public URI uploadSegment(File segmentFile, LLCSegmentName segmentName) {
     if (_segmentStoreUriStr == null || _segmentStoreUriStr.isEmpty()) {
+      LOGGER.error("Missing segment store uri. Failed to upload segment file {} for {}.", segmentFile.getName(),

Review Comment:
   This missing log should suffice? You don't need the additional log above in the segment data manager?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang merged pull request #10216: Allow segment upload to deepstore only when server segment store uri is configured

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #10216:
URL: https://github.com/apache/pinot/pull/10216


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] navina commented on a diff in pull request #10216: Allow segment upload to deepstore only when server segment store uri is configured

Posted by "navina (via GitHub)" <gi...@apache.org>.
navina commented on code in PR #10216:
URL: https://github.com/apache/pinot/pull/10216#discussion_r1097918289


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitterFactory.java:
##########
@@ -69,9 +70,13 @@ public SegmentCommitter createSegmentCommitter(boolean isSplitCommit, SegmentCom
 
     boolean uploadToFs = _streamConfig.isServerUploadToDeepStore();
     String peerSegmentDownloadScheme = _tableConfig.getValidationConfig().getPeerSegmentDownloadScheme();
-    // TODO: exists for backwards compatibility. remove peerDownloadScheme non-null check once users have migrated
-    if (uploadToFs || peerSegmentDownloadScheme != null) {
-      segmentUploader = new PinotFSSegmentUploader(_indexLoadingConfig.getSegmentStoreURI(),
+    String segmentStoreUri = _indexLoadingConfig.getSegmentStoreURI();
+
+    // We seem to allow the server instance to come up without a valid segment store uri. Hence, this check is needed.
+
+    if (peerSegmentDownloadScheme != null || (!StringUtils.isEmpty(segmentStoreUri) && uploadToFs)) {

Review Comment:
   ah true. Will fix it. ty



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] navina commented on a diff in pull request #10216: Allow segment upload to deepstore only when server segment store uri is configured

Posted by "navina (via GitHub)" <gi...@apache.org>.
navina commented on code in PR #10216:
URL: https://github.com/apache/pinot/pull/10216#discussion_r1096255028


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitterFactory.java:
##########
@@ -69,9 +70,13 @@ public SegmentCommitter createSegmentCommitter(boolean isSplitCommit, SegmentCom
 
     boolean uploadToFs = _streamConfig.isServerUploadToDeepStore();
     String peerSegmentDownloadScheme = _tableConfig.getValidationConfig().getPeerSegmentDownloadScheme();
-    // TODO: exists for backwards compatibility. remove peerDownloadScheme non-null check once users have migrated
-    if (uploadToFs || peerSegmentDownloadScheme != null) {
-      segmentUploader = new PinotFSSegmentUploader(_indexLoadingConfig.getSegmentStoreURI(),
+    String segmentStoreUri = _indexLoadingConfig.getSegmentStoreURI();
+
+    // We seem to allow the server instance to come up without a valid segment store uri. Hence, this check is needed.
+    if (!Strings.isNullOrEmpty(segmentStoreUri)

Review Comment:
   Fixed based on our offline discussion. Modified to not default to controller upload. Instead added config validation before table data manager is created. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] navina commented on pull request #10216: Default to segment upload to controller when server segment store uri is missing

Posted by "navina (via GitHub)" <gi...@apache.org>.
navina commented on PR #10216:
URL: https://github.com/apache/pinot/pull/10216#issuecomment-1412853652

   @Jackie-Jiang : review pls.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10216: Default to segment upload to controller when server segment store uri is missing

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #10216:
URL: https://github.com/apache/pinot/pull/10216#discussion_r1095085905


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitterFactory.java:
##########
@@ -69,9 +70,13 @@ public SegmentCommitter createSegmentCommitter(boolean isSplitCommit, SegmentCom
 
     boolean uploadToFs = _streamConfig.isServerUploadToDeepStore();
     String peerSegmentDownloadScheme = _tableConfig.getValidationConfig().getPeerSegmentDownloadScheme();
-    // TODO: exists for backwards compatibility. remove peerDownloadScheme non-null check once users have migrated
-    if (uploadToFs || peerSegmentDownloadScheme != null) {
-      segmentUploader = new PinotFSSegmentUploader(_indexLoadingConfig.getSegmentStoreURI(),
+    String segmentStoreUri = _indexLoadingConfig.getSegmentStoreURI();
+
+    // We seem to allow the server instance to come up without a valid segment store uri. Hence, this check is needed.
+    if (!Strings.isNullOrEmpty(segmentStoreUri)

Review Comment:
   (minor) We usually use `StringUtils.isEmpty()` (from `org.apache.commons.lang3`)



##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitterFactory.java:
##########
@@ -69,9 +70,13 @@ public SegmentCommitter createSegmentCommitter(boolean isSplitCommit, SegmentCom
 
     boolean uploadToFs = _streamConfig.isServerUploadToDeepStore();
     String peerSegmentDownloadScheme = _tableConfig.getValidationConfig().getPeerSegmentDownloadScheme();
-    // TODO: exists for backwards compatibility. remove peerDownloadScheme non-null check once users have migrated
-    if (uploadToFs || peerSegmentDownloadScheme != null) {
-      segmentUploader = new PinotFSSegmentUploader(_indexLoadingConfig.getSegmentStoreURI(),
+    String segmentStoreUri = _indexLoadingConfig.getSegmentStoreURI();
+
+    // We seem to allow the server instance to come up without a valid segment store uri. Hence, this check is needed.
+    if (!Strings.isNullOrEmpty(segmentStoreUri)

Review Comment:
   (MAJOR) When segment upload fails, but peer download is enabled, we still want to use the `PinotFSSegmentUploader` because peer download doesn't require segment to be successfully uploaded. We can attach this check to the `uploadToFs`



##########
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java:
##########
@@ -111,6 +112,10 @@ public synchronized void init(PinotConfiguration config, HelixManager helixManag
     _instanceId = _instanceDataManagerConfig.getInstanceId();
     _helixManager = helixManager;
     _serverMetrics = serverMetrics;
+    if (Strings.isNullOrEmpty(_instanceDataManagerConfig.getSegmentStoreUri())) {

Review Comment:
   (minor) We usually use `StringUtils.isEmpty()`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10216: Allow segment upload to deepstore only when server segment store uri is configured

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #10216:
URL: https://github.com/apache/pinot/pull/10216#discussion_r1097843619


##########
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java:
##########
@@ -111,6 +112,10 @@ public synchronized void init(PinotConfiguration config, HelixManager helixManag
     _instanceId = _instanceDataManagerConfig.getInstanceId();
     _helixManager = helixManager;
     _serverMetrics = serverMetrics;
+    if (StringUtils.isEmpty(_instanceDataManagerConfig.getSegmentStoreUri())) {

Review Comment:
   Suggest not adding this check because this is not mandatory config



##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/TableDataManagerProvider.java:
##########
@@ -72,6 +77,15 @@ public static TableDataManager getTableDataManager(TableDataManagerConfig tableD
         }
         break;
       case REALTIME:
+        Map<String, String> streamConfigMap = IngestionConfigUtils.getStreamConfigMap(
+            tableDataManagerConfig.getTableConfig());
+        if (Boolean.parseBoolean(streamConfigMap.getOrDefault(StreamConfigProperties.SERVER_UPLOAD_TO_DEEPSTORE,
+            "false"))

Review Comment:
   (nit)
   ```suggestion
           if (Boolean.parseBoolean(streamConfigMap.get(StreamConfigProperties.SERVER_UPLOAD_TO_DEEPSTORE))
   ```



##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitterFactory.java:
##########
@@ -69,9 +70,13 @@ public SegmentCommitter createSegmentCommitter(boolean isSplitCommit, SegmentCom
 
     boolean uploadToFs = _streamConfig.isServerUploadToDeepStore();
     String peerSegmentDownloadScheme = _tableConfig.getValidationConfig().getPeerSegmentDownloadScheme();
-    // TODO: exists for backwards compatibility. remove peerDownloadScheme non-null check once users have migrated
-    if (uploadToFs || peerSegmentDownloadScheme != null) {
-      segmentUploader = new PinotFSSegmentUploader(_indexLoadingConfig.getSegmentStoreURI(),
+    String segmentStoreUri = _indexLoadingConfig.getSegmentStoreURI();
+
+    // We seem to allow the server instance to come up without a valid segment store uri. Hence, this check is needed.
+
+    if (peerSegmentDownloadScheme != null || (!StringUtils.isEmpty(segmentStoreUri) && uploadToFs)) {

Review Comment:
   We don't need this extra check because we already checked at table creation time



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #10216: Default to segment upload to controller when server segment store uri is missing

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10216:
URL: https://github.com/apache/pinot/pull/10216#issuecomment-1413212145

   # [Codecov](https://codecov.io/gh/apache/pinot/pull/10216?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#10216](https://codecov.io/gh/apache/pinot/pull/10216?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9ac08ae) into [master](https://codecov.io/gh/apache/pinot/commit/76a38bffab2530ca427e1ee281d50056a6b90633?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (76a38bf) will **decrease** coverage by `56.71%`.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #10216       +/-   ##
   =============================================
   - Coverage     70.40%   13.70%   -56.71%     
   + Complexity     5756      178     -5578     
   =============================================
     Files          2016     1961       -55     
     Lines        109100   106663     -2437     
     Branches      16562    16281      -281     
   =============================================
   - Hits          76815    14614    -62201     
   - Misses        26911    90898    +63987     
   + Partials       5374     1151     -4223     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `13.70% <0.00%> (-0.02%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/10216?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...manager/realtime/LLRealtimeSegmentDataManager.java](https://codecov.io/gh/apache/pinot/pull/10216?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvTExSZWFsdGltZVNlZ21lbnREYXRhTWFuYWdlci5qYXZh) | `0.00% <0.00%> (-71.10%)` | :arrow_down: |
   | [.../data/manager/realtime/PinotFSSegmentUploader.java](https://codecov.io/gh/apache/pinot/pull/10216?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUGlub3RGU1NlZ21lbnRVcGxvYWRlci5qYXZh) | `0.00% <0.00%> (-83.88%)` | :arrow_down: |
   | [...data/manager/realtime/SegmentCommitterFactory.java](https://codecov.io/gh/apache/pinot/pull/10216?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VnbWVudENvbW1pdHRlckZhY3RvcnkuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...src/main/java/org/apache/pinot/sql/FilterKind.java](https://codecov.io/gh/apache/pinot/pull/10216?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvRmlsdGVyS2luZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ain/java/org/apache/pinot/core/data/table/Key.java](https://codecov.io/gh/apache/pinot/pull/10216?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0tleS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...in/java/org/apache/pinot/spi/utils/BytesUtils.java](https://codecov.io/gh/apache/pinot/pull/10216?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQnl0ZXNVdGlscy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ain/java/org/apache/pinot/common/CustomObject.java](https://codecov.io/gh/apache/pinot/pull/10216?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vQ3VzdG9tT2JqZWN0LmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...n/java/org/apache/pinot/core/data/table/Table.java](https://codecov.io/gh/apache/pinot/pull/10216?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1RhYmxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/core/data/table/Record.java](https://codecov.io/gh/apache/pinot/pull/10216?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1JlY29yZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/core/util/GroupByUtils.java](https://codecov.io/gh/apache/pinot/pull/10216?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS91dGlsL0dyb3VwQnlVdGlscy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1587 more](https://codecov.io/gh/apache/pinot/pull/10216?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] navina commented on a diff in pull request #10216: Default to segment upload to controller when server segment store uri is missing

Posted by "navina (via GitHub)" <gi...@apache.org>.
navina commented on code in PR #10216:
URL: https://github.com/apache/pinot/pull/10216#discussion_r1095143970


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitterFactory.java:
##########
@@ -69,9 +70,13 @@ public SegmentCommitter createSegmentCommitter(boolean isSplitCommit, SegmentCom
 
     boolean uploadToFs = _streamConfig.isServerUploadToDeepStore();
     String peerSegmentDownloadScheme = _tableConfig.getValidationConfig().getPeerSegmentDownloadScheme();
-    // TODO: exists for backwards compatibility. remove peerDownloadScheme non-null check once users have migrated
-    if (uploadToFs || peerSegmentDownloadScheme != null) {
-      segmentUploader = new PinotFSSegmentUploader(_indexLoadingConfig.getSegmentStoreURI(),
+    String segmentStoreUri = _indexLoadingConfig.getSegmentStoreURI();
+
+    // We seem to allow the server instance to come up without a valid segment store uri. Hence, this check is needed.
+    if (!Strings.isNullOrEmpty(segmentStoreUri)

Review Comment:
   > (MAJOR) When segment upload fails, but peer download is enabled, we still want to use the `PinotFSSegmentUploader` because peer download doesn't require segment to be successfully uploaded. We can attach this check to the `uploadToFs`
   
   This PR doesn't really change anything with respect to upload failure handling. These configs only control what uploader is being used. If you see SplitSegmentCommitter [here](https://github.com/apache/pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SplitSegmentCommitter.java#L99), this part handles the upload failure in the presence of a peer download scheme. The behavior is the same as it was before this PR. 
   
   Note: It might be cleaner to have a failure handler provided as the committer context. but that still doesn't change any existing behavior



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org