You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/06/23 23:49:58 UTC

[GitHub] [incubator-pinot] mqliang opened a new pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

mqliang opened a new pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086


   ## Description
   <!-- Add a description of your PR here.
   A good description should include pointers to an issue or design document, etc.
   -->
   
   This PR check segment CRC before start a SegmentConversion task. Without this logic, minion will start the task blindly and fail when uploading new segment, which will make minion do unnecessary works and increase the waiting time of other tasks.
   
   cc @mcvsubbu @jackjlli 
   
   ## Upgrade Notes
   Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)
   * [ ] Yes (Please label as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
   
   Does this PR fix a zero-downtime upgrade introduced earlier?
   * [ ] Yes (Please label this as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
   
   Does this PR otherwise need attention when creating release notes? Things to consider:
   - New configuration options
   - Deprecation of configurations
   - Signature changes to public methods/interfaces
   - New plugins added or old plugins removed
   * [ ] Yes (Please label this PR as **<code>release-notes</code>** and complete the section on Release Notes)
   ## Release Notes
   <!-- If you have tagged this as either backward-incompat or release-notes,
   you MUST add text here that you would like to see appear in release notes of the
   next release. -->
   
   <!-- If you have a series of commits adding or enabling a feature, then
   add this section only in final commit that marks the feature completed.
   Refer to earlier release notes to see examples of text.
   -->
   ## Documentation
   <!-- If you have introduced a new feature or configuration, please add it to the documentation as well.
   See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r658357255



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +71,14 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      throw new FatalException(taskType + " on table: " + tableNameWithType + ", segment: " + segmentName

Review comment:
       Why are you throwing an exception here? The job will be treated as an ERROR case, when it is not an error. We are just optimizing saying that this is done.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#issuecomment-867256423


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7086](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b3f4120) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/80fda686272d52f74057588d95faefbc016bca9f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (80fda68) will **decrease** coverage by `8.10%`.
   > The diff coverage is `47.36%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/7086/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7086      +/-   ##
   ============================================
   - Coverage     73.57%   65.46%   -8.11%     
     Complexity       91       91              
   ============================================
     Files          1486     1490       +4     
     Lines         73009    73163     +154     
     Branches      10492    10528      +36     
   ============================================
   - Hits          53713    47893    -5820     
   - Misses        15818    21883    +6065     
   + Partials       3478     3387      -91     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration | `?` | |
   | unittests | `65.46% <47.36%> (-0.08%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `10.69% <0.00%> (-57.68%)` | :arrow_down: |
   | [...ion/tasks/BaseSingleSegmentConversionExecutor.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVNpbmdsZVNlZ21lbnRDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `3.92% <0.00%> (-89.56%)` | :arrow_down: |
   | [...he/pinot/plugin/minion/tasks/BaseTaskExecutor.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVRhc2tFeGVjdXRvci5qYXZh) | `47.05% <0.00%> (-36.28%)` | :arrow_down: |
   | [...ocal/indexsegment/immutable/EmptyIndexSegment.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0VtcHR5SW5kZXhTZWdtZW50LmphdmE=) | `36.84% <ø> (ø)` | |
   | [...ocal/indexsegment/mutable/IntermediateSegment.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvbXV0YWJsZS9JbnRlcm1lZGlhdGVTZWdtZW50LmphdmE=) | `70.06% <ø> (ø)` | |
   | [...local/realtime/impl/geospatial/MutableH3Index.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2dlb3NwYXRpYWwvTXV0YWJsZUgzSW5kZXguamF2YQ==) | `93.33% <ø> (ø)` | |
   | [...mpl/invertedindex/RealtimeInvertedIndexReader.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2ludmVydGVkaW5kZXgvUmVhbHRpbWVJbnZlcnRlZEluZGV4UmVhZGVyLmphdmE=) | `100.00% <ø> (ø)` | |
   | [...ealtime/impl/nullvalue/MutableNullValueVector.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL251bGx2YWx1ZS9NdXRhYmxlTnVsbFZhbHVlVmVjdG9yLmphdmE=) | `85.71% <ø> (-14.29%)` | :arrow_down: |
   | [...segment/index/readers/ValidDocIndexReaderImpl.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvVmFsaWREb2NJbmRleFJlYWRlckltcGwuamF2YQ==) | `0.00% <ø> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/segment/spi/IndexSegment.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL0luZGV4U2VnbWVudC5qYXZh) | `0.00% <ø> (ø)` | |
   | ... and [383 more](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f4e5518...b3f4120](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mqliang commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r658327196



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +70,14 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      return new SegmentConversionResult.Builder().setTableNameWithType(tableNameWithType).setSegmentName(segmentName)
+          .build();

Review comment:
       Helix expose following result status for external usage:
   ```
   public class TaskResult {
     /**
      * An enumeration of status codes.
      */
     public enum Status {
       /** The task completed normally. */
       COMPLETED,
       /**
        * The task was cancelled externally, i.e. {@link org.apache.helix.task.Task#cancel()} was
        * called.
        */
       CANCELED,
       /** The task encountered an error from which it can not recover.
        * This is equivalent to {@link org.apache.helix.task.TaskResult.Status#FAILED}.*/
       @Deprecated
       ERROR,
       /** The task encountered an error which can not be recovered from this run, but it may still succeed by retrying the task. */
       FAILED,
       /** The task encountered an error, which will not be recoverable even with retrying the task */
       FATAL_FAILED
     }
   ```
   
   Helix does have an `TASK_ABORTED ` in it's state model, but it does not exposed in TaskResult:
   ```
   public enum TaskPartitionState {
     /** The initial state of the state model. */
     INIT,
     /** Indicates that the task is currently running. */
     RUNNING,
     /** Indicates that the task was stopped by the controller. */
     STOPPED,
     /** Indicates that the task completed normally. */
     COMPLETED,
     /** Indicates that the task timed out. */
     TIMED_OUT,
     /** Indicates an error occurred during task execution, but the task can be retried. */
     TASK_ERROR,
     /** Indicates an error occurred during task execution, and the task should not be retried. */
     TASK_ABORTED,
     /** Helix's own internal error state. */
     ERROR,
     /** A Helix internal state. */
     DROPPED
   }
   ```
   We need change Helix code if we want mark the task as `Aborted `
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
jackjlli commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r657534082



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseTaskExecutor.java
##########
@@ -56,4 +57,12 @@ protected Schema getSchema(String tableName) {
     Preconditions.checkState(schema != null, "Failed to find schema for table: %s", tableName);
     return schema;
   }
+
+  protected long getSegmentCrc(String tableNameWithType, String segmentName) {
+    OfflineSegmentZKMetadata segmentZKMetadata = ZKMetadataProvider
+        .getOfflineSegmentZKMetadata(MINION_CONTEXT.getHelixPropertyStore(), tableNameWithType, segmentName);

Review comment:
       It doesn't have to be an offline segment. Might be good to check the table type first?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-commenter commented on pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#issuecomment-867256423


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > :exclamation: No coverage uploaded for pull request base (`master@f4e5518`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#section-missing-base-commit).
   > The diff coverage is `n/a`.
   
   > :exclamation: Current head 5cef2fb differs from pull request most recent head 2862b13. Consider uploading reports for the commit 2862b13 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/7086/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff            @@
   ##             master    #7086   +/-   ##
   =========================================
     Coverage          ?   65.47%           
     Complexity        ?       91           
   =========================================
     Files             ?     1490           
     Lines             ?    73163           
     Branches          ?    10527           
   =========================================
     Hits              ?    47901           
     Misses            ?    21874           
     Partials          ?     3388           
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | unittests | `65.47% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f4e5518...2862b13](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mqliang commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r658297466



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +70,14 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      return new SegmentConversionResult.Builder().setTableNameWithType(tableNameWithType).setSegmentName(segmentName)
+          .build();

Review comment:
       @mcvsubbu @jackjlli 
   By return a `new SegmentConversionResult` here, task will been marked as completed. We can also throw `TaskCancelledException` here so that task will been marked as "cancelled", or `FatalException` so that task will been marked as"failed". Any thoughts?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#issuecomment-867256423


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > :exclamation: No coverage uploaded for pull request base (`master@f4e5518`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#section-missing-base-commit).
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/7086/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff            @@
   ##             master    #7086   +/-   ##
   =========================================
     Coverage          ?   73.55%           
     Complexity        ?       91           
   =========================================
     Files             ?     1490           
     Lines             ?    73165           
     Branches          ?    10528           
   =========================================
     Hits              ?    53817           
     Misses            ?    15843           
     Partials          ?     3505           
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration | `41.75% <ø> (?)` | |
   | unittests | `65.45% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f4e5518...3860df4](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r657532660



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +70,13 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      new SegmentConversionResult.Builder().setTableNameWithType(tableNameWithType).setSegmentName(segmentName).build();

Review comment:
       Missing a return?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mqliang commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r659971793



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +71,14 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      throw new FatalException(taskType + " on table: " + tableNameWithType + ", segment: " + segmentName

Review comment:
       updated




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mqliang commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r658335970



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +70,14 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      return new SegmentConversionResult.Builder().setTableNameWithType(tableNameWithType).setSegmentName(segmentName)
+          .build();

Review comment:
       After reading the helix code carefully, `TaskResult.FAILED` is equivalent to `TaskPartitionState.TASK_ERROR `, `TaskResult.FATAL_FAILED` is equivalent to `TaskPartitionState.TASK_ABORTED`. So `FATAL_FAILED ` seems the best choice for us, it does not change the current behavior: if we don't check CRC first, the task will turn into `FATAL_FAILED ` when upload anyway.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r657604714



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseTaskExecutor.java
##########
@@ -56,4 +59,15 @@ protected Schema getSchema(String tableName) {
     Preconditions.checkState(schema != null, "Failed to find schema for table: %s", tableName);
     return schema;
   }
+
+  protected long getSegmentCrc(String tableNameWithType, String segmentName) {
+    TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+    SegmentZKMetadata segmentZKMetadata = tableType == TableType.OFFLINE ? ZKMetadataProvider
+        .getOfflineSegmentZKMetadata(MINION_CONTEXT.getHelixPropertyStore(), tableNameWithType, segmentName)
+        : ZKMetadataProvider
+            .getRealtimeSegmentZKMetadata(MINION_CONTEXT.getHelixPropertyStore(), tableNameWithType, segmentName);
+    Preconditions.checkState(segmentZKMetadata != null, "Failed to find segment metadata for table: %s, segment: %s",

Review comment:
       Do not add this as pre-condition. Just log a warning and return success. If the metadata is not there, it is likely that the segment has been deleted. If you throw exception, helix will keep retrying this forever.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r659980113



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +70,14 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      return new SegmentConversionResult.Builder().setTableNameWithType(tableNameWithType).setSegmentName(segmentName)
+          .build();

Review comment:
       If the task returns a failure, helix will keep retrying it, and we don't want that. Best is to return Success




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mqliang commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r658335970



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +70,14 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      return new SegmentConversionResult.Builder().setTableNameWithType(tableNameWithType).setSegmentName(segmentName)
+          .build();

Review comment:
       After reading the helix code carefully, `TaskResult.FAILED` is equivalent to `TaskPartitionState.TASK_ERROR `, `TaskResult.FATAL_FAILED` is equivalent to `TaskPartitionState.TASK_ABORTED`. So `FATAL_FAILED ` seems the best choice for us, if we don't check CRC first, the task will be in `FATAL_FAILED ` when uploading anyway.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mqliang commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r658297466



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +70,14 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      return new SegmentConversionResult.Builder().setTableNameWithType(tableNameWithType).setSegmentName(segmentName)
+          .build();

Review comment:
       @mcvsubbu @jackjlli 
   By return an `new SegmentConversionResult` here, task will been marked as completed. We can also throw `TaskCancelledException` here so that task will been marked as "cancelled", or `FatalException` so that task will been marked as"failed". Any thoughts?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#issuecomment-867256423


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7086](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ce910ee) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/80fda686272d52f74057588d95faefbc016bca9f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (80fda68) will **decrease** coverage by `8.12%`.
   > The diff coverage is `47.16%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/7086/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7086      +/-   ##
   ============================================
   - Coverage     73.57%   65.44%   -8.13%     
     Complexity       91       91              
   ============================================
     Files          1486     1490       +4     
     Lines         73009    73164     +155     
     Branches      10492    10528      +36     
   ============================================
   - Hits          53713    47884    -5829     
   - Misses        15818    21889    +6071     
   + Partials       3478     3391      -87     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration | `?` | |
   | unittests | `65.44% <47.16%> (-0.09%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `10.69% <0.00%> (-57.68%)` | :arrow_down: |
   | [...ion/tasks/BaseSingleSegmentConversionExecutor.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVNpbmdsZVNlZ21lbnRDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `3.84% <0.00%> (-89.64%)` | :arrow_down: |
   | [...he/pinot/plugin/minion/tasks/BaseTaskExecutor.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVRhc2tFeGVjdXRvci5qYXZh) | `47.05% <0.00%> (-36.28%)` | :arrow_down: |
   | [...ocal/indexsegment/immutable/EmptyIndexSegment.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0VtcHR5SW5kZXhTZWdtZW50LmphdmE=) | `36.84% <ø> (ø)` | |
   | [...ocal/indexsegment/mutable/IntermediateSegment.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvbXV0YWJsZS9JbnRlcm1lZGlhdGVTZWdtZW50LmphdmE=) | `70.06% <ø> (ø)` | |
   | [...local/realtime/impl/geospatial/MutableH3Index.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2dlb3NwYXRpYWwvTXV0YWJsZUgzSW5kZXguamF2YQ==) | `93.33% <ø> (ø)` | |
   | [...mpl/invertedindex/RealtimeInvertedIndexReader.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2ludmVydGVkaW5kZXgvUmVhbHRpbWVJbnZlcnRlZEluZGV4UmVhZGVyLmphdmE=) | `100.00% <ø> (ø)` | |
   | [...ealtime/impl/nullvalue/MutableNullValueVector.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL251bGx2YWx1ZS9NdXRhYmxlTnVsbFZhbHVlVmVjdG9yLmphdmE=) | `85.71% <ø> (-14.29%)` | :arrow_down: |
   | [...segment/index/readers/ValidDocIndexReaderImpl.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvVmFsaWREb2NJbmRleFJlYWRlckltcGwuamF2YQ==) | `0.00% <ø> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/segment/spi/IndexSegment.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL0luZGV4U2VnbWVudC5qYXZh) | `0.00% <ø> (ø)` | |
   | ... and [382 more](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f4e5518...ce910ee](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mqliang commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r657534211



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseTaskExecutor.java
##########
@@ -56,4 +57,12 @@ protected Schema getSchema(String tableName) {
     Preconditions.checkState(schema != null, "Failed to find schema for table: %s", tableName);
     return schema;
   }
+
+  protected long getSegmentCrc(String tableNameWithType, String segmentName) {
+    OfflineSegmentZKMetadata segmentZKMetadata = ZKMetadataProvider

Review comment:
       I am assuming BaseSingleSegmentConversionExecutor are only used by Offline type table. Do we need add support for realtime case? @mcvsubbu @jackjlli 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mqliang commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r657591721



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseTaskExecutor.java
##########
@@ -56,4 +57,12 @@ protected Schema getSchema(String tableName) {
     Preconditions.checkState(schema != null, "Failed to find schema for table: %s", tableName);
     return schema;
   }
+
+  protected long getSegmentCrc(String tableNameWithType, String segmentName) {
+    OfflineSegmentZKMetadata segmentZKMetadata = ZKMetadataProvider
+        .getOfflineSegmentZKMetadata(MINION_CONTEXT.getHelixPropertyStore(), tableNameWithType, segmentName);

Review comment:
       done

##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseTaskExecutor.java
##########
@@ -56,4 +57,12 @@ protected Schema getSchema(String tableName) {
     Preconditions.checkState(schema != null, "Failed to find schema for table: %s", tableName);
     return schema;
   }
+
+  protected long getSegmentCrc(String tableNameWithType, String segmentName) {
+    OfflineSegmentZKMetadata segmentZKMetadata = ZKMetadataProvider

Review comment:
       done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r659980113



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +70,14 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      return new SegmentConversionResult.Builder().setTableNameWithType(tableNameWithType).setSegmentName(segmentName)
+          .build();

Review comment:
       If the task returns a failure, helix will keep retrying it, and we don't want that. Best is to return Success




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
jackjlli commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r657535651



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseTaskExecutor.java
##########
@@ -56,4 +57,12 @@ protected Schema getSchema(String tableName) {
     Preconditions.checkState(schema != null, "Failed to find schema for table: %s", tableName);
     return schema;
   }
+
+  protected long getSegmentCrc(String tableNameWithType, String segmentName) {
+    OfflineSegmentZKMetadata segmentZKMetadata = ZKMetadataProvider

Review comment:
       I do see some task executor like `RealtimeToOfflineSegmentsTaskExecutor` that works on realtime segments, although it extends `BaseMultipleSegmentsConversionExecutor`. I think it's a good practice to eliminate all these pre-assumptions to make the code clean, in case  once `BaseSingleSegmentConversionExecutor` starts to support realtime segment in the future. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
jackjlli commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r658136510



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseTaskExecutor.java
##########
@@ -56,4 +59,15 @@ protected Schema getSchema(String tableName) {
     Preconditions.checkState(schema != null, "Failed to find schema for table: %s", tableName);
     return schema;
   }
+
+  protected long getSegmentCrc(String tableNameWithType, String segmentName) {
+    TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+    SegmentZKMetadata segmentZKMetadata = tableType == TableType.OFFLINE ? ZKMetadataProvider
+        .getOfflineSegmentZKMetadata(MINION_CONTEXT.getHelixPropertyStore(), tableNameWithType, segmentName)
+        : ZKMetadataProvider
+            .getRealtimeSegmentZKMetadata(MINION_CONTEXT.getHelixPropertyStore(), tableNameWithType, segmentName);
+    Preconditions.checkState(segmentZKMetadata != null, "Failed to find segment metadata for table: %s, segment: %s",

Review comment:
       +1 on that. Another drawback of just throwing an exception here is that the task status would be left unchanged without proper cleanup.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] jackjlli merged pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
jackjlli merged pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mqliang commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r658289311



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseTaskExecutor.java
##########
@@ -56,4 +59,15 @@ protected Schema getSchema(String tableName) {
     Preconditions.checkState(schema != null, "Failed to find schema for table: %s", tableName);
     return schema;
   }
+
+  protected long getSegmentCrc(String tableNameWithType, String segmentName) {
+    TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableNameWithType);
+    SegmentZKMetadata segmentZKMetadata = tableType == TableType.OFFLINE ? ZKMetadataProvider
+        .getOfflineSegmentZKMetadata(MINION_CONTEXT.getHelixPropertyStore(), tableNameWithType, segmentName)
+        : ZKMetadataProvider
+            .getRealtimeSegmentZKMetadata(MINION_CONTEXT.getHelixPropertyStore(), tableNameWithType, segmentName);
+    Preconditions.checkState(segmentZKMetadata != null, "Failed to find segment metadata for table: %s, segment: %s",

Review comment:
       done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#issuecomment-867256423


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7086](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ce910ee) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/80fda686272d52f74057588d95faefbc016bca9f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (80fda68) will **decrease** coverage by `0.10%`.
   > The diff coverage is `57.64%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/7086/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7086      +/-   ##
   ============================================
   - Coverage     73.57%   73.46%   -0.11%     
     Complexity       91       91              
   ============================================
     Files          1486     1490       +4     
     Lines         73009    73164     +155     
     Branches      10492    10528      +36     
   ============================================
   + Hits          53713    53753      +40     
   - Misses        15818    15907      +89     
   - Partials       3478     3504      +26     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration | `41.73% <29.25%> (-0.17%)` | :arrow_down: |
   | unittests | `65.44% <47.16%> (-0.09%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ocal/indexsegment/immutable/EmptyIndexSegment.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0VtcHR5SW5kZXhTZWdtZW50LmphdmE=) | `36.84% <ø> (ø)` | |
   | [...ocal/indexsegment/mutable/IntermediateSegment.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvbXV0YWJsZS9JbnRlcm1lZGlhdGVTZWdtZW50LmphdmE=) | `70.06% <ø> (ø)` | |
   | [...local/realtime/impl/geospatial/MutableH3Index.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2dlb3NwYXRpYWwvTXV0YWJsZUgzSW5kZXguamF2YQ==) | `93.33% <ø> (ø)` | |
   | [...mpl/invertedindex/RealtimeInvertedIndexReader.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2ludmVydGVkaW5kZXgvUmVhbHRpbWVJbnZlcnRlZEluZGV4UmVhZGVyLmphdmE=) | `100.00% <ø> (ø)` | |
   | [...ealtime/impl/nullvalue/MutableNullValueVector.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL251bGx2YWx1ZS9NdXRhYmxlTnVsbFZhbHVlVmVjdG9yLmphdmE=) | `100.00% <ø> (ø)` | |
   | [...segment/index/readers/ValidDocIndexReaderImpl.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvVmFsaWREb2NJbmRleFJlYWRlckltcGwuamF2YQ==) | `0.00% <ø> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/segment/spi/IndexSegment.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL0luZGV4U2VnbWVudC5qYXZh) | `0.00% <ø> (ø)` | |
   | [...ment/spi/index/ThreadSafeMutableRoaringBitmap.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2luZGV4L1RocmVhZFNhZmVNdXRhYmxlUm9hcmluZ0JpdG1hcC5qYXZh) | `100.00% <ø> (ø)` | |
   | [...ion/tasks/BaseSingleSegmentConversionExecutor.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVNpbmdsZVNlZ21lbnRDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `84.61% <16.66%> (-8.87%)` | :arrow_down: |
   | [...not/segment/local/upsert/PartialUpsertHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91cHNlcnQvUGFydGlhbFVwc2VydEhhbmRsZXIuamF2YQ==) | `33.33% <33.33%> (ø)` | |
   | ... and [51 more](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f4e5518...ce910ee](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#issuecomment-867256423


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > :exclamation: No coverage uploaded for pull request base (`master@f4e5518`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#section-missing-base-commit).
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/7086/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff            @@
   ##             master    #7086   +/-   ##
   =========================================
     Coverage          ?   41.75%           
     Complexity        ?        7           
   =========================================
     Files             ?     1490           
     Lines             ?    73165           
     Branches          ?    10528           
   =========================================
     Hits              ?    30548           
     Misses            ?    40045           
     Partials          ?     2572           
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration | `41.75% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f4e5518...3860df4](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#issuecomment-867256423


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7086](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (87fd716) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/80fda686272d52f74057588d95faefbc016bca9f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (80fda68) will **decrease** coverage by `31.77%`.
   > The diff coverage is `29.25%`.
   
   > :exclamation: Current head 87fd716 differs from pull request most recent head b3f4120. Consider uploading reports for the commit b3f4120 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/7086/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7086       +/-   ##
   =============================================
   - Coverage     73.57%   41.79%   -31.78%     
   + Complexity       91        7       -84     
   =============================================
     Files          1486     1490        +4     
     Lines         73009    73163      +154     
     Branches      10492    10528       +36     
   =============================================
   - Hits          53713    30582    -23131     
   - Misses        15818    40004    +24186     
   + Partials       3478     2577      -901     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration | `41.79% <29.25%> (-0.11%)` | :arrow_down: |
   | unittests | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...core/query/pruner/SelectionQuerySegmentPruner.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9wcnVuZXIvU2VsZWN0aW9uUXVlcnlTZWdtZW50UHJ1bmVyLmphdmE=) | `82.95% <0.00%> (-3.41%)` | :arrow_down: |
   | [.../pinot/ingestion/jobs/SegmentPreprocessingJob.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vdjBfZGVwcmVjYXRlZC9waW5vdC1pbmdlc3Rpb24tY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9pbmdlc3Rpb24vam9icy9TZWdtZW50UHJlcHJvY2Vzc2luZ0pvYi5qYXZh) | `0.00% <ø> (ø)` | |
   | [...ocal/indexsegment/immutable/EmptyIndexSegment.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0VtcHR5SW5kZXhTZWdtZW50LmphdmE=) | `0.00% <ø> (-36.85%)` | :arrow_down: |
   | [...ocal/indexsegment/mutable/IntermediateSegment.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvbXV0YWJsZS9JbnRlcm1lZGlhdGVTZWdtZW50LmphdmE=) | `0.00% <ø> (-70.07%)` | :arrow_down: |
   | [...local/realtime/impl/geospatial/MutableH3Index.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2dlb3NwYXRpYWwvTXV0YWJsZUgzSW5kZXguamF2YQ==) | `0.00% <ø> (-93.34%)` | :arrow_down: |
   | [...mpl/invertedindex/RealtimeInvertedIndexReader.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2ludmVydGVkaW5kZXgvUmVhbHRpbWVJbnZlcnRlZEluZGV4UmVhZGVyLmphdmE=) | `90.47% <ø> (-9.53%)` | :arrow_down: |
   | [...ealtime/impl/nullvalue/MutableNullValueVector.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL251bGx2YWx1ZS9NdXRhYmxlTnVsbFZhbHVlVmVjdG9yLmphdmE=) | `85.71% <ø> (-14.29%)` | :arrow_down: |
   | [...segment/index/readers/ValidDocIndexReaderImpl.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvVmFsaWREb2NJbmRleFJlYWRlckltcGwuamF2YQ==) | `0.00% <ø> (-100.00%)` | :arrow_down: |
   | [...not/segment/local/upsert/PartialUpsertHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91cHNlcnQvUGFydGlhbFVwc2VydEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...t/segment/local/upsert/merger/IncrementMerger.java](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91cHNlcnQvbWVyZ2VyL0luY3JlbWVudE1lcmdlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | ... and [967 more](https://codecov.io/gh/apache/incubator-pinot/pull/7086/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f4e5518...b3f4120](https://codecov.io/gh/apache/incubator-pinot/pull/7086?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
jackjlli commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r657535651



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseTaskExecutor.java
##########
@@ -56,4 +57,12 @@ protected Schema getSchema(String tableName) {
     Preconditions.checkState(schema != null, "Failed to find schema for table: %s", tableName);
     return schema;
   }
+
+  protected long getSegmentCrc(String tableNameWithType, String segmentName) {
+    OfflineSegmentZKMetadata segmentZKMetadata = ZKMetadataProvider

Review comment:
       I do see some task executor like `RealtimeToOfflineSegmentsTaskExecutor` that works on realtime segments, although it extends `BaseMultipleSegmentsConversionExecutor`. I think it's a good practice to eliminate all these pre-assumptions to make the code clean. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mqliang commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r659970454



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +71,14 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      throw new FatalException(taskType + " on table: " + tableNameWithType + ", segment: " + segmentName

Review comment:
       @mcvsubbu 
   I was thinking this will make our change be consistent with the old logic: old logic will fail when upload, here we fail early. I am fine with treating this as an no-error case by return an empty`SegmentConversionResult`.

##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +71,14 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      throw new FatalException(taskType + " on table: " + tableNameWithType + ", segment: " + segmentName

Review comment:
       updated




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mqliang commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r657533922



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +70,13 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      new SegmentConversionResult.Builder().setTableNameWithType(tableNameWithType).setSegmentName(segmentName).build();

Review comment:
       updated!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] jackjlli merged pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
jackjlli merged pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
jackjlli commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r658305639



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +70,14 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      return new SegmentConversionResult.Builder().setTableNameWithType(tableNameWithType).setSegmentName(segmentName)
+          .build();

Review comment:
       `Cancelled` usually means that a task gets canceled by someone.  `Failed` seems okay as it doesn't meet the task's expectation. What else status does it support? Is there sth like `Aborted` available?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
jackjlli commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r657535651



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseTaskExecutor.java
##########
@@ -56,4 +57,12 @@ protected Schema getSchema(String tableName) {
     Preconditions.checkState(schema != null, "Failed to find schema for table: %s", tableName);
     return schema;
   }
+
+  protected long getSegmentCrc(String tableNameWithType, String segmentName) {
+    OfflineSegmentZKMetadata segmentZKMetadata = ZKMetadataProvider

Review comment:
       I do see some task executor like `RealtimeToOfflineSegmentsTaskExecutor` that works on realtime segments, although it extends `BaseMultipleSegmentsConversionExecutor`. I think it's a good practice to eliminate all these assumptions to make the code clean. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mqliang commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r659970454



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +71,14 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      throw new FatalException(taskType + " on table: " + tableNameWithType + ", segment: " + segmentName

Review comment:
       @mcvsubbu 
   I was thinking this will make our change be consistent with the old logic: old logic will fail when upload, here we fail early. I am fine with treating this as an no-error case by return an empty`SegmentConversionResult`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] mqliang commented on a change in pull request #7086: Optimize BaseSingleSegmentConversionExecutor by skipping task if CRC not match

Posted by GitBox <gi...@apache.org>.
mqliang commented on a change in pull request #7086:
URL: https://github.com/apache/incubator-pinot/pull/7086#discussion_r658297466



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
##########
@@ -70,6 +70,14 @@ public SegmentConversionResult executeTask(PinotTaskConfig pinotTaskConfig)
     String originalSegmentCrc = configs.get(MinionConstants.ORIGINAL_SEGMENT_CRC_KEY);
     String authToken = configs.get(MinionConstants.AUTH_TOKEN);
 
+    long currentSegmentCrc = getSegmentCrc(tableNameWithType, segmentName);
+    if (Long.parseLong(originalSegmentCrc) != currentSegmentCrc) {
+      LOGGER.info("Segment CRC does not match, skip the task. Original CRC: {}, current CRC: {}", originalSegmentCrc,
+          currentSegmentCrc);
+      return new SegmentConversionResult.Builder().setTableNameWithType(tableNameWithType).setSegmentName(segmentName)
+          .build();

Review comment:
       @mcvsubbu @jackjlli 
   By return a `new SegmentConversionResult` here, task will been marked as completed. We can also throw `TaskCancelledException` here so that task will be marked as "cancelled", or `FatalException` so that task will be marked as"failed". Any thoughts?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org