You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/09/24 23:30:52 UTC
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7481: MergeRollupTaskGenerator enhancement: enable parallel buckets scheduling
Jackie-Jiang commented on a change in pull request #7481:
URL: https://github.com/apache/pinot/pull/7481#discussion_r715951030
##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java
##########
@@ -193,18 +196,20 @@ public String getTaskType() {
continue;
}
- // Get the bucket size and buffer
+ // Get the bucket size, buffer and number of parallel buckets (1 as default)
long bucketMs =
- TimeUtils.convertPeriodToMillis(mergeConfigs.get(MinionConstants.MergeTask.BUCKET_TIME_PERIOD_KEY));
+ TimeUtils.convertPeriodToMillis(mergeConfigs.get(MergeTask.BUCKET_TIME_PERIOD_KEY));
long bufferMs =
- TimeUtils.convertPeriodToMillis(mergeConfigs.get(MinionConstants.MergeTask.BUFFER_TIME_PERIOD_KEY));
+ TimeUtils.convertPeriodToMillis(mergeConfigs.get(MergeTask.BUFFER_TIME_PERIOD_KEY));
+ int numParallelBuckets = mergeConfigs.get(MergeRollupTask.NUM_PARALLEL_BUCKETS) != null
+ ? Integer.parseInt(mergeConfigs.get(MergeRollupTask.NUM_PARALLEL_BUCKETS)) : DEFAULT_NUM_PARALLEL_BUCKETS;
// Get watermark from MergeRollupTaskMetadata ZNode
- // windowStartMs = watermarkMs, windowEndMs = windowStartMs + bucketTimeMs
+ // windowStartMs = watermarkMs, windowEndMs = windowStartMs + bucketTimeMs * numParallelBuckets
long waterMarkMs =
getWatermarkMs(preSelectedSegments.get(0).getStartTimeMs(), bucketMs, mergeLevel, mergeRollupTaskMetadata);
long windowStartMs = waterMarkMs;
- long windowEndMs = windowStartMs + bucketMs;
+ long windowEndMs = windowStartMs + bucketMs * numParallelBuckets;
Review comment:
+1. I feel the algorithm should be loop over bucket one by one, and keep tracking if there are spill over segments.
Alternatively, we might also be able to put multiple buckets into the same task as long as it follows the `maxNumRecordsPerTask`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org