You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/11/15 14:35:50 UTC
[GitHub] [druid] kfaraz opened a new pull request, #13369: Add SegmentAllocationQueue to batch allocation actions
kfaraz opened a new pull request, #13369:
URL: https://github.com/apache/druid/pull/13369
### Description
In a cluster with a large number of streaming tasks (~1000), `SegmentAllocateAction`s on the overlord can often take very long intervals of time to finish thus causing spikes in the `task/action/run/time`. This may result in lag building up while a task waits for a segment to get allocated.
The root causes are:
- large number of metadata calls made to the segments and pending segments tables
- `giant` lock held in `TaskLockbox.tryLock()` to acquire task locks and allocate segments
Since the contention typically arises when several tasks of the same datasource try to allocate segments for the same interval/granularity, the allocation run times can be improved by batching the requests together.
### Changes
#### Broad strokes
- Queue handling: `SegmentAllocationQueue`
- Rest of the behaviour remains exactly the same (order of steps, retries, etc.)
- Redundant metadata calls have been eliminated
- Query to fetch matching pending segment has been made into a single query
- `giant` lock is acquired just once per batch per tryInterval in `TaskLockbox`
- Old code has been retained for reference and in case we want to put this behind a feature flag
#### Adding a request
- Add method `SegmentAllocateAction.performAsync()` which returns a `Future<SegmentId>`
- Submit each allocate action to a `SegmentAllocationQueue`
- Add the action to a batch and execute it when it is due
#### Processing a request batch
- Wait a certain amount of time (currently 5s) before processing a batch
- Send all requests in the batch to `TaskLockbox` to acquire task locks and allocate segments
- Respond to leadership changes and stop processing queue when not leader
- Emit batch and request level metrics
#### Allocating segments
- Add `TaskLockbox.allocateSegments()` retaining the logic from `TaskLockbox.tryLock()`
- Add `IndexerMetadataStorageCoordinator.allocatePendingSegments()` retaining the logic from `allocatePendingSegment()`
#### New metrics
Per batch (dims: `dataSource`, ):
- `task/action/batch/runTime`
- `task/action/batch/queueTime`
- `task/action/batch/size`
Per request (dims: `taskId`, `taskType`, `dataSource`):
- `task/action/attempt/count`
- `task/action/success/count`
### Pending changes
- Tests for new flow
- Docs for metrics
- (Optional) feature flag for new flow
#### Release note
Speed up segment allocation and reduce metadata calls by clubbing multiple requests together.
<hr>
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027785848
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1034384415
##########
core/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -68,6 +72,50 @@ public static boolean isEternity(final Interval interval)
return ETERNITY.equals(interval);
}
+ /**
+ * Finds an interval from the given set of sortedIntervals which overlaps with
+ * the searchInterval. If multiple candidate intervals overlap with the
+ * searchInterval, the "smallest" interval based on the
+ * {@link Comparators#intervalsByStartThenEnd()} is returned.
+ *
+ * @param searchInterval Interval which should overlap with the result
+ * @param sortedIntervals Candidate overlapping intervals, sorted in ascending
+ * order, using {@link Comparators#intervalsByStartThenEnd()}.
+ * @return The first overlapping interval, if one exists, otherwise null.
+ */
+ @Nullable
+ public static Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ Arrays.sort(sortedIntervals, Comparators.intervalsByStartThenEnd());
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, index returned from binarySearch is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
Review Comment:
This seems like an incorrect assumption with segments of different granularities.
Consider the sorted intervals [ (2022-01-01 / 2023-01-01), (2022-05-01 / 2022-06-01) ]
If we were to take target interval [2022-07-01 / 2022-08-01], index = 1 would not overlap but index = 0 would.
If the intervals to this method are all of the same granularity, could you please add it to the javadoc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1037766628
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 2000;
+
+ private final long maxWaitTimeMillis;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final AtomicBoolean isLeader = new AtomicBoolean(false);
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ ServiceEmitter emitter,
+ ScheduledExecutorFactory executorFactory
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+
+ this.executor = taskLockConfig.isBatchSegmentAllocation()
+ ? executorFactory.create(1, "SegmentAllocQueue-%s") : null;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ if (isEnabled()) {
+ log.info("Initializing segment allocation queue.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ }
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ if (isEnabled()) {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+ }
+
+ public void becomeLeader()
+ {
+ if (!isLeader.compareAndSet(false, true)) {
+ log.info("Already the leader. Queue processing has started.");
+ } else if (isEnabled()) {
+ log.info("Elected leader. Starting queue processing.");
+ } else {
+ log.info(
+ "Elected leader but batched segment allocation is disabled. "
+ + "Segment allocation queue will not be used."
+ );
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ if (!isLeader.compareAndSet(true, false)) {
+ log.info("Already surrendered leadership. Queue processing is stopped.");
+ } else if (isEnabled()) {
+ log.info("Not leader anymore. Stopping queue processing.");
+ } else {
+ log.info("Not leader anymore. Segment allocation queue is already disabled.");
+ }
+ }
+
+ public boolean isEnabled()
+ {
+ return executor != null && !executor.isShutdown();
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Gets the number of batches currently in the queue.
+ */
+ public int size()
+ {
+ return processingQueue.size();
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!isLeader.get()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. Fails all the pending
+ * requests in the batch if we are not leader or if the queue is full.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (!isLeader.get()) {
+ batch.failPendingRequests("Cannot allocate segment if not leader");
+ return false;
+ } else if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ batch.failPendingRequests(
+ "Segment allocation queue is full. Check the metric `task/action/batch/runTime` "
+ + "to determine if metadata operations are slow."
+ );
+ return false;
+ }
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If a batch already
+ * exists for this key, transfers all the requests from this batch to the
+ * existing one.
+ */
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ return addBatchToQueue(batch) ? batch : null;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.transferRequestsFrom(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ clearQueueIfNotLeader();
+
+ // Process all batches which are due
+ log.debug("Processing batches which are due. Queue size [%d].", processingQueue.size());
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.pollFirst();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ }
+ catch (Throwable t) {
+ nextBatch.failPendingRequests(t);
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ // Requeue if not fully processed yet
+ if (processed) {
+ ++numProcessedBatches;
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.info("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Removes items from the queue as long as we are not leader.
+ */
+ private void clearQueueIfNotLeader()
+ {
+ int failedBatches = 0;
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && !isLeader.get()) {
+ processingQueue.pollFirst();
+ keyToBatch.remove(nextBatch.key);
+ nextBatch.failPendingRequests("Cannot allocate segment if not leader");
+ ++failedBatches;
+
+ nextBatch = processingQueue.peekFirst();
+ }
+ if (failedBatches > 0) {
+ log.info("Not leader. Failed [%d] batches, remaining in queue [%d].", failedBatches, processingQueue.size());
+ }
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ } else if (!isLeader.get()) {
+ requestBatch.failPendingRequests("Cannot allocate segment if not leader");
+ return true;
+ }
+
+ log.debug(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/attempts", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.debug("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.debug("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ requestBatch.failPendingRequests("Allocation failed probably due to conflicting segments.");
+ return true;
+ } else {
+ log.debug("Used segments have changed. Requeuing failed requests.");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ // Find requests whose row interval overlaps with an existing used segment
+ final Set<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> requestsWithNoOverlappingSegment = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ requestsWithNoOverlappingSegment.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> overlapIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment, the interval of the used segment
Review Comment:
The used segments for the allocate interval (`segmentGranularity.bucket(row.timestamp())`) are fetched so that we don't have to make the metadata call multiple times.
The logic hasn't changed because we use the `rowInterval` (`queryGranularity.bucket(row.timestamp())`) to find the actual overlapping interval. Please look at the line after this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1331687284
This pull request **introduces 2 alerts** when merging a03ba2e1b3159a392905b07e89090a06ff6e39cb into 50963edcae70150f13520b619f167512d951a71b - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-323479dd368bdc05ef314416bbe90ef6fe68bb0e)
**new alerts:**
* 2 for User\-controlled data in numeric cast
---
**Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027781599
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1317699131
This pull request **introduces 2 alerts** when merging 0524e8378a476f8328a49f38fb45cfe4a394fa35 into 78d0b0abce2b27b88a53790684d81530a2318fc9 - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-14c92590baacff5ccfead015b7313ff3585bbaad)
**new alerts:**
* 2 for User\-controlled data in numeric cast
---
**Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1034851486
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -446,6 +449,140 @@ public LockResult tryLock(final Task task, final LockRequest request)
}
}
+ /**
+ * Attempts to allocate segments for the given requests. Each request contains
+ * a {@link Task} and a {@link SegmentAllocateAction}. This method tries to
+ * acquire the task locks on the required intervals/segments and then performs
+ * a batch allocation of segments. It is possible that some requests succeed
+ * successfully and others failed. In that case, only the failed ones should be
+ * retried.
+ *
+ * @param requests List of allocation requests
+ * @param dataSource Datasource for which segment is to be allocated.
+ * @param interval Interval for which segment is to be allocated.
+ * @param skipSegmentLineageCheck Whether lineage check is to be skipped
+ * (this is true for streaming ingestion)
+ * @param lockGranularity Granularity of task lock
+ * @return List of allocation results in the same order as the requests.
+ */
+ public List<SegmentAllocateResult> allocateSegments(
+ List<SegmentAllocateRequest> requests,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ LockGranularity lockGranularity
+ )
+ {
+ log.info("Allocating [%d] segments for datasource [%s], interval [%s]", requests.size(), dataSource, interval);
+ final boolean isTimeChunkLock = lockGranularity == LockGranularity.TIME_CHUNK;
+
+ final AllocationHolderList holderList = new AllocationHolderList(requests, interval);
+ holderList.getPending().forEach(this::verifyTaskIsActive);
+
+ giant.lock();
+ try {
+ if (isTimeChunkLock) {
+ holderList.getPending().forEach(holder -> acquireTaskLock(holder, true));
+ allocateSegmentIds(dataSource, interval, skipSegmentLineageCheck, holderList.getPending());
+ } else {
+ allocateSegmentIds(dataSource, interval, skipSegmentLineageCheck, holderList.getPending());
+ holderList.getPending().forEach(holder -> acquireTaskLock(holder, false));
+ }
+
+ // TODO: for failed allocations, cleanup newly created locks from the posse map
Review Comment:
Yeah, let me check this. If not done, this might leave some garbage locks in the posse map.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1034367935
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/LocalTaskActionClient.java:
##########
@@ -76,11 +76,34 @@ public <RetType> RetType submit(TaskAction<RetType> taskAction)
}
final long performStartTime = System.currentTimeMillis();
- final RetType result = taskAction.perform(task, toolbox);
+ final RetType result;
+ if (isBatchAllocateAction(taskAction)) {
+ result = performBatchAllocateAction(taskAction);
+ } else {
+ result = taskAction.perform(task, toolbox);
+ }
emitTimerMetric("task/action/run/time", taskAction, System.currentTimeMillis() - performStartTime);
return result;
}
+ private <R> boolean isBatchAllocateAction(TaskAction<R> taskAction)
+ {
+ return toolbox.canBatchSegmentAllocation()
+ && taskAction instanceof SegmentAllocateAction;
Review Comment:
Add `canPerformAsync` and `performAsync` to `TaskAction`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1035547408
##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -777,6 +994,202 @@ private void insertPendingSegmentIntoMetastore(
.execute();
}
+ private Map<SegmentCreateRequest, SegmentIdWithShardSpec> createNewSegments(
+ Handle handle,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ List<SegmentCreateRequest> requests
+ ) throws IOException
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ // Get the time chunk and associated data segments for the given interval, if any
+ final List<TimelineObjectHolder<String, DataSegment>> existingChunks =
+ getTimelineForIntervalsWithHandle(handle, dataSource, Collections.singletonList(interval))
+ .lookup(interval);
+
+ if (existingChunks.size() > 1) {
+ // Not possible to expand more than one chunk with a single segment.
+ log.warn(
+ "Cannot allocate new segments for dataSource[%s], interval[%s]: already have [%,d] chunks.",
+ dataSource,
+ interval,
+ existingChunks.size()
+ );
+ return Collections.emptyMap();
+ }
+
+ // Shard spec of any of the requests (as they are all compatible) can be used to
+ // identify existing shard specs that share partition space with the requested ones.
+ final PartialShardSpec partialShardSpec = requests.get(0).getPartialShardSpec();
+
+ // max partitionId of published data segments which share the same partition space.
+ SegmentIdWithShardSpec committedMaxId = null;
+
+ @Nullable
+ final String versionOfExistingChunk;
+ if (existingChunks.isEmpty()) {
+ versionOfExistingChunk = null;
+ } else {
+ TimelineObjectHolder<String, DataSegment> existingHolder = Iterables.getOnlyElement(existingChunks);
+ versionOfExistingChunk = existingHolder.getVersion();
+
+ // Don't use the stream API for performance.
+ for (DataSegment segment : FluentIterable
+ .from(existingHolder.getObject())
+ .transform(PartitionChunk::getObject)
+ // Here we check only the segments of the shardSpec which shares the same partition space with the given
+ // partialShardSpec. Note that OverwriteShardSpec doesn't share the partition space with others.
+ // See PartitionIds.
+ .filter(segment -> segment.getShardSpec().sharePartitionSpace(partialShardSpec))) {
+ if (committedMaxId == null
+ || committedMaxId.getShardSpec().getPartitionNum() < segment.getShardSpec().getPartitionNum()) {
+ committedMaxId = SegmentIdWithShardSpec.fromDataSegment(segment);
+ }
+ }
+ }
+
+
+ // Fetch the pending segments for this interval to determine max partitionId
+ // across all shard specs (published + pending).
+ // A pending segment having a higher partitionId must also be considered
+ // to avoid clashes when inserting the pending segment created here.
+ final Set<SegmentIdWithShardSpec> pendingSegments =
+ getPendingSegmentsForIntervalWithHandle(handle, dataSource, interval);
+ if (committedMaxId != null) {
+ pendingSegments.add(committedMaxId);
+ }
+
+ final Map<SegmentCreateRequest, SegmentIdWithShardSpec> createdSegments = new HashMap<>();
+ final Map<String, SegmentIdWithShardSpec> sequenceHashToSegment = new HashMap<>();
+
+ for (SegmentCreateRequest request : requests) {
+ // Check if the required segment has already been created in this batch
+ final String sequenceHash = getSequenceNameAndPrevIdSha(request, interval, skipSegmentLineageCheck);
+
+ final SegmentIdWithShardSpec createdSegment;
+ if (sequenceHashToSegment.containsKey(sequenceHash)) {
+ createdSegment = sequenceHashToSegment.get(sequenceHash);
+ } else {
+ createdSegment = createNewSegment(
+ request,
+ dataSource,
+ interval,
+ versionOfExistingChunk,
+ committedMaxId,
+ pendingSegments
+ );
+
+ // Add to pendingSegments to consider for partitionId
+ if (createdSegment != null) {
+ pendingSegments.add(createdSegment);
+ sequenceHashToSegment.put(sequenceHash, createdSegment);
+ log.info("Created new segment [%s]", createdSegment);
+ }
+ }
+
+ if (createdSegment != null) {
+ createdSegments.put(request, createdSegment);
+ }
+ }
+
+ log.info("Created [%d] new segments for [%d] allocate requests.", sequenceHashToSegment.size(), requests.size());
+ return createdSegments;
+ }
+
+ private SegmentIdWithShardSpec createNewSegment(
+ SegmentCreateRequest request,
+ String dataSource,
+ Interval interval,
+ String versionOfExistingChunk,
+ SegmentIdWithShardSpec committedMaxId,
+ Set<SegmentIdWithShardSpec> pendingSegments
+ )
+ {
+ final PartialShardSpec partialShardSpec = request.getPartialShardSpec();
+ final String existingVersion = request.getVersion();
+
+ // If there is an existing chunk, find the max id with the same version as the existing chunk.
+ // There may still be a pending segment with a higher version (but no corresponding used segments)
+ // which may generate a clash with an existing segment once the new id is generated
+ final SegmentIdWithShardSpec overallMaxId =
Review Comment:
The overallMaxId is computed assuming that committedMaxId has already added to the set of pendingSegments.
I think it's better to not make that assumption and add committedMaxId explicitly in this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1034832069
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -446,6 +449,140 @@ public LockResult tryLock(final Task task, final LockRequest request)
}
}
+ /**
+ * Attempts to allocate segments for the given requests. Each request contains
+ * a {@link Task} and a {@link SegmentAllocateAction}. This method tries to
+ * acquire the task locks on the required intervals/segments and then performs
+ * a batch allocation of segments. It is possible that some requests succeed
+ * successfully and others failed. In that case, only the failed ones should be
+ * retried.
+ *
+ * @param requests List of allocation requests
+ * @param dataSource Datasource for which segment is to be allocated.
+ * @param interval Interval for which segment is to be allocated.
+ * @param skipSegmentLineageCheck Whether lineage check is to be skipped
+ * (this is true for streaming ingestion)
+ * @param lockGranularity Granularity of task lock
+ * @return List of allocation results in the same order as the requests.
+ */
+ public List<SegmentAllocateResult> allocateSegments(
+ List<SegmentAllocateRequest> requests,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ LockGranularity lockGranularity
+ )
+ {
+ log.info("Allocating [%d] segments for datasource [%s], interval [%s]", requests.size(), dataSource, interval);
+ final boolean isTimeChunkLock = lockGranularity == LockGranularity.TIME_CHUNK;
+
+ final AllocationHolderList holderList = new AllocationHolderList(requests, interval);
+ holderList.getPending().forEach(this::verifyTaskIsActive);
+
+ giant.lock();
+ try {
+ if (isTimeChunkLock) {
+ holderList.getPending().forEach(holder -> acquireTaskLock(holder, true));
+ allocateSegmentIds(dataSource, interval, skipSegmentLineageCheck, holderList.getPending());
+ } else {
+ allocateSegmentIds(dataSource, interval, skipSegmentLineageCheck, holderList.getPending());
+ holderList.getPending().forEach(holder -> acquireTaskLock(holder, false));
+ }
+
+ // TODO: for failed allocations, cleanup newly created locks from the posse map
Review Comment:
Does this need to be addressed as part of these changes?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,694 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 2000;
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final AtomicBoolean isLeader = new AtomicBoolean(false);
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ ServiceEmitter emitter,
+ ScheduledExecutorFactory executorFactory
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = executorFactory.create(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Initializing segment allocation queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Gets the number of batches currently in the queue.
+ */
+ public int size()
+ {
+ return processingQueue.size();
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!isLeader.get()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If the queue is full,
+ * marks the batch as completed failing all the pending requests.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ log.warn("Cannot add batch [%s] as queue is full. Failing [%d] requests.", batch.key, batch.size());
+ batch.markCompleted();
+ return false;
+ }
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If a batch already
+ * exists for this key, transfers all the requests from this batch to the
+ * existing one.
+ */
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ return addBatchToQueue(batch) ? batch : null;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.transferRequestsFrom(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!isLeader.get()) {
+ log.info("Not leader anymore. Failing [%d] batches in queue.", processingQueue.size());
+
+ // Keep removing items from the queue as long as not leader
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && !isLeader.get()) {
+ processingQueue.pollFirst();
+ keyToBatch.remove(nextBatch.key);
+ nextBatch.markCompleted();
+ nextBatch = processingQueue.peekFirst();
+ }
+ }
+
+ // Check once again for leadership
+ if (!isLeader.get()) {
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution. Overall queue size [%d].", processingQueue.size());
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.pollFirst();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ ++numProcessedBatches;
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.info("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.debug(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.debug("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
Review Comment:
Could you please explain a bit more about this?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,694 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 2000;
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final AtomicBoolean isLeader = new AtomicBoolean(false);
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ ServiceEmitter emitter,
+ ScheduledExecutorFactory executorFactory
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = executorFactory.create(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Initializing segment allocation queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Gets the number of batches currently in the queue.
+ */
+ public int size()
+ {
+ return processingQueue.size();
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!isLeader.get()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If the queue is full,
+ * marks the batch as completed failing all the pending requests.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ log.warn("Cannot add batch [%s] as queue is full. Failing [%d] requests.", batch.key, batch.size());
+ batch.markCompleted();
+ return false;
+ }
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If a batch already
+ * exists for this key, transfers all the requests from this batch to the
+ * existing one.
+ */
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ return addBatchToQueue(batch) ? batch : null;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.transferRequestsFrom(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!isLeader.get()) {
+ log.info("Not leader anymore. Failing [%d] batches in queue.", processingQueue.size());
+
+ // Keep removing items from the queue as long as not leader
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && !isLeader.get()) {
+ processingQueue.pollFirst();
+ keyToBatch.remove(nextBatch.key);
+ nextBatch.markCompleted();
+ nextBatch = processingQueue.peekFirst();
+ }
+ }
+
+ // Check once again for leadership
+ if (!isLeader.get()) {
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution. Overall queue size [%d].", processingQueue.size());
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.pollFirst();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ ++numProcessedBatches;
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.info("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.debug(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
Review Comment:
nit: `task/action/batch/attempts` seems more accurate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1037766628
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 2000;
+
+ private final long maxWaitTimeMillis;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final AtomicBoolean isLeader = new AtomicBoolean(false);
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ ServiceEmitter emitter,
+ ScheduledExecutorFactory executorFactory
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+
+ this.executor = taskLockConfig.isBatchSegmentAllocation()
+ ? executorFactory.create(1, "SegmentAllocQueue-%s") : null;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ if (isEnabled()) {
+ log.info("Initializing segment allocation queue.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ }
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ if (isEnabled()) {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+ }
+
+ public void becomeLeader()
+ {
+ if (!isLeader.compareAndSet(false, true)) {
+ log.info("Already the leader. Queue processing has started.");
+ } else if (isEnabled()) {
+ log.info("Elected leader. Starting queue processing.");
+ } else {
+ log.info(
+ "Elected leader but batched segment allocation is disabled. "
+ + "Segment allocation queue will not be used."
+ );
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ if (!isLeader.compareAndSet(true, false)) {
+ log.info("Already surrendered leadership. Queue processing is stopped.");
+ } else if (isEnabled()) {
+ log.info("Not leader anymore. Stopping queue processing.");
+ } else {
+ log.info("Not leader anymore. Segment allocation queue is already disabled.");
+ }
+ }
+
+ public boolean isEnabled()
+ {
+ return executor != null && !executor.isShutdown();
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Gets the number of batches currently in the queue.
+ */
+ public int size()
+ {
+ return processingQueue.size();
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!isLeader.get()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. Fails all the pending
+ * requests in the batch if we are not leader or if the queue is full.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (!isLeader.get()) {
+ batch.failPendingRequests("Cannot allocate segment if not leader");
+ return false;
+ } else if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ batch.failPendingRequests(
+ "Segment allocation queue is full. Check the metric `task/action/batch/runTime` "
+ + "to determine if metadata operations are slow."
+ );
+ return false;
+ }
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If a batch already
+ * exists for this key, transfers all the requests from this batch to the
+ * existing one.
+ */
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ return addBatchToQueue(batch) ? batch : null;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.transferRequestsFrom(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ clearQueueIfNotLeader();
+
+ // Process all batches which are due
+ log.debug("Processing batches which are due. Queue size [%d].", processingQueue.size());
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.pollFirst();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ }
+ catch (Throwable t) {
+ nextBatch.failPendingRequests(t);
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ // Requeue if not fully processed yet
+ if (processed) {
+ ++numProcessedBatches;
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.info("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Removes items from the queue as long as we are not leader.
+ */
+ private void clearQueueIfNotLeader()
+ {
+ int failedBatches = 0;
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && !isLeader.get()) {
+ processingQueue.pollFirst();
+ keyToBatch.remove(nextBatch.key);
+ nextBatch.failPendingRequests("Cannot allocate segment if not leader");
+ ++failedBatches;
+
+ nextBatch = processingQueue.peekFirst();
+ }
+ if (failedBatches > 0) {
+ log.info("Not leader. Failed [%d] batches, remaining in queue [%d].", failedBatches, processingQueue.size());
+ }
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ } else if (!isLeader.get()) {
+ requestBatch.failPendingRequests("Cannot allocate segment if not leader");
+ return true;
+ }
+
+ log.debug(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/attempts", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.debug("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.debug("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ requestBatch.failPendingRequests("Allocation failed probably due to conflicting segments.");
+ return true;
+ } else {
+ log.debug("Used segments have changed. Requeuing failed requests.");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ // Find requests whose row interval overlaps with an existing used segment
+ final Set<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> requestsWithNoOverlappingSegment = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ requestsWithNoOverlappingSegment.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> overlapIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment, the interval of the used segment
Review Comment:
The used segments are fetched for the allocate interval (`segmentGranularity.bucket(row.timestamp())`) so that we don't have to make the metadata call multiple times.
The logic hasn't changed because we use the `rowInterval` (`queryGranularity.bucket(row.timestamp())`) to find the actual overlapping interval. Please look at the line after this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027651278
##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -777,6 +994,202 @@ private void insertPendingSegmentIntoMetastore(
.execute();
}
+ private Map<SegmentCreateRequest, SegmentIdWithShardSpec> createNewSegments(
+ Handle handle,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ List<SegmentCreateRequest> requests
+ ) throws IOException
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ // Get the time chunk and associated data segments for the given interval, if any
+ final List<TimelineObjectHolder<String, DataSegment>> existingChunks =
+ getTimelineForIntervalsWithHandle(handle, dataSource, Collections.singletonList(interval))
+ .lookup(interval);
+
+ if (existingChunks.size() > 1) {
+ // Not possible to expand more than one chunk with a single segment.
+ log.warn(
+ "Cannot allocate new segments for dataSource[%s], interval[%s]: already have [%,d] chunks.",
+ dataSource,
+ interval,
+ existingChunks.size()
+ );
+ return Collections.emptyMap();
+ }
+
+ // Shard spec of any of the requests (as they are all compatible) can be used to
+ // identify existing shard specs that share partition space with the requested ones.
+ final PartialShardSpec partialShardSpec = requests.get(0).getPartialShardSpec();
+
+ // max partitionId of published data segments which share the same partition space.
+ SegmentIdWithShardSpec committedMaxId = null;
+
+ @Nullable
+ final String versionOfExistingChunk;
+ if (existingChunks.isEmpty()) {
+ versionOfExistingChunk = null;
+ } else {
+ TimelineObjectHolder<String, DataSegment> existingHolder = Iterables.getOnlyElement(existingChunks);
+ versionOfExistingChunk = existingHolder.getVersion();
+
+ // Don't use the stream API for performance.
+ for (DataSegment segment : FluentIterable
+ .from(existingHolder.getObject())
+ .transform(PartitionChunk::getObject)
+ // Here we check only the segments of the shardSpec which shares the same partition space with the given
+ // partialShardSpec. Note that OverwriteShardSpec doesn't share the partition space with others.
+ // See PartitionIds.
+ .filter(segment -> segment.getShardSpec().sharePartitionSpace(partialShardSpec))) {
+ if (committedMaxId == null
+ || committedMaxId.getShardSpec().getPartitionNum() < segment.getShardSpec().getPartitionNum()) {
+ committedMaxId = SegmentIdWithShardSpec.fromDataSegment(segment);
+ }
+ }
+ }
+
+
+ // Fetch the pending segments for this interval to determine max partitionId
+ // across all shard specs (published + pending).
+ // A pending segment having a higher partitionId must also be considered
+ // to avoid clashes when inserting the pending segment created here.
+ final Set<SegmentIdWithShardSpec> pendingSegments =
+ getPendingSegmentsForIntervalWithHandle(handle, dataSource, interval);
+ if (committedMaxId != null) {
+ pendingSegments.add(committedMaxId);
+ }
+
+ final Map<SegmentCreateRequest, SegmentIdWithShardSpec> createdSegments = new HashMap<>();
+ final Map<String, SegmentIdWithShardSpec> sequenceHashToSegment = new HashMap<>();
+
+ for (SegmentCreateRequest request : requests) {
+ // Check if the required segment has already been created in this batch
+ final String sequenceHash = getSequenceNameAndPrevIdSha(request, interval, skipSegmentLineageCheck);
+
+ final SegmentIdWithShardSpec createdSegment;
+ if (sequenceHashToSegment.containsKey(sequenceHash)) {
+ createdSegment = sequenceHashToSegment.get(sequenceHash);
+ } else {
+ createdSegment = createNewSegment(
+ request,
+ dataSource,
+ interval,
+ versionOfExistingChunk,
+ committedMaxId,
+ pendingSegments
+ );
+
+ // Add to pendingSegments to consider for partitionId
+ if (createdSegment != null) {
+ pendingSegments.add(createdSegment);
+ sequenceHashToSegment.put(sequenceHash, createdSegment);
+ log.info("Created new segment [%s]", createdSegment);
+ }
+ }
+
+ if (createdSegment != null) {
+ createdSegments.put(request, createdSegment);
+ }
+ }
+
+ log.info("Created [%d] new segments for [%d] allocate requests.", sequenceHashToSegment.size(), requests.size());
+ return createdSegments;
+ }
+
+ private SegmentIdWithShardSpec createNewSegment(
Review Comment:
Would it be possible to utilize the older method `createNewSegment` and pass the number of segments to be created as an argument?
If all shardSpecs for common properties can be created with increasing partitionIds in the same call, it may make things simpler and help with performance as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027784364
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027629003
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
Review Comment:
Do you think a no-op impl would be useful for tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1028125120
##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -777,6 +994,202 @@ private void insertPendingSegmentIntoMetastore(
.execute();
}
+ private Map<SegmentCreateRequest, SegmentIdWithShardSpec> createNewSegments(
+ Handle handle,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ List<SegmentCreateRequest> requests
+ ) throws IOException
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ // Get the time chunk and associated data segments for the given interval, if any
+ final List<TimelineObjectHolder<String, DataSegment>> existingChunks =
+ getTimelineForIntervalsWithHandle(handle, dataSource, Collections.singletonList(interval))
+ .lookup(interval);
+
+ if (existingChunks.size() > 1) {
+ // Not possible to expand more than one chunk with a single segment.
+ log.warn(
+ "Cannot allocate new segments for dataSource[%s], interval[%s]: already have [%,d] chunks.",
+ dataSource,
+ interval,
+ existingChunks.size()
+ );
+ return Collections.emptyMap();
+ }
+
+ // Shard spec of any of the requests (as they are all compatible) can be used to
+ // identify existing shard specs that share partition space with the requested ones.
+ final PartialShardSpec partialShardSpec = requests.get(0).getPartialShardSpec();
+
+ // max partitionId of published data segments which share the same partition space.
+ SegmentIdWithShardSpec committedMaxId = null;
+
+ @Nullable
+ final String versionOfExistingChunk;
+ if (existingChunks.isEmpty()) {
+ versionOfExistingChunk = null;
+ } else {
+ TimelineObjectHolder<String, DataSegment> existingHolder = Iterables.getOnlyElement(existingChunks);
+ versionOfExistingChunk = existingHolder.getVersion();
+
+ // Don't use the stream API for performance.
+ for (DataSegment segment : FluentIterable
+ .from(existingHolder.getObject())
+ .transform(PartitionChunk::getObject)
+ // Here we check only the segments of the shardSpec which shares the same partition space with the given
+ // partialShardSpec. Note that OverwriteShardSpec doesn't share the partition space with others.
+ // See PartitionIds.
+ .filter(segment -> segment.getShardSpec().sharePartitionSpace(partialShardSpec))) {
+ if (committedMaxId == null
+ || committedMaxId.getShardSpec().getPartitionNum() < segment.getShardSpec().getPartitionNum()) {
+ committedMaxId = SegmentIdWithShardSpec.fromDataSegment(segment);
+ }
+ }
+ }
+
+
+ // Fetch the pending segments for this interval to determine max partitionId
+ // across all shard specs (published + pending).
+ // A pending segment having a higher partitionId must also be considered
+ // to avoid clashes when inserting the pending segment created here.
+ final Set<SegmentIdWithShardSpec> pendingSegments =
+ getPendingSegmentsForIntervalWithHandle(handle, dataSource, interval);
+ if (committedMaxId != null) {
+ pendingSegments.add(committedMaxId);
+ }
+
+ final Map<SegmentCreateRequest, SegmentIdWithShardSpec> createdSegments = new HashMap<>();
+ final Map<String, SegmentIdWithShardSpec> sequenceHashToSegment = new HashMap<>();
+
+ for (SegmentCreateRequest request : requests) {
+ // Check if the required segment has already been created in this batch
+ final String sequenceHash = getSequenceNameAndPrevIdSha(request, interval, skipSegmentLineageCheck);
+
+ final SegmentIdWithShardSpec createdSegment;
+ if (sequenceHashToSegment.containsKey(sequenceHash)) {
+ createdSegment = sequenceHashToSegment.get(sequenceHash);
+ } else {
+ createdSegment = createNewSegment(
+ request,
+ dataSource,
+ interval,
+ versionOfExistingChunk,
+ committedMaxId,
+ pendingSegments
+ );
+
+ // Add to pendingSegments to consider for partitionId
+ if (createdSegment != null) {
+ pendingSegments.add(createdSegment);
+ sequenceHashToSegment.put(sequenceHash, createdSegment);
+ log.info("Created new segment [%s]", createdSegment);
+ }
+ }
+
+ if (createdSegment != null) {
+ createdSegments.put(request, createdSegment);
+ }
+ }
+
+ log.info("Created [%d] new segments for [%d] allocate requests.", sequenceHashToSegment.size(), requests.size());
+ return createdSegments;
+ }
+
+ private SegmentIdWithShardSpec createNewSegment(
Review Comment:
Thanks for the clarification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1034848291
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,694 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 2000;
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final AtomicBoolean isLeader = new AtomicBoolean(false);
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ ServiceEmitter emitter,
+ ScheduledExecutorFactory executorFactory
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = executorFactory.create(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Initializing segment allocation queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Gets the number of batches currently in the queue.
+ */
+ public int size()
+ {
+ return processingQueue.size();
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!isLeader.get()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If the queue is full,
+ * marks the batch as completed failing all the pending requests.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ log.warn("Cannot add batch [%s] as queue is full. Failing [%d] requests.", batch.key, batch.size());
+ batch.markCompleted();
+ return false;
+ }
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If a batch already
+ * exists for this key, transfers all the requests from this batch to the
+ * existing one.
+ */
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ return addBatchToQueue(batch) ? batch : null;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.transferRequestsFrom(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!isLeader.get()) {
+ log.info("Not leader anymore. Failing [%d] batches in queue.", processingQueue.size());
+
+ // Keep removing items from the queue as long as not leader
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && !isLeader.get()) {
+ processingQueue.pollFirst();
+ keyToBatch.remove(nextBatch.key);
+ nextBatch.markCompleted();
+ nextBatch = processingQueue.peekFirst();
+ }
+ }
+
+ // Check once again for leadership
+ if (!isLeader.get()) {
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution. Overall queue size [%d].", processingQueue.size());
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.pollFirst();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ ++numProcessedBatches;
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.info("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.debug(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
Review Comment:
Thanks! Yeah, I had originally used `attempts` but then moved the code around a bit and this got missed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027900519
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 5000;
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Initializing segment allocation queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If the queue is full,
+ * marks the batch as completed failing all the pending requests.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ log.warn("Cannot add batch [%s] as queue is full. Failing [%d] requests.", batch.key, batch.size());
+ batch.markCompleted();
+ return false;
+ }
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If a batch already
+ * exists for this key, transfers all the requests from this batch to the
+ * existing one.
+ */
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ return addBatchToQueue(batch) ? batch : null;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.transferRequestsFrom(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Failing [%d] batches in queue.", processingQueue.size());
+
+ AllocateRequestBatch nextBatch = processingQueue.pollFirst();
+ while (nextBatch != null) {
+ nextBatch.markCompleted();
+ nextBatch = processingQueue.pollFirst();
+ }
+
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution. Overall queue size [%d].", processingQueue.size());
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.pollFirst();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, index returned from binarySearch is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info(
+ "Elected leader but batched segment allocation is disabled. "
+ + "Segment allocation queue will not be used."
+ );
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ if (isEnabled()) {
+ log.info("Not leader anymore. Stopping queue processing.");
+ } else {
+ log.info("Not leader anymore. Segment allocation queue is already disabled.");
+ }
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * Map from allocate requests (represents a single SegmentAllocateAction)
+ * to the future of allocated segment id.
+ * <p>
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void transferRequestsFrom(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
+ requestToFuture.keySet().forEach(
+ request -> emitTaskMetric("task/action/failed/count", 1L, request)
+ );
+ requestToFuture.clear();
+ }
+ }
+
+ synchronized void handleResult(SegmentAllocateResult result, SegmentAllocateRequest request)
+ {
+ request.incrementAttempts();
+
+ if (result.isSuccess()) {
+ emitTaskMetric("task/action/success/count", 1L, request);
+ requestToFuture.remove(request).complete(result.getSegmentId());
+ } else if (request.canRetry()) {
+ log.debug(
Review Comment:
this should be `info` or `warn` I think. If the retry is causing a delay, we will only know it from logs.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
Review Comment:
well there could be db failures for example. If we don't retry, we should surface the failure to the caller.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,694 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 2000;
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final AtomicBoolean isLeader = new AtomicBoolean(false);
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ ServiceEmitter emitter,
+ ScheduledExecutorFactory executorFactory
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = executorFactory.create(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Initializing segment allocation queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Gets the number of batches currently in the queue.
+ */
+ public int size()
+ {
+ return processingQueue.size();
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!isLeader.get()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If the queue is full,
+ * marks the batch as completed failing all the pending requests.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ log.warn("Cannot add batch [%s] as queue is full. Failing [%d] requests.", batch.key, batch.size());
+ batch.markCompleted();
+ return false;
+ }
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If a batch already
+ * exists for this key, transfers all the requests from this batch to the
+ * existing one.
+ */
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ return addBatchToQueue(batch) ? batch : null;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.transferRequestsFrom(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!isLeader.get()) {
+ log.info("Not leader anymore. Failing [%d] batches in queue.", processingQueue.size());
+
+ // Keep removing items from the queue as long as not leader
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && !isLeader.get()) {
+ processingQueue.pollFirst();
+ keyToBatch.remove(nextBatch.key);
+ nextBatch.markCompleted();
+ nextBatch = processingQueue.peekFirst();
+ }
+ }
+
+ // Check once again for leadership
+ if (!isLeader.get()) {
+ return;
+ }
Review Comment:
is this necessary? what if some items were added to the queue just before line 225?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,694 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 2000;
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final AtomicBoolean isLeader = new AtomicBoolean(false);
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ ServiceEmitter emitter,
+ ScheduledExecutorFactory executorFactory
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = executorFactory.create(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Initializing segment allocation queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Gets the number of batches currently in the queue.
+ */
+ public int size()
+ {
+ return processingQueue.size();
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!isLeader.get()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If the queue is full,
+ * marks the batch as completed failing all the pending requests.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ log.warn("Cannot add batch [%s] as queue is full. Failing [%d] requests.", batch.key, batch.size());
Review Comment:
what happens as a result of this? I assume it will be caused when the metadata store is slow. can there be any other reason? should we add some possible corrective action here like checking metadata store sizing?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
Review Comment:
oh, I see.
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -446,6 +449,140 @@ public LockResult tryLock(final Task task, final LockRequest request)
}
}
+ /**
+ * Attempts to allocate segments for the given requests. Each request contains
+ * a {@link Task} and a {@link SegmentAllocateAction}. This method tries to
+ * acquire the task locks on the required intervals/segments and then performs
+ * a batch allocation of segments. It is possible that some requests succeed
+ * successfully and others failed. In that case, only the failed ones should be
+ * retried.
+ *
+ * @param requests List of allocation requests
+ * @param dataSource Datasource for which segment is to be allocated.
+ * @param interval Interval for which segment is to be allocated.
+ * @param skipSegmentLineageCheck Whether lineage check is to be skipped
+ * (this is true for streaming ingestion)
+ * @param lockGranularity Granularity of task lock
+ * @return List of allocation results in the same order as the requests.
+ */
+ public List<SegmentAllocateResult> allocateSegments(
+ List<SegmentAllocateRequest> requests,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ LockGranularity lockGranularity
+ )
+ {
+ log.info("Allocating [%d] segments for datasource [%s], interval [%s]", requests.size(), dataSource, interval);
+ final boolean isTimeChunkLock = lockGranularity == LockGranularity.TIME_CHUNK;
+
+ final AllocationHolderList holderList = new AllocationHolderList(requests, interval);
+ holderList.getPending().forEach(this::verifyTaskIsActive);
+
+ giant.lock();
+ try {
+ if (isTimeChunkLock) {
+ holderList.getPending().forEach(holder -> acquireTaskLock(holder, true));
+ allocateSegmentIds(dataSource, interval, skipSegmentLineageCheck, holderList.getPending());
+ } else {
+ allocateSegmentIds(dataSource, interval, skipSegmentLineageCheck, holderList.getPending());
+ holderList.getPending().forEach(holder -> acquireTaskLock(holder, false));
+ }
+
+ // TODO: for failed allocations, cleanup newly created locks from the posse map
Review Comment:
can this todo be resolved?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz merged pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz merged PR #13369:
URL: https://github.com/apache/druid/pull/13369
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1034350956
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -183,19 +205,25 @@ private void processBatchesDue()
{
// If not leader, clear the queue and do not schedule any more rounds of processing
if (!leaderSelector.isLeader()) {
- log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
- processingQueue.clear();
+ log.info("Not leader anymore. Failing [%d] batches in queue.", processingQueue.size());
+
+ AllocateRequestBatch nextBatch = processingQueue.pollFirst();
+ while (nextBatch != null) {
+ nextBatch.markCompleted();
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1035555907
##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -777,6 +994,202 @@ private void insertPendingSegmentIntoMetastore(
.execute();
}
+ private Map<SegmentCreateRequest, SegmentIdWithShardSpec> createNewSegments(
+ Handle handle,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ List<SegmentCreateRequest> requests
+ ) throws IOException
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ // Get the time chunk and associated data segments for the given interval, if any
+ final List<TimelineObjectHolder<String, DataSegment>> existingChunks =
+ getTimelineForIntervalsWithHandle(handle, dataSource, Collections.singletonList(interval))
+ .lookup(interval);
+
+ if (existingChunks.size() > 1) {
+ // Not possible to expand more than one chunk with a single segment.
+ log.warn(
+ "Cannot allocate new segments for dataSource[%s], interval[%s]: already have [%,d] chunks.",
+ dataSource,
+ interval,
+ existingChunks.size()
+ );
+ return Collections.emptyMap();
+ }
+
+ // Shard spec of any of the requests (as they are all compatible) can be used to
+ // identify existing shard specs that share partition space with the requested ones.
+ final PartialShardSpec partialShardSpec = requests.get(0).getPartialShardSpec();
+
+ // max partitionId of published data segments which share the same partition space.
+ SegmentIdWithShardSpec committedMaxId = null;
+
+ @Nullable
+ final String versionOfExistingChunk;
+ if (existingChunks.isEmpty()) {
+ versionOfExistingChunk = null;
+ } else {
+ TimelineObjectHolder<String, DataSegment> existingHolder = Iterables.getOnlyElement(existingChunks);
+ versionOfExistingChunk = existingHolder.getVersion();
+
+ // Don't use the stream API for performance.
+ for (DataSegment segment : FluentIterable
+ .from(existingHolder.getObject())
+ .transform(PartitionChunk::getObject)
+ // Here we check only the segments of the shardSpec which shares the same partition space with the given
+ // partialShardSpec. Note that OverwriteShardSpec doesn't share the partition space with others.
+ // See PartitionIds.
+ .filter(segment -> segment.getShardSpec().sharePartitionSpace(partialShardSpec))) {
+ if (committedMaxId == null
+ || committedMaxId.getShardSpec().getPartitionNum() < segment.getShardSpec().getPartitionNum()) {
+ committedMaxId = SegmentIdWithShardSpec.fromDataSegment(segment);
+ }
+ }
+ }
+
+
+ // Fetch the pending segments for this interval to determine max partitionId
+ // across all shard specs (published + pending).
+ // A pending segment having a higher partitionId must also be considered
+ // to avoid clashes when inserting the pending segment created here.
+ final Set<SegmentIdWithShardSpec> pendingSegments =
+ getPendingSegmentsForIntervalWithHandle(handle, dataSource, interval);
+ if (committedMaxId != null) {
+ pendingSegments.add(committedMaxId);
+ }
+
+ final Map<SegmentCreateRequest, SegmentIdWithShardSpec> createdSegments = new HashMap<>();
+ final Map<String, SegmentIdWithShardSpec> sequenceHashToSegment = new HashMap<>();
+
+ for (SegmentCreateRequest request : requests) {
+ // Check if the required segment has already been created in this batch
+ final String sequenceHash = getSequenceNameAndPrevIdSha(request, interval, skipSegmentLineageCheck);
Review Comment:
> SegmentCreateRequest states that requests must be treated distinctly even if all fields are the same.
The `SegmentCreateRequest` is different, the segment id created is not different.
##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -777,6 +994,202 @@ private void insertPendingSegmentIntoMetastore(
.execute();
}
+ private Map<SegmentCreateRequest, SegmentIdWithShardSpec> createNewSegments(
+ Handle handle,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ List<SegmentCreateRequest> requests
+ ) throws IOException
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ // Get the time chunk and associated data segments for the given interval, if any
+ final List<TimelineObjectHolder<String, DataSegment>> existingChunks =
+ getTimelineForIntervalsWithHandle(handle, dataSource, Collections.singletonList(interval))
+ .lookup(interval);
+
+ if (existingChunks.size() > 1) {
+ // Not possible to expand more than one chunk with a single segment.
+ log.warn(
+ "Cannot allocate new segments for dataSource[%s], interval[%s]: already have [%,d] chunks.",
+ dataSource,
+ interval,
+ existingChunks.size()
+ );
+ return Collections.emptyMap();
+ }
+
+ // Shard spec of any of the requests (as they are all compatible) can be used to
+ // identify existing shard specs that share partition space with the requested ones.
+ final PartialShardSpec partialShardSpec = requests.get(0).getPartialShardSpec();
+
+ // max partitionId of published data segments which share the same partition space.
+ SegmentIdWithShardSpec committedMaxId = null;
+
+ @Nullable
+ final String versionOfExistingChunk;
+ if (existingChunks.isEmpty()) {
+ versionOfExistingChunk = null;
+ } else {
+ TimelineObjectHolder<String, DataSegment> existingHolder = Iterables.getOnlyElement(existingChunks);
+ versionOfExistingChunk = existingHolder.getVersion();
+
+ // Don't use the stream API for performance.
+ for (DataSegment segment : FluentIterable
+ .from(existingHolder.getObject())
+ .transform(PartitionChunk::getObject)
+ // Here we check only the segments of the shardSpec which shares the same partition space with the given
+ // partialShardSpec. Note that OverwriteShardSpec doesn't share the partition space with others.
+ // See PartitionIds.
+ .filter(segment -> segment.getShardSpec().sharePartitionSpace(partialShardSpec))) {
+ if (committedMaxId == null
+ || committedMaxId.getShardSpec().getPartitionNum() < segment.getShardSpec().getPartitionNum()) {
+ committedMaxId = SegmentIdWithShardSpec.fromDataSegment(segment);
+ }
+ }
+ }
+
+
+ // Fetch the pending segments for this interval to determine max partitionId
+ // across all shard specs (published + pending).
+ // A pending segment having a higher partitionId must also be considered
+ // to avoid clashes when inserting the pending segment created here.
+ final Set<SegmentIdWithShardSpec> pendingSegments =
+ getPendingSegmentsForIntervalWithHandle(handle, dataSource, interval);
+ if (committedMaxId != null) {
+ pendingSegments.add(committedMaxId);
+ }
+
+ final Map<SegmentCreateRequest, SegmentIdWithShardSpec> createdSegments = new HashMap<>();
+ final Map<String, SegmentIdWithShardSpec> sequenceHashToSegment = new HashMap<>();
+
+ for (SegmentCreateRequest request : requests) {
+ // Check if the required segment has already been created in this batch
+ final String sequenceHash = getSequenceNameAndPrevIdSha(request, interval, skipSegmentLineageCheck);
Review Comment:
> SegmentCreateRequest states that requests must be treated distinctly even if all fields are the same.
The `SegmentCreateRequest` is different, the segment id created is not different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1035570729
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
Review Comment:
We can complete the futures exceptionally. That would also be in line with what the current code would do in case of an exception. Thanks for pointing this out.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
Review Comment:
We can complete the futures exceptionally. That would also be in line with what the old flow would do in case of an exception. Thanks for pointing this out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1034367935
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/LocalTaskActionClient.java:
##########
@@ -76,11 +76,34 @@ public <RetType> RetType submit(TaskAction<RetType> taskAction)
}
final long performStartTime = System.currentTimeMillis();
- final RetType result = taskAction.perform(task, toolbox);
+ final RetType result;
+ if (isBatchAllocateAction(taskAction)) {
+ result = performBatchAllocateAction(taskAction);
+ } else {
+ result = taskAction.perform(task, toolbox);
+ }
emitTimerMetric("task/action/run/time", taskAction, System.currentTimeMillis() - performStartTime);
return result;
}
+ private <R> boolean isBatchAllocateAction(TaskAction<R> taskAction)
+ {
+ return toolbox.canBatchSegmentAllocation()
+ && taskAction instanceof SegmentAllocateAction;
Review Comment:
Added `canPerformAsync` and `performAsync` to `TaskAction`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027783928
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
+ requestToFuture.keySet().forEach(
+ request -> emitTaskMetric("task/action/failed/count", 1L, request)
+ );
+ requestToFuture.clear();
+ }
+ }
+
+ synchronized void handleResult(SegmentAllocateResult result, SegmentAllocateRequest request)
+ {
+ request.incrementAttempts();
+
+ if (result.isSuccess()) {
+ emitTaskMetric("task/action/success/count", 1L, request);
+ requestToFuture.remove(request).complete(result.getSegmentId());
+ return;
+ }
+
+ log.info("Failed to allocate segment for action [%s]: %s", request.getAction(), result.getErrorMessage());
+ if (request.canRetry()) {
+ log.debug(
+ "Can requeue action [%s] after [%d] failed attempts.",
+ request.getAction(),
+ request.getAttempts()
+ );
+ } else {
+ emitTaskMetric("task/action/failed/count", 1L, request);
+ log.error(
+ "Removing allocation action [%s] from batch after [%d] failed attempts.",
+ request.getAction(),
+ request.getAttempts()
+ );
+ requestToFuture.remove(request).complete(null);
+ }
+ }
+
+ synchronized boolean isEmpty()
+ {
+ return requestToFuture.isEmpty();
+ }
+
+ synchronized int size()
+ {
+ return requestToFuture.size();
+ }
+
+ void resetQueueTime()
+ {
+ queueTimeMillis = System.currentTimeMillis();
+ }
+
+ long getQueueTime()
+ {
+ return queueTimeMillis;
+ }
+
+ boolean isDue()
+ {
+ return System.currentTimeMillis() - queueTimeMillis > maxWaitTimeMillis;
+ }
+ }
+
+ /**
+ * Key to identify a batch of allocation requests.
+ */
+ private static class AllocateRequestKey
+ {
+ private final boolean unique;
+
+ private final String dataSource;
+ private final String groupId;
+ private final Interval preferredAllocationInterval;
+ private final Granularity preferredSegmentGranularity;
+
+ private final boolean skipSegmentLineageCheck;
+ private final LockGranularity lockGranularity;
+
+ private final boolean useNonRootGenPartitionSpace;
+
+ private final int hash;
+ private final String serialized;
Review Comment:
Put a limit of 5000 on the queue size.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027629003
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
Review Comment:
Do you think a no-op impl would be useful for tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1319543024
This pull request **introduces 2 alerts** when merging e07e8526ac4d27d5e9e968174e2a51ef6c1efdc1 into bf10ff73a83defda18befb06b8cab1472dead10b - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-3d0ab5ad0f63ab625fd1a27c747d6f12d8deeba3)
**new alerts:**
* 2 for User\-controlled data in numeric cast
---
**Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1318100287
This pull request **introduces 2 alerts** when merging b546f6c8b9b88e4dbdd6bda84d2b6b23b613854b into 78d0b0abce2b27b88a53790684d81530a2318fc9 - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-70ce6d09d4f19f9fcda98bc87ef1d30ad1ba1f3a)
**new alerts:**
* 2 for User\-controlled data in numeric cast
---
**Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1028109019
##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -777,6 +994,202 @@ private void insertPendingSegmentIntoMetastore(
.execute();
}
+ private Map<SegmentCreateRequest, SegmentIdWithShardSpec> createNewSegments(
+ Handle handle,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ List<SegmentCreateRequest> requests
+ ) throws IOException
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ // Get the time chunk and associated data segments for the given interval, if any
+ final List<TimelineObjectHolder<String, DataSegment>> existingChunks =
+ getTimelineForIntervalsWithHandle(handle, dataSource, Collections.singletonList(interval))
+ .lookup(interval);
+
+ if (existingChunks.size() > 1) {
+ // Not possible to expand more than one chunk with a single segment.
+ log.warn(
+ "Cannot allocate new segments for dataSource[%s], interval[%s]: already have [%,d] chunks.",
+ dataSource,
+ interval,
+ existingChunks.size()
+ );
+ return Collections.emptyMap();
+ }
+
+ // Shard spec of any of the requests (as they are all compatible) can be used to
+ // identify existing shard specs that share partition space with the requested ones.
+ final PartialShardSpec partialShardSpec = requests.get(0).getPartialShardSpec();
+
+ // max partitionId of published data segments which share the same partition space.
+ SegmentIdWithShardSpec committedMaxId = null;
+
+ @Nullable
+ final String versionOfExistingChunk;
+ if (existingChunks.isEmpty()) {
+ versionOfExistingChunk = null;
+ } else {
+ TimelineObjectHolder<String, DataSegment> existingHolder = Iterables.getOnlyElement(existingChunks);
+ versionOfExistingChunk = existingHolder.getVersion();
+
+ // Don't use the stream API for performance.
+ for (DataSegment segment : FluentIterable
+ .from(existingHolder.getObject())
+ .transform(PartitionChunk::getObject)
+ // Here we check only the segments of the shardSpec which shares the same partition space with the given
+ // partialShardSpec. Note that OverwriteShardSpec doesn't share the partition space with others.
+ // See PartitionIds.
+ .filter(segment -> segment.getShardSpec().sharePartitionSpace(partialShardSpec))) {
+ if (committedMaxId == null
+ || committedMaxId.getShardSpec().getPartitionNum() < segment.getShardSpec().getPartitionNum()) {
+ committedMaxId = SegmentIdWithShardSpec.fromDataSegment(segment);
+ }
+ }
+ }
+
+
+ // Fetch the pending segments for this interval to determine max partitionId
+ // across all shard specs (published + pending).
+ // A pending segment having a higher partitionId must also be considered
+ // to avoid clashes when inserting the pending segment created here.
+ final Set<SegmentIdWithShardSpec> pendingSegments =
+ getPendingSegmentsForIntervalWithHandle(handle, dataSource, interval);
+ if (committedMaxId != null) {
+ pendingSegments.add(committedMaxId);
+ }
+
+ final Map<SegmentCreateRequest, SegmentIdWithShardSpec> createdSegments = new HashMap<>();
+ final Map<String, SegmentIdWithShardSpec> sequenceHashToSegment = new HashMap<>();
+
+ for (SegmentCreateRequest request : requests) {
+ // Check if the required segment has already been created in this batch
+ final String sequenceHash = getSequenceNameAndPrevIdSha(request, interval, skipSegmentLineageCheck);
+
+ final SegmentIdWithShardSpec createdSegment;
+ if (sequenceHashToSegment.containsKey(sequenceHash)) {
+ createdSegment = sequenceHashToSegment.get(sequenceHash);
+ } else {
+ createdSegment = createNewSegment(
+ request,
+ dataSource,
+ interval,
+ versionOfExistingChunk,
+ committedMaxId,
+ pendingSegments
+ );
+
+ // Add to pendingSegments to consider for partitionId
+ if (createdSegment != null) {
+ pendingSegments.add(createdSegment);
+ sequenceHashToSegment.put(sequenceHash, createdSegment);
+ log.info("Created new segment [%s]", createdSegment);
+ }
+ }
+
+ if (createdSegment != null) {
+ createdSegments.put(request, createdSegment);
+ }
+ }
+
+ log.info("Created [%d] new segments for [%d] allocate requests.", sequenceHashToSegment.size(), requests.size());
+ return createdSegments;
+ }
+
+ private SegmentIdWithShardSpec createNewSegment(
Review Comment:
With time chunk locking, yes. With segment locking, they may be in a batch but still have different versions. This is just because of the way a segment lock is acquired (see `TaskLockbox.tryLock()` and `LockRequestForNewSegment.getVersion()`). The datasource, groupId and interval are always the same in both the cases. (see `AllocateRequestKey` in `SegmentAllocationQueue`).
The new code is written like this in order to minimize risk by retaining the same behaviour as the old flow except club the metadata calls together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027675736
##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -777,6 +994,202 @@ private void insertPendingSegmentIntoMetastore(
.execute();
}
+ private Map<SegmentCreateRequest, SegmentIdWithShardSpec> createNewSegments(
+ Handle handle,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ List<SegmentCreateRequest> requests
+ ) throws IOException
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ // Get the time chunk and associated data segments for the given interval, if any
+ final List<TimelineObjectHolder<String, DataSegment>> existingChunks =
+ getTimelineForIntervalsWithHandle(handle, dataSource, Collections.singletonList(interval))
+ .lookup(interval);
+
+ if (existingChunks.size() > 1) {
+ // Not possible to expand more than one chunk with a single segment.
+ log.warn(
+ "Cannot allocate new segments for dataSource[%s], interval[%s]: already have [%,d] chunks.",
+ dataSource,
+ interval,
+ existingChunks.size()
+ );
+ return Collections.emptyMap();
+ }
+
+ // Shard spec of any of the requests (as they are all compatible) can be used to
+ // identify existing shard specs that share partition space with the requested ones.
+ final PartialShardSpec partialShardSpec = requests.get(0).getPartialShardSpec();
+
+ // max partitionId of published data segments which share the same partition space.
+ SegmentIdWithShardSpec committedMaxId = null;
+
+ @Nullable
+ final String versionOfExistingChunk;
+ if (existingChunks.isEmpty()) {
+ versionOfExistingChunk = null;
+ } else {
+ TimelineObjectHolder<String, DataSegment> existingHolder = Iterables.getOnlyElement(existingChunks);
+ versionOfExistingChunk = existingHolder.getVersion();
+
+ // Don't use the stream API for performance.
+ for (DataSegment segment : FluentIterable
+ .from(existingHolder.getObject())
+ .transform(PartitionChunk::getObject)
+ // Here we check only the segments of the shardSpec which shares the same partition space with the given
+ // partialShardSpec. Note that OverwriteShardSpec doesn't share the partition space with others.
+ // See PartitionIds.
+ .filter(segment -> segment.getShardSpec().sharePartitionSpace(partialShardSpec))) {
+ if (committedMaxId == null
+ || committedMaxId.getShardSpec().getPartitionNum() < segment.getShardSpec().getPartitionNum()) {
+ committedMaxId = SegmentIdWithShardSpec.fromDataSegment(segment);
+ }
+ }
+ }
+
+
+ // Fetch the pending segments for this interval to determine max partitionId
+ // across all shard specs (published + pending).
+ // A pending segment having a higher partitionId must also be considered
+ // to avoid clashes when inserting the pending segment created here.
+ final Set<SegmentIdWithShardSpec> pendingSegments =
+ getPendingSegmentsForIntervalWithHandle(handle, dataSource, interval);
+ if (committedMaxId != null) {
+ pendingSegments.add(committedMaxId);
+ }
+
+ final Map<SegmentCreateRequest, SegmentIdWithShardSpec> createdSegments = new HashMap<>();
+ final Map<String, SegmentIdWithShardSpec> sequenceHashToSegment = new HashMap<>();
+
+ for (SegmentCreateRequest request : requests) {
+ // Check if the required segment has already been created in this batch
+ final String sequenceHash = getSequenceNameAndPrevIdSha(request, interval, skipSegmentLineageCheck);
+
+ final SegmentIdWithShardSpec createdSegment;
+ if (sequenceHashToSegment.containsKey(sequenceHash)) {
+ createdSegment = sequenceHashToSegment.get(sequenceHash);
+ } else {
+ createdSegment = createNewSegment(
+ request,
+ dataSource,
+ interval,
+ versionOfExistingChunk,
+ committedMaxId,
+ pendingSegments
+ );
+
+ // Add to pendingSegments to consider for partitionId
+ if (createdSegment != null) {
+ pendingSegments.add(createdSegment);
+ sequenceHashToSegment.put(sequenceHash, createdSegment);
+ log.info("Created new segment [%s]", createdSegment);
+ }
+ }
+
+ if (createdSegment != null) {
+ createdSegments.put(request, createdSegment);
+ }
+ }
+
+ log.info("Created [%d] new segments for [%d] allocate requests.", sequenceHashToSegment.size(), requests.size());
+ return createdSegments;
+ }
+
+ private SegmentIdWithShardSpec createNewSegment(
Review Comment:
I suppose we could pull out the computation of the `overallMaxId` as that remains the same across the requests in a batch (except including the id of the ids allocated so far). The rest of the stuff in this method will be complicated to pull out because each request can have a different version, depending on the underlying lock that it has created/acquired. Plus, it is not computationally heavy and will not give any perf benefit on commoning out either.
To deduplicate code, the older `createNewSegment` can call this method instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1330261488
This pull request **introduces 2 alerts** when merging 92479da5235b0aeea90d3116fc0a268f400016fd into edd076ca696e3ebcb45c4b0219440a50add9e13a - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-ebe0d530a022b9687f1d8f60f2f465a2cd3f7903)
**new alerts:**
* 2 for User\-controlled data in numeric cast
---
**Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1037766628
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 2000;
+
+ private final long maxWaitTimeMillis;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final AtomicBoolean isLeader = new AtomicBoolean(false);
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ ServiceEmitter emitter,
+ ScheduledExecutorFactory executorFactory
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+
+ this.executor = taskLockConfig.isBatchSegmentAllocation()
+ ? executorFactory.create(1, "SegmentAllocQueue-%s") : null;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ if (isEnabled()) {
+ log.info("Initializing segment allocation queue.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ }
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ if (isEnabled()) {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+ }
+
+ public void becomeLeader()
+ {
+ if (!isLeader.compareAndSet(false, true)) {
+ log.info("Already the leader. Queue processing has started.");
+ } else if (isEnabled()) {
+ log.info("Elected leader. Starting queue processing.");
+ } else {
+ log.info(
+ "Elected leader but batched segment allocation is disabled. "
+ + "Segment allocation queue will not be used."
+ );
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ if (!isLeader.compareAndSet(true, false)) {
+ log.info("Already surrendered leadership. Queue processing is stopped.");
+ } else if (isEnabled()) {
+ log.info("Not leader anymore. Stopping queue processing.");
+ } else {
+ log.info("Not leader anymore. Segment allocation queue is already disabled.");
+ }
+ }
+
+ public boolean isEnabled()
+ {
+ return executor != null && !executor.isShutdown();
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Gets the number of batches currently in the queue.
+ */
+ public int size()
+ {
+ return processingQueue.size();
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!isLeader.get()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. Fails all the pending
+ * requests in the batch if we are not leader or if the queue is full.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (!isLeader.get()) {
+ batch.failPendingRequests("Cannot allocate segment if not leader");
+ return false;
+ } else if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ batch.failPendingRequests(
+ "Segment allocation queue is full. Check the metric `task/action/batch/runTime` "
+ + "to determine if metadata operations are slow."
+ );
+ return false;
+ }
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If a batch already
+ * exists for this key, transfers all the requests from this batch to the
+ * existing one.
+ */
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ return addBatchToQueue(batch) ? batch : null;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.transferRequestsFrom(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ clearQueueIfNotLeader();
+
+ // Process all batches which are due
+ log.debug("Processing batches which are due. Queue size [%d].", processingQueue.size());
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.pollFirst();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ }
+ catch (Throwable t) {
+ nextBatch.failPendingRequests(t);
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ // Requeue if not fully processed yet
+ if (processed) {
+ ++numProcessedBatches;
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.info("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Removes items from the queue as long as we are not leader.
+ */
+ private void clearQueueIfNotLeader()
+ {
+ int failedBatches = 0;
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && !isLeader.get()) {
+ processingQueue.pollFirst();
+ keyToBatch.remove(nextBatch.key);
+ nextBatch.failPendingRequests("Cannot allocate segment if not leader");
+ ++failedBatches;
+
+ nextBatch = processingQueue.peekFirst();
+ }
+ if (failedBatches > 0) {
+ log.info("Not leader. Failed [%d] batches, remaining in queue [%d].", failedBatches, processingQueue.size());
+ }
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ } else if (!isLeader.get()) {
+ requestBatch.failPendingRequests("Cannot allocate segment if not leader");
+ return true;
+ }
+
+ log.debug(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/attempts", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.debug("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.debug("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ requestBatch.failPendingRequests("Allocation failed probably due to conflicting segments.");
+ return true;
+ } else {
+ log.debug("Used segments have changed. Requeuing failed requests.");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ // Find requests whose row interval overlaps with an existing used segment
+ final Set<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> requestsWithNoOverlappingSegment = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ requestsWithNoOverlappingSegment.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> overlapIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment, the interval of the used segment
Review Comment:
The used segments for the allocate interval (`segmentGranularity.bucket(row.timestamp())`) are fetched so that we don't have to make the metadata call multiple times.
The logic hasn't changed because we use the `rowInterval` (`queryGranularity.bucket(row.timestamp())`) to find the final overlapping segments. Please look at the line after this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1035544105
##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -777,6 +994,202 @@ private void insertPendingSegmentIntoMetastore(
.execute();
}
+ private Map<SegmentCreateRequest, SegmentIdWithShardSpec> createNewSegments(
+ Handle handle,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ List<SegmentCreateRequest> requests
+ ) throws IOException
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ // Get the time chunk and associated data segments for the given interval, if any
+ final List<TimelineObjectHolder<String, DataSegment>> existingChunks =
+ getTimelineForIntervalsWithHandle(handle, dataSource, Collections.singletonList(interval))
+ .lookup(interval);
+
+ if (existingChunks.size() > 1) {
+ // Not possible to expand more than one chunk with a single segment.
+ log.warn(
+ "Cannot allocate new segments for dataSource[%s], interval[%s]: already have [%,d] chunks.",
+ dataSource,
+ interval,
+ existingChunks.size()
+ );
+ return Collections.emptyMap();
+ }
+
+ // Shard spec of any of the requests (as they are all compatible) can be used to
+ // identify existing shard specs that share partition space with the requested ones.
+ final PartialShardSpec partialShardSpec = requests.get(0).getPartialShardSpec();
+
+ // max partitionId of published data segments which share the same partition space.
+ SegmentIdWithShardSpec committedMaxId = null;
+
+ @Nullable
+ final String versionOfExistingChunk;
+ if (existingChunks.isEmpty()) {
+ versionOfExistingChunk = null;
+ } else {
+ TimelineObjectHolder<String, DataSegment> existingHolder = Iterables.getOnlyElement(existingChunks);
+ versionOfExistingChunk = existingHolder.getVersion();
+
+ // Don't use the stream API for performance.
+ for (DataSegment segment : FluentIterable
+ .from(existingHolder.getObject())
+ .transform(PartitionChunk::getObject)
+ // Here we check only the segments of the shardSpec which shares the same partition space with the given
+ // partialShardSpec. Note that OverwriteShardSpec doesn't share the partition space with others.
+ // See PartitionIds.
+ .filter(segment -> segment.getShardSpec().sharePartitionSpace(partialShardSpec))) {
+ if (committedMaxId == null
+ || committedMaxId.getShardSpec().getPartitionNum() < segment.getShardSpec().getPartitionNum()) {
+ committedMaxId = SegmentIdWithShardSpec.fromDataSegment(segment);
+ }
+ }
+ }
+
+
+ // Fetch the pending segments for this interval to determine max partitionId
+ // across all shard specs (published + pending).
+ // A pending segment having a higher partitionId must also be considered
+ // to avoid clashes when inserting the pending segment created here.
+ final Set<SegmentIdWithShardSpec> pendingSegments =
+ getPendingSegmentsForIntervalWithHandle(handle, dataSource, interval);
+ if (committedMaxId != null) {
+ pendingSegments.add(committedMaxId);
+ }
+
+ final Map<SegmentCreateRequest, SegmentIdWithShardSpec> createdSegments = new HashMap<>();
+ final Map<String, SegmentIdWithShardSpec> sequenceHashToSegment = new HashMap<>();
+
+ for (SegmentCreateRequest request : requests) {
+ // Check if the required segment has already been created in this batch
+ final String sequenceHash = getSequenceNameAndPrevIdSha(request, interval, skipSegmentLineageCheck);
Review Comment:
When can this occur?
Also, SegmentCreateRequest states that requests must be treated distinctly even if all fields are the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027783928
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
+ requestToFuture.keySet().forEach(
+ request -> emitTaskMetric("task/action/failed/count", 1L, request)
+ );
+ requestToFuture.clear();
+ }
+ }
+
+ synchronized void handleResult(SegmentAllocateResult result, SegmentAllocateRequest request)
+ {
+ request.incrementAttempts();
+
+ if (result.isSuccess()) {
+ emitTaskMetric("task/action/success/count", 1L, request);
+ requestToFuture.remove(request).complete(result.getSegmentId());
+ return;
+ }
+
+ log.info("Failed to allocate segment for action [%s]: %s", request.getAction(), result.getErrorMessage());
+ if (request.canRetry()) {
+ log.debug(
+ "Can requeue action [%s] after [%d] failed attempts.",
+ request.getAction(),
+ request.getAttempts()
+ );
+ } else {
+ emitTaskMetric("task/action/failed/count", 1L, request);
+ log.error(
+ "Removing allocation action [%s] from batch after [%d] failed attempts.",
+ request.getAction(),
+ request.getAttempts()
+ );
+ requestToFuture.remove(request).complete(null);
+ }
+ }
+
+ synchronized boolean isEmpty()
+ {
+ return requestToFuture.isEmpty();
+ }
+
+ synchronized int size()
+ {
+ return requestToFuture.size();
+ }
+
+ void resetQueueTime()
+ {
+ queueTimeMillis = System.currentTimeMillis();
+ }
+
+ long getQueueTime()
+ {
+ return queueTimeMillis;
+ }
+
+ boolean isDue()
+ {
+ return System.currentTimeMillis() - queueTimeMillis > maxWaitTimeMillis;
+ }
+ }
+
+ /**
+ * Key to identify a batch of allocation requests.
+ */
+ private static class AllocateRequestKey
+ {
+ private final boolean unique;
+
+ private final String dataSource;
+ private final String groupId;
+ private final Interval preferredAllocationInterval;
+ private final Granularity preferredSegmentGranularity;
+
+ private final boolean skipSegmentLineageCheck;
+ private final LockGranularity lockGranularity;
+
+ private final boolean useNonRootGenPartitionSpace;
+
+ private final int hash;
+ private final String serialized;
Review Comment:
Put a limit of 2000 on the queue size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1333336126
@AmatyaAvadhanula , @abhishekagarwal87 , thanks for the review. I have addressed your feedback.
Changes made:
- Queue polling is now always active irrespective of being leader or not. This can be improved later if needed.
- Queue polling starts on lifecycle start and not on becoming leader.
- A future now either completes successfully or exceptionally, i.e. it never completes with null. This allows propagation of actual error message to the caller. Added tests for this.
- Clarify usage of `committedMaxId` in the set of `pendingSegments`
- Rename metric `batch/retries` to `batch/attempts`
Changes in follow up PR:
- docs (mostly just metrics)
- cleanup of empty locks from posse map (both old and new flow)
- limit number of requests in each batch
- cancel the future if it times out on `LocalTaskActionClient`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1325111435
This pull request **introduces 2 alerts** when merging 1d7a688bb4b9d0edffe37c1e235a677277c8ae2c into f524c68f08eb58047064e8234d320b51e83499ce - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-751770365d436ca1efeeb309956de8ca31ab2dba)
**new alerts:**
* 2 for User\-controlled data in numeric cast
---
**Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027783217
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
+ requestToFuture.keySet().forEach(
+ request -> emitTaskMetric("task/action/failed/count", 1L, request)
+ );
+ requestToFuture.clear();
+ }
+ }
+
+ synchronized void handleResult(SegmentAllocateResult result, SegmentAllocateRequest request)
+ {
+ request.incrementAttempts();
+
+ if (result.isSuccess()) {
+ emitTaskMetric("task/action/success/count", 1L, request);
+ requestToFuture.remove(request).complete(result.getSegmentId());
+ return;
+ }
+
+ log.info("Failed to allocate segment for action [%s]: %s", request.getAction(), result.getErrorMessage());
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027785622
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
Review Comment:
A new batch may get added but that shouldn't be a problem as only thread would be polling from the queue at any given time. So the result returned by a `poll` would be the same as the result returned by the preceding `peek`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1335366333
This pull request **introduces 2 alerts** when merging 29bc0d4d3e3c67c0f84bf431a9d39778d204d35e into 30498c1f98346050a9121bae113ace4fd4059302 - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-95dc9b0e1e8cd1a934554eb0db1502c9951570ce)
**new alerts:**
* 2 for User\-controlled data in numeric cast
---
**Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1035529643
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,694 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 2000;
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final AtomicBoolean isLeader = new AtomicBoolean(false);
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ ServiceEmitter emitter,
+ ScheduledExecutorFactory executorFactory
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = executorFactory.create(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Initializing segment allocation queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Gets the number of batches currently in the queue.
+ */
+ public int size()
+ {
+ return processingQueue.size();
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!isLeader.get()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If the queue is full,
+ * marks the batch as completed failing all the pending requests.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ log.warn("Cannot add batch [%s] as queue is full. Failing [%d] requests.", batch.key, batch.size());
+ batch.markCompleted();
+ return false;
+ }
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If a batch already
+ * exists for this key, transfers all the requests from this batch to the
+ * existing one.
+ */
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ return addBatchToQueue(batch) ? batch : null;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.transferRequestsFrom(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!isLeader.get()) {
+ log.info("Not leader anymore. Failing [%d] batches in queue.", processingQueue.size());
+
+ // Keep removing items from the queue as long as not leader
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && !isLeader.get()) {
+ processingQueue.pollFirst();
+ keyToBatch.remove(nextBatch.key);
+ nextBatch.markCompleted();
+ nextBatch = processingQueue.peekFirst();
+ }
+ }
+
+ // Check once again for leadership
+ if (!isLeader.get()) {
+ return;
+ }
Review Comment:
I guess that is not likely to happen. New items are rejected if we are not the leader.
But maybe it could happen if the leader is flapping?
- t1: we are not leader, we clear the queue
- t2: become leader
- t3: add a request
- t4: stop being leader
- t1: we are not leader, return
the request added by t3 will thus remain stuck in the queue, until we become leader again and process it. The future will timeout after 5 minutes.
A simpler alternative to all of this can be to have the leadership check done for each batch separately in `processBatch`. If not leader, fail it there.
The differences would be:
- even non-leaders would keep polling the queue
- requests would wait until their due time to be failed
Please let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1035529643
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,694 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 2000;
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final AtomicBoolean isLeader = new AtomicBoolean(false);
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ ServiceEmitter emitter,
+ ScheduledExecutorFactory executorFactory
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = executorFactory.create(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Initializing segment allocation queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Gets the number of batches currently in the queue.
+ */
+ public int size()
+ {
+ return processingQueue.size();
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!isLeader.get()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If the queue is full,
+ * marks the batch as completed failing all the pending requests.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ log.warn("Cannot add batch [%s] as queue is full. Failing [%d] requests.", batch.key, batch.size());
+ batch.markCompleted();
+ return false;
+ }
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If a batch already
+ * exists for this key, transfers all the requests from this batch to the
+ * existing one.
+ */
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ return addBatchToQueue(batch) ? batch : null;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.transferRequestsFrom(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!isLeader.get()) {
+ log.info("Not leader anymore. Failing [%d] batches in queue.", processingQueue.size());
+
+ // Keep removing items from the queue as long as not leader
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && !isLeader.get()) {
+ processingQueue.pollFirst();
+ keyToBatch.remove(nextBatch.key);
+ nextBatch.markCompleted();
+ nextBatch = processingQueue.peekFirst();
+ }
+ }
+
+ // Check once again for leadership
+ if (!isLeader.get()) {
+ return;
+ }
Review Comment:
I guess that is not likely to happen. New items are rejected if we are not the leader.
But maybe it could happen if the leader is flapping?
- t1: we are not leader, we clear the queue
- t2: become leader
- t3: add a request
- t4: stop being leader
- t1: we are not leader, return
the request added by t3 will thus remain stuck in the queue, until we become leader again and process it. The future will timeout after 5 minutes.
A simpler alternative to all of this can be to have the leadership check done for each batch separately in `processBatch`. If not leader, fail it there. The only difference would be that even non-leaders would keep polling the queue. I wanted to avoid that somehow. Please let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1035555727
##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -777,6 +994,202 @@ private void insertPendingSegmentIntoMetastore(
.execute();
}
+ private Map<SegmentCreateRequest, SegmentIdWithShardSpec> createNewSegments(
+ Handle handle,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ List<SegmentCreateRequest> requests
+ ) throws IOException
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ // Get the time chunk and associated data segments for the given interval, if any
+ final List<TimelineObjectHolder<String, DataSegment>> existingChunks =
+ getTimelineForIntervalsWithHandle(handle, dataSource, Collections.singletonList(interval))
+ .lookup(interval);
+
+ if (existingChunks.size() > 1) {
+ // Not possible to expand more than one chunk with a single segment.
+ log.warn(
+ "Cannot allocate new segments for dataSource[%s], interval[%s]: already have [%,d] chunks.",
+ dataSource,
+ interval,
+ existingChunks.size()
+ );
+ return Collections.emptyMap();
+ }
+
+ // Shard spec of any of the requests (as they are all compatible) can be used to
+ // identify existing shard specs that share partition space with the requested ones.
+ final PartialShardSpec partialShardSpec = requests.get(0).getPartialShardSpec();
+
+ // max partitionId of published data segments which share the same partition space.
+ SegmentIdWithShardSpec committedMaxId = null;
+
+ @Nullable
+ final String versionOfExistingChunk;
+ if (existingChunks.isEmpty()) {
+ versionOfExistingChunk = null;
+ } else {
+ TimelineObjectHolder<String, DataSegment> existingHolder = Iterables.getOnlyElement(existingChunks);
+ versionOfExistingChunk = existingHolder.getVersion();
+
+ // Don't use the stream API for performance.
+ for (DataSegment segment : FluentIterable
+ .from(existingHolder.getObject())
+ .transform(PartitionChunk::getObject)
+ // Here we check only the segments of the shardSpec which shares the same partition space with the given
+ // partialShardSpec. Note that OverwriteShardSpec doesn't share the partition space with others.
+ // See PartitionIds.
+ .filter(segment -> segment.getShardSpec().sharePartitionSpace(partialShardSpec))) {
+ if (committedMaxId == null
+ || committedMaxId.getShardSpec().getPartitionNum() < segment.getShardSpec().getPartitionNum()) {
+ committedMaxId = SegmentIdWithShardSpec.fromDataSegment(segment);
+ }
+ }
+ }
+
+
+ // Fetch the pending segments for this interval to determine max partitionId
+ // across all shard specs (published + pending).
+ // A pending segment having a higher partitionId must also be considered
+ // to avoid clashes when inserting the pending segment created here.
+ final Set<SegmentIdWithShardSpec> pendingSegments =
+ getPendingSegmentsForIntervalWithHandle(handle, dataSource, interval);
+ if (committedMaxId != null) {
+ pendingSegments.add(committedMaxId);
+ }
+
+ final Map<SegmentCreateRequest, SegmentIdWithShardSpec> createdSegments = new HashMap<>();
+ final Map<String, SegmentIdWithShardSpec> sequenceHashToSegment = new HashMap<>();
+
+ for (SegmentCreateRequest request : requests) {
+ // Check if the required segment has already been created in this batch
+ final String sequenceHash = getSequenceNameAndPrevIdSha(request, interval, skipSegmentLineageCheck);
Review Comment:
Multiple requests for the same segment, in the case of task replicas, I should think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1034415163
##########
core/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -68,6 +72,50 @@ public static boolean isEternity(final Interval interval)
return ETERNITY.equals(interval);
}
+ /**
+ * Finds an interval from the given set of sortedIntervals which overlaps with
+ * the searchInterval. If multiple candidate intervals overlap with the
+ * searchInterval, the "smallest" interval based on the
+ * {@link Comparators#intervalsByStartThenEnd()} is returned.
+ *
+ * @param searchInterval Interval which should overlap with the result
+ * @param sortedIntervals Candidate overlapping intervals, sorted in ascending
+ * order, using {@link Comparators#intervalsByStartThenEnd()}.
+ * @return The first overlapping interval, if one exists, otherwise null.
+ */
+ @Nullable
+ public static Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ Arrays.sort(sortedIntervals, Comparators.intervalsByStartThenEnd());
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, index returned from binarySearch is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
Review Comment:
Thanks for catching this, @AmatyaAvadhanula ! Let me fix up the method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1034817428
##########
core/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -68,6 +72,50 @@ public static boolean isEternity(final Interval interval)
return ETERNITY.equals(interval);
}
+ /**
+ * Finds an interval from the given set of sortedIntervals which overlaps with
+ * the searchInterval. If multiple candidate intervals overlap with the
+ * searchInterval, the "smallest" interval based on the
+ * {@link Comparators#intervalsByStartThenEnd()} is returned.
+ *
+ * @param searchInterval Interval which should overlap with the result
+ * @param sortedIntervals Candidate overlapping intervals, sorted in ascending
+ * order, using {@link Comparators#intervalsByStartThenEnd()}.
+ * @return The first overlapping interval, if one exists, otherwise null.
+ */
+ @Nullable
+ public static Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ Arrays.sort(sortedIntervals, Comparators.intervalsByStartThenEnd());
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, index returned from binarySearch is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
Review Comment:
Simplified interval search. The binary search doesn't seem to have a significant benefit in the avg case even with 100k search intervals and 100k invocations of `findOverlappingInterval`. The method can be optimized in the future if needed, seems overkill for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1319768090
This pull request **introduces 2 alerts** when merging 3a8556811295ec67bb62b45f38cb8b743890b771 into 7f4e386509855a2646f7aeb4ecee97145f5437ca - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-91f7c1781e81045596ee3507c50b2f97c434d0d2)
**new alerts:**
* 2 for User\-controlled data in numeric cast
---
**Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027637052
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
Review Comment:
Hmm, I think having a no-op implementation would require a factory that creates the appropriate `SegmentAllocationQueue` based on the given configs. But that would be cleaner, I suppose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027613635
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
Review Comment:
Shouldn't be a lot. If there are too many tasks running, chances are they would end up sharing a batch.
This log information could be useful when not looking at metrics and trying to verify if batches are getting added to the queue. But I guess it can be a debug log line.
Maybe I should just update the log line in the processing method which prints the current queue size. The should be enough, as we already print the number of requests inside a batch when we start processing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027792447
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -183,19 +205,25 @@ private void processBatchesDue()
{
// If not leader, clear the queue and do not schedule any more rounds of processing
if (!leaderSelector.isLeader()) {
- log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
- processingQueue.clear();
+ log.info("Not leader anymore. Failing [%d] batches in queue.", processingQueue.size());
+
+ AllocateRequestBatch nextBatch = processingQueue.pollFirst();
+ while (nextBatch != null) {
+ nextBatch.markCompleted();
Review Comment:
Note: Need to make sure that this doesn't keep failing batches and meanwhile the overlord gets re-elected as the leader. Probably re-check the leader condition in the while loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1322019421
This pull request **introduces 2 alerts** when merging b6da3c48a525adef2d4f08841511084855308074 into 6c3f688a66775c26bb33630766b0badf6c004541 - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-fc2c4b2ff37d54b18aba139736416b2c75841361)
**new alerts:**
* 2 for User\-controlled data in numeric cast
---
**Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027782456
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
Review Comment:
Updated these logs to be less verbose.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
Review Comment:
Thanks for the suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] rohangarg commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
rohangarg commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1321552689
One note : While I thought that batching was probably the way to go earlier, probably it could be re-considered in favour of caching and combining it with more parallelism if that's easier. The current batching idea also caches data in memory for pendingSegments table and updates the in-memory state for all requests in a batch to resolve their partition-num assignment conflict.
In a local experiment, I was trying the following :
1. Add striped locking for `(datasource, interval, sequence name)` tuple in `allocatePendingSegment` method
2. The `createNewSegment` method is the critical section which requires a global lock, but it operates on cached results of `pendingSegments` table. The cache could be for key `(datasource, interval)` and gets invalidated with time, access-count or some transaction failure while committing to pendingSegments table
3. The write to pendingSegments table is also done in the striped lock
4. More things could be extracted from `createNewSegment` (like fetching used segments) to the striped locking flow so that there's more parallelism
Generally, the idea was to do DB calls in parallel threads and use cached objects in the global lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027781282
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
Review Comment:
Done.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1319183699
This pull request **introduces 2 alerts** when merging 4c66afd2f11365b234936ea59a1050364a07edce into bf10ff73a83defda18befb06b8cab1472dead10b - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-9164c9645308208a8a4b5754e247dde25fe19c4e)
**new alerts:**
* 2 for User\-controlled data in numeric cast
---
**Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1333509092
This pull request **introduces 2 alerts** when merging 0d09a6ecc6382358d176ec0dead09d543dcd1242 into f6f625ee088e955af01974c3850e095d7d52496a - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-0a3d24bac0b7ce2aae99333d878aca679cf65173)
**new alerts:**
* 2 for User\-controlled data in numeric cast
---
**Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1037759883
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 2000;
+
+ private final long maxWaitTimeMillis;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final AtomicBoolean isLeader = new AtomicBoolean(false);
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ ServiceEmitter emitter,
+ ScheduledExecutorFactory executorFactory
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+
+ this.executor = taskLockConfig.isBatchSegmentAllocation()
+ ? executorFactory.create(1, "SegmentAllocQueue-%s") : null;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ if (isEnabled()) {
+ log.info("Initializing segment allocation queue.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ }
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ if (isEnabled()) {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+ }
+
+ public void becomeLeader()
+ {
+ if (!isLeader.compareAndSet(false, true)) {
+ log.info("Already the leader. Queue processing has started.");
+ } else if (isEnabled()) {
+ log.info("Elected leader. Starting queue processing.");
+ } else {
+ log.info(
+ "Elected leader but batched segment allocation is disabled. "
+ + "Segment allocation queue will not be used."
+ );
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ if (!isLeader.compareAndSet(true, false)) {
+ log.info("Already surrendered leadership. Queue processing is stopped.");
+ } else if (isEnabled()) {
+ log.info("Not leader anymore. Stopping queue processing.");
+ } else {
+ log.info("Not leader anymore. Segment allocation queue is already disabled.");
+ }
+ }
+
+ public boolean isEnabled()
+ {
+ return executor != null && !executor.isShutdown();
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Gets the number of batches currently in the queue.
+ */
+ public int size()
+ {
+ return processingQueue.size();
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!isLeader.get()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. Fails all the pending
+ * requests in the batch if we are not leader or if the queue is full.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (!isLeader.get()) {
+ batch.failPendingRequests("Cannot allocate segment if not leader");
+ return false;
+ } else if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ batch.failPendingRequests(
+ "Segment allocation queue is full. Check the metric `task/action/batch/runTime` "
+ + "to determine if metadata operations are slow."
+ );
+ return false;
+ }
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If a batch already
+ * exists for this key, transfers all the requests from this batch to the
+ * existing one.
+ */
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ return addBatchToQueue(batch) ? batch : null;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.transferRequestsFrom(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ clearQueueIfNotLeader();
+
+ // Process all batches which are due
+ log.debug("Processing batches which are due. Queue size [%d].", processingQueue.size());
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.pollFirst();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ }
+ catch (Throwable t) {
+ nextBatch.failPendingRequests(t);
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ // Requeue if not fully processed yet
+ if (processed) {
+ ++numProcessedBatches;
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.info("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Removes items from the queue as long as we are not leader.
+ */
+ private void clearQueueIfNotLeader()
+ {
+ int failedBatches = 0;
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && !isLeader.get()) {
+ processingQueue.pollFirst();
+ keyToBatch.remove(nextBatch.key);
+ nextBatch.failPendingRequests("Cannot allocate segment if not leader");
+ ++failedBatches;
+
+ nextBatch = processingQueue.peekFirst();
+ }
+ if (failedBatches > 0) {
+ log.info("Not leader. Failed [%d] batches, remaining in queue [%d].", failedBatches, processingQueue.size());
+ }
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ } else if (!isLeader.get()) {
+ requestBatch.failPendingRequests("Cannot allocate segment if not leader");
+ return true;
+ }
+
+ log.debug(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/attempts", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.debug("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.debug("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ requestBatch.failPendingRequests("Allocation failed probably due to conflicting segments.");
+ return true;
+ } else {
+ log.debug("Used segments have changed. Requeuing failed requests.");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ // Find requests whose row interval overlaps with an existing used segment
+ final Set<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> requestsWithNoOverlappingSegment = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ requestsWithNoOverlappingSegment.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> overlapIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment, the interval of the used segment
Review Comment:
The allocation for a single segment considers overlap with queryGranularity.bucket(row.timestamp())
This code fetches all used segments overlapping with segmentGranularity.bucket(row.timestamp()). I think this could lead to a case where the row is allocated to a used segment interval to which it doesn't belong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1034850126
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,694 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 2000;
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final AtomicBoolean isLeader = new AtomicBoolean(false);
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ ServiceEmitter emitter,
+ ScheduledExecutorFactory executorFactory
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = executorFactory.create(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Initializing segment allocation queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Gets the number of batches currently in the queue.
+ */
+ public int size()
+ {
+ return processingQueue.size();
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!isLeader.get()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If the queue is full,
+ * marks the batch as completed failing all the pending requests.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ log.warn("Cannot add batch [%s] as queue is full. Failing [%d] requests.", batch.key, batch.size());
+ batch.markCompleted();
+ return false;
+ }
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If a batch already
+ * exists for this key, transfers all the requests from this batch to the
+ * existing one.
+ */
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ return addBatchToQueue(batch) ? batch : null;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.transferRequestsFrom(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!isLeader.get()) {
+ log.info("Not leader anymore. Failing [%d] batches in queue.", processingQueue.size());
+
+ // Keep removing items from the queue as long as not leader
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && !isLeader.get()) {
+ processingQueue.pollFirst();
+ keyToBatch.remove(nextBatch.key);
+ nextBatch.markCompleted();
+ nextBatch = processingQueue.peekFirst();
+ }
+ }
+
+ // Check once again for leadership
+ if (!isLeader.get()) {
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution. Overall queue size [%d].", processingQueue.size());
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.pollFirst();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ ++numProcessedBatches;
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.info("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.debug(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.debug("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
Review Comment:
This is to retain behaviour from the existing flow. If the set of used segments has changed since we started processing the request, it's possible that retrying might make it pass.
https://github.com/apache/druid/blob/79df11c16cd2fd1c6a9867855c42768a079917ff/indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocateAction.java#L223
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1035555907
##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -777,6 +994,202 @@ private void insertPendingSegmentIntoMetastore(
.execute();
}
+ private Map<SegmentCreateRequest, SegmentIdWithShardSpec> createNewSegments(
+ Handle handle,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ List<SegmentCreateRequest> requests
+ ) throws IOException
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ // Get the time chunk and associated data segments for the given interval, if any
+ final List<TimelineObjectHolder<String, DataSegment>> existingChunks =
+ getTimelineForIntervalsWithHandle(handle, dataSource, Collections.singletonList(interval))
+ .lookup(interval);
+
+ if (existingChunks.size() > 1) {
+ // Not possible to expand more than one chunk with a single segment.
+ log.warn(
+ "Cannot allocate new segments for dataSource[%s], interval[%s]: already have [%,d] chunks.",
+ dataSource,
+ interval,
+ existingChunks.size()
+ );
+ return Collections.emptyMap();
+ }
+
+ // Shard spec of any of the requests (as they are all compatible) can be used to
+ // identify existing shard specs that share partition space with the requested ones.
+ final PartialShardSpec partialShardSpec = requests.get(0).getPartialShardSpec();
+
+ // max partitionId of published data segments which share the same partition space.
+ SegmentIdWithShardSpec committedMaxId = null;
+
+ @Nullable
+ final String versionOfExistingChunk;
+ if (existingChunks.isEmpty()) {
+ versionOfExistingChunk = null;
+ } else {
+ TimelineObjectHolder<String, DataSegment> existingHolder = Iterables.getOnlyElement(existingChunks);
+ versionOfExistingChunk = existingHolder.getVersion();
+
+ // Don't use the stream API for performance.
+ for (DataSegment segment : FluentIterable
+ .from(existingHolder.getObject())
+ .transform(PartitionChunk::getObject)
+ // Here we check only the segments of the shardSpec which shares the same partition space with the given
+ // partialShardSpec. Note that OverwriteShardSpec doesn't share the partition space with others.
+ // See PartitionIds.
+ .filter(segment -> segment.getShardSpec().sharePartitionSpace(partialShardSpec))) {
+ if (committedMaxId == null
+ || committedMaxId.getShardSpec().getPartitionNum() < segment.getShardSpec().getPartitionNum()) {
+ committedMaxId = SegmentIdWithShardSpec.fromDataSegment(segment);
+ }
+ }
+ }
+
+
+ // Fetch the pending segments for this interval to determine max partitionId
+ // across all shard specs (published + pending).
+ // A pending segment having a higher partitionId must also be considered
+ // to avoid clashes when inserting the pending segment created here.
+ final Set<SegmentIdWithShardSpec> pendingSegments =
+ getPendingSegmentsForIntervalWithHandle(handle, dataSource, interval);
+ if (committedMaxId != null) {
+ pendingSegments.add(committedMaxId);
+ }
+
+ final Map<SegmentCreateRequest, SegmentIdWithShardSpec> createdSegments = new HashMap<>();
+ final Map<String, SegmentIdWithShardSpec> sequenceHashToSegment = new HashMap<>();
+
+ for (SegmentCreateRequest request : requests) {
+ // Check if the required segment has already been created in this batch
+ final String sequenceHash = getSequenceNameAndPrevIdSha(request, interval, skipSegmentLineageCheck);
Review Comment:
The `SegmentCreateRequest` is different, the segment id created is not different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1336739387
All unit and integration tests have run successfully with segment allocation enabled in test PR #13484 .
There was only one flaky test. I couldn't retry that test due to some Travis access issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1336942537
Merging as build is green. Latest commit is only comment change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027666790
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
+ requestToFuture.keySet().forEach(
+ request -> emitTaskMetric("task/action/failed/count", 1L, request)
+ );
+ requestToFuture.clear();
+ }
+ }
+
+ synchronized void handleResult(SegmentAllocateResult result, SegmentAllocateRequest request)
+ {
+ request.incrementAttempts();
+
+ if (result.isSuccess()) {
+ emitTaskMetric("task/action/success/count", 1L, request);
+ requestToFuture.remove(request).complete(result.getSegmentId());
+ return;
+ }
+
+ log.info("Failed to allocate segment for action [%s]: %s", request.getAction(), result.getErrorMessage());
+ if (request.canRetry()) {
+ log.debug(
+ "Can requeue action [%s] after [%d] failed attempts.",
+ request.getAction(),
+ request.getAttempts()
+ );
+ } else {
+ emitTaskMetric("task/action/failed/count", 1L, request);
+ log.error(
+ "Removing allocation action [%s] from batch after [%d] failed attempts.",
+ request.getAction(),
+ request.getAttempts()
+ );
+ requestToFuture.remove(request).complete(null);
+ }
+ }
+
+ synchronized boolean isEmpty()
+ {
+ return requestToFuture.isEmpty();
+ }
+
+ synchronized int size()
+ {
+ return requestToFuture.size();
+ }
+
+ void resetQueueTime()
+ {
+ queueTimeMillis = System.currentTimeMillis();
+ }
+
+ long getQueueTime()
+ {
+ return queueTimeMillis;
+ }
+
+ boolean isDue()
+ {
+ return System.currentTimeMillis() - queueTimeMillis > maxWaitTimeMillis;
+ }
+ }
+
+ /**
+ * Key to identify a batch of allocation requests.
+ */
+ private static class AllocateRequestKey
+ {
+ private final boolean unique;
+
+ private final String dataSource;
+ private final String groupId;
+ private final Interval preferredAllocationInterval;
+ private final Granularity preferredSegmentGranularity;
+
+ private final boolean skipSegmentLineageCheck;
+ private final LockGranularity lockGranularity;
+
+ private final boolean useNonRootGenPartitionSpace;
+
+ private final int hash;
+ private final String serialized;
Review Comment:
yeah. we don't want the Overlord to crash with an OOM. So having some guardrail will be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027782761
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
Review Comment:
added.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
Review Comment:
added. let me know if it seems fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027781825
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/LocalTaskActionClient.java:
##########
@@ -76,11 +76,34 @@ public <RetType> RetType submit(TaskAction<RetType> taskAction)
}
final long performStartTime = System.currentTimeMillis();
- final RetType result = taskAction.perform(task, toolbox);
+ final RetType result;
+ if (isBatchAllocateAction(taskAction)) {
+ result = performBatchAllocateAction(taskAction);
+ } else {
+ result = taskAction.perform(task, toolbox);
+ }
emitTimerMetric("task/action/run/time", taskAction, System.currentTimeMillis() - performStartTime);
return result;
}
+ private <R> boolean isBatchAllocateAction(TaskAction<R> taskAction)
+ {
+ return toolbox.canBatchSegmentAllocation()
+ && taskAction instanceof SegmentAllocateAction;
+ }
+
+ @SuppressWarnings("unchecked")
+ private <R> R performBatchAllocateAction(TaskAction<R> taskAction)
+ {
+ try {
+ SegmentAllocateAction allocateAction = (SegmentAllocateAction) taskAction;
+ return (R) allocateAction.performAsync(task, toolbox).get();
Review Comment:
Added a timeout of 5 minutes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027584447
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
Review Comment:
```suggestion
log.info("Initializing segment allocation queue.");
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/LocalTaskActionClient.java:
##########
@@ -76,11 +76,34 @@ public <RetType> RetType submit(TaskAction<RetType> taskAction)
}
final long performStartTime = System.currentTimeMillis();
- final RetType result = taskAction.perform(task, toolbox);
+ final RetType result;
+ if (isBatchAllocateAction(taskAction)) {
+ result = performBatchAllocateAction(taskAction);
+ } else {
+ result = taskAction.perform(task, toolbox);
+ }
emitTimerMetric("task/action/run/time", taskAction, System.currentTimeMillis() - performStartTime);
return result;
}
+ private <R> boolean isBatchAllocateAction(TaskAction<R> taskAction)
+ {
+ return toolbox.canBatchSegmentAllocation()
+ && taskAction instanceof SegmentAllocateAction;
+ }
+
+ @SuppressWarnings("unchecked")
+ private <R> R performBatchAllocateAction(TaskAction<R> taskAction)
+ {
+ try {
+ SegmentAllocateAction allocateAction = (SegmentAllocateAction) taskAction;
+ return (R) allocateAction.performAsync(task, toolbox).get();
Review Comment:
The `get` call should have a timeout. Even a 10 minute timeout is fine but it shouldn't block forever.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
Review Comment:
how much logging does it result in?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
Review Comment:
can you add few lines about what future objects in map are for?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
Review Comment:
this should be false, right? so that we can requeue it.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
Review Comment:
index can be negative too.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
Review Comment:
invoking future calls can also result in an exception. Ideally, the 4 lines below should be done first. Or you can wrap the `future.complete` in a try/catch block.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
Review Comment:
you could have also added a no-op implementation that throws an exception if used.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/LocalTaskActionClient.java:
##########
@@ -76,11 +76,34 @@ public <RetType> RetType submit(TaskAction<RetType> taskAction)
}
final long performStartTime = System.currentTimeMillis();
- final RetType result = taskAction.perform(task, toolbox);
+ final RetType result;
+ if (isBatchAllocateAction(taskAction)) {
+ result = performBatchAllocateAction(taskAction);
+ } else {
+ result = taskAction.perform(task, toolbox);
+ }
emitTimerMetric("task/action/run/time", taskAction, System.currentTimeMillis() - performStartTime);
return result;
}
+ private <R> boolean isBatchAllocateAction(TaskAction<R> taskAction)
+ {
+ return toolbox.canBatchSegmentAllocation()
+ && taskAction instanceof SegmentAllocateAction;
Review Comment:
could we instead add a method in TaskAction class itself?
`shouldPerformAsync` and then call `performAsync` instead?
The method can take toolbox as a parameter and thus the impl in `SegmentAllocationAction` can also check `toolbox.canBatchSegmentAllocation()`
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
Review Comment:
`transferFrom` seems a better name.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
Review Comment:
```suggestion
log.info("Elected leader but batched segment allocation is disabled. segment allocation queue will not be used.");
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
Review Comment:
```suggestion
log.info("Tearing down queue.");
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
Review Comment:
what if `offer` returns `false`?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
Review Comment:
should have different logging depending on whether the batch allocation is enabled or not.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
Review Comment:
shouldn't this be future.completeExceptionally?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
Review Comment:
what if the head of the queue is not due but others in queue are?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
Review Comment:
shouldn't this process take care of the backlog instead of just clearing it?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
+ requestToFuture.keySet().forEach(
+ request -> emitTaskMetric("task/action/failed/count", 1L, request)
+ );
+ requestToFuture.clear();
+ }
+ }
+
+ synchronized void handleResult(SegmentAllocateResult result, SegmentAllocateRequest request)
+ {
+ request.incrementAttempts();
+
+ if (result.isSuccess()) {
+ emitTaskMetric("task/action/success/count", 1L, request);
+ requestToFuture.remove(request).complete(result.getSegmentId());
+ return;
+ }
+
+ log.info("Failed to allocate segment for action [%s]: %s", request.getAction(), result.getErrorMessage());
+ if (request.canRetry()) {
+ log.debug(
+ "Can requeue action [%s] after [%d] failed attempts.",
+ request.getAction(),
+ request.getAttempts()
+ );
+ } else {
+ emitTaskMetric("task/action/failed/count", 1L, request);
+ log.error(
+ "Removing allocation action [%s] from batch after [%d] failed attempts.",
+ request.getAction(),
+ request.getAttempts()
+ );
+ requestToFuture.remove(request).complete(null);
+ }
+ }
+
+ synchronized boolean isEmpty()
+ {
+ return requestToFuture.isEmpty();
+ }
+
+ synchronized int size()
+ {
+ return requestToFuture.size();
+ }
+
+ void resetQueueTime()
+ {
+ queueTimeMillis = System.currentTimeMillis();
+ }
+
+ long getQueueTime()
+ {
+ return queueTimeMillis;
+ }
+
+ boolean isDue()
+ {
+ return System.currentTimeMillis() - queueTimeMillis > maxWaitTimeMillis;
+ }
+ }
+
+ /**
+ * Key to identify a batch of allocation requests.
+ */
+ private static class AllocateRequestKey
+ {
+ private final boolean unique;
+
+ private final String dataSource;
+ private final String groupId;
+ private final Interval preferredAllocationInterval;
+ private final Granularity preferredSegmentGranularity;
+
+ private final boolean skipSegmentLineageCheck;
+ private final LockGranularity lockGranularity;
+
+ private final boolean useNonRootGenPartitionSpace;
+
+ private final int hash;
+ private final String serialized;
Review Comment:
does this string need to be stored? Have you thought about possible GC impact of storing these requests as batches? what will happen if segment allocation is slow for some reason? would Overlord continue to take in requests?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
+ requestToFuture.keySet().forEach(
+ request -> emitTaskMetric("task/action/failed/count", 1L, request)
+ );
+ requestToFuture.clear();
+ }
+ }
+
+ synchronized void handleResult(SegmentAllocateResult result, SegmentAllocateRequest request)
+ {
+ request.incrementAttempts();
+
+ if (result.isSuccess()) {
+ emitTaskMetric("task/action/success/count", 1L, request);
+ requestToFuture.remove(request).complete(result.getSegmentId());
+ return;
+ }
+
+ log.info("Failed to allocate segment for action [%s]: %s", request.getAction(), result.getErrorMessage());
Review Comment:
```suggestion
log.warn("Failed to allocate segment for action [%s]: %s", request.getAction(), result.getErrorMessage());
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
Review Comment:
result checking?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
Review Comment:
what if another batch gets added to processingQueue in this time after line 197?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
Review Comment:
where is index returned?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
+ requestToFuture.keySet().forEach(
+ request -> emitTaskMetric("task/action/failed/count", 1L, request)
+ );
+ requestToFuture.clear();
+ }
+ }
+
+ synchronized void handleResult(SegmentAllocateResult result, SegmentAllocateRequest request)
+ {
+ request.incrementAttempts();
+
+ if (result.isSuccess()) {
+ emitTaskMetric("task/action/success/count", 1L, request);
+ requestToFuture.remove(request).complete(result.getSegmentId());
+ return;
+ }
+
+ log.info("Failed to allocate segment for action [%s]: %s", request.getAction(), result.getErrorMessage());
Review Comment:
can this action be anything else other than SegmentAllocationAction?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] AmatyaAvadhanula commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1028077554
##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -777,6 +994,202 @@ private void insertPendingSegmentIntoMetastore(
.execute();
}
+ private Map<SegmentCreateRequest, SegmentIdWithShardSpec> createNewSegments(
+ Handle handle,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ List<SegmentCreateRequest> requests
+ ) throws IOException
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ // Get the time chunk and associated data segments for the given interval, if any
+ final List<TimelineObjectHolder<String, DataSegment>> existingChunks =
+ getTimelineForIntervalsWithHandle(handle, dataSource, Collections.singletonList(interval))
+ .lookup(interval);
+
+ if (existingChunks.size() > 1) {
+ // Not possible to expand more than one chunk with a single segment.
+ log.warn(
+ "Cannot allocate new segments for dataSource[%s], interval[%s]: already have [%,d] chunks.",
+ dataSource,
+ interval,
+ existingChunks.size()
+ );
+ return Collections.emptyMap();
+ }
+
+ // Shard spec of any of the requests (as they are all compatible) can be used to
+ // identify existing shard specs that share partition space with the requested ones.
+ final PartialShardSpec partialShardSpec = requests.get(0).getPartialShardSpec();
+
+ // max partitionId of published data segments which share the same partition space.
+ SegmentIdWithShardSpec committedMaxId = null;
+
+ @Nullable
+ final String versionOfExistingChunk;
+ if (existingChunks.isEmpty()) {
+ versionOfExistingChunk = null;
+ } else {
+ TimelineObjectHolder<String, DataSegment> existingHolder = Iterables.getOnlyElement(existingChunks);
+ versionOfExistingChunk = existingHolder.getVersion();
+
+ // Don't use the stream API for performance.
+ for (DataSegment segment : FluentIterable
+ .from(existingHolder.getObject())
+ .transform(PartitionChunk::getObject)
+ // Here we check only the segments of the shardSpec which shares the same partition space with the given
+ // partialShardSpec. Note that OverwriteShardSpec doesn't share the partition space with others.
+ // See PartitionIds.
+ .filter(segment -> segment.getShardSpec().sharePartitionSpace(partialShardSpec))) {
+ if (committedMaxId == null
+ || committedMaxId.getShardSpec().getPartitionNum() < segment.getShardSpec().getPartitionNum()) {
+ committedMaxId = SegmentIdWithShardSpec.fromDataSegment(segment);
+ }
+ }
+ }
+
+
+ // Fetch the pending segments for this interval to determine max partitionId
+ // across all shard specs (published + pending).
+ // A pending segment having a higher partitionId must also be considered
+ // to avoid clashes when inserting the pending segment created here.
+ final Set<SegmentIdWithShardSpec> pendingSegments =
+ getPendingSegmentsForIntervalWithHandle(handle, dataSource, interval);
+ if (committedMaxId != null) {
+ pendingSegments.add(committedMaxId);
+ }
+
+ final Map<SegmentCreateRequest, SegmentIdWithShardSpec> createdSegments = new HashMap<>();
+ final Map<String, SegmentIdWithShardSpec> sequenceHashToSegment = new HashMap<>();
+
+ for (SegmentCreateRequest request : requests) {
+ // Check if the required segment has already been created in this batch
+ final String sequenceHash = getSequenceNameAndPrevIdSha(request, interval, skipSegmentLineageCheck);
+
+ final SegmentIdWithShardSpec createdSegment;
+ if (sequenceHashToSegment.containsKey(sequenceHash)) {
+ createdSegment = sequenceHashToSegment.get(sequenceHash);
+ } else {
+ createdSegment = createNewSegment(
+ request,
+ dataSource,
+ interval,
+ versionOfExistingChunk,
+ committedMaxId,
+ pendingSegments
+ );
+
+ // Add to pendingSegments to consider for partitionId
+ if (createdSegment != null) {
+ pendingSegments.add(createdSegment);
+ sequenceHashToSegment.put(sequenceHash, createdSegment);
+ log.info("Created new segment [%s]", createdSegment);
+ }
+ }
+
+ if (createdSegment != null) {
+ createdSegments.put(request, createdSegment);
+ }
+ }
+
+ log.info("Created [%d] new segments for [%d] allocate requests.", sequenceHashToSegment.size(), requests.size());
+ return createdSegments;
+ }
+
+ private SegmentIdWithShardSpec createNewSegment(
Review Comment:
The idea behind this suggestion was that the requests being batched to createXNewSegments would all have the same version and be for the same interval, datasource as well.
It would then simply allocate X new segments with shard specs ranging from overallMaxId + 1 ... overallMaxId + X with the rest of the logic remaining the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1334773996
This pull request **introduces 2 alerts** when merging c13274890d730086916aeb02a509dd8175a08a94 into 138a6de5074e787788d682eb5522775558c9e6a7 - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-cf75a9db0f600387bac06fd5b56e3040c7e79bdc)
**new alerts:**
* 2 for User\-controlled data in numeric cast
---
**Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1035529643
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,694 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 2000;
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final AtomicBoolean isLeader = new AtomicBoolean(false);
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ ServiceEmitter emitter,
+ ScheduledExecutorFactory executorFactory
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = executorFactory.create(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Initializing segment allocation queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Gets the number of batches currently in the queue.
+ */
+ public int size()
+ {
+ return processingQueue.size();
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!isLeader.get()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If the queue is full,
+ * marks the batch as completed failing all the pending requests.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ log.warn("Cannot add batch [%s] as queue is full. Failing [%d] requests.", batch.key, batch.size());
+ batch.markCompleted();
+ return false;
+ }
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If a batch already
+ * exists for this key, transfers all the requests from this batch to the
+ * existing one.
+ */
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ return addBatchToQueue(batch) ? batch : null;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.transferRequestsFrom(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!isLeader.get()) {
+ log.info("Not leader anymore. Failing [%d] batches in queue.", processingQueue.size());
+
+ // Keep removing items from the queue as long as not leader
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && !isLeader.get()) {
+ processingQueue.pollFirst();
+ keyToBatch.remove(nextBatch.key);
+ nextBatch.markCompleted();
+ nextBatch = processingQueue.peekFirst();
+ }
+ }
+
+ // Check once again for leadership
+ if (!isLeader.get()) {
+ return;
+ }
Review Comment:
I guess that is not likely to happen. New items are rejected if we are not the leader.
But I guess it could happen if the leader is flapping?
- t1: we are not leader, we clear the queue
- t2: become leader
- t3: add a request
- t4: stop being leader
- t1: we are not leader, return
the request added by t3 will thus remain stuck in the queue, until we become leader again and process it. The future will timeout after 5 minutes.
I guess a simpler alternative to all of this can be to have the leadership check done for each batch separately in `processBatch`. If not leader, fail it there. The only difference would be that even non-leaders would keep polling the queue. I wanted to avoid that somehow. Please let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1035529643
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,694 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 2000;
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final AtomicBoolean isLeader = new AtomicBoolean(false);
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ ServiceEmitter emitter,
+ ScheduledExecutorFactory executorFactory
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = executorFactory.create(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Initializing segment allocation queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Gets the number of batches currently in the queue.
+ */
+ public int size()
+ {
+ return processingQueue.size();
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!isLeader.get()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If the queue is full,
+ * marks the batch as completed failing all the pending requests.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ log.warn("Cannot add batch [%s] as queue is full. Failing [%d] requests.", batch.key, batch.size());
+ batch.markCompleted();
+ return false;
+ }
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If a batch already
+ * exists for this key, transfers all the requests from this batch to the
+ * existing one.
+ */
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ return addBatchToQueue(batch) ? batch : null;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.transferRequestsFrom(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!isLeader.get()) {
+ log.info("Not leader anymore. Failing [%d] batches in queue.", processingQueue.size());
+
+ // Keep removing items from the queue as long as not leader
+ AllocateRequestBatch nextBatch = processingQueue.peekFirst();
+ while (nextBatch != null && !isLeader.get()) {
+ processingQueue.pollFirst();
+ keyToBatch.remove(nextBatch.key);
+ nextBatch.markCompleted();
+ nextBatch = processingQueue.peekFirst();
+ }
+ }
+
+ // Check once again for leadership
+ if (!isLeader.get()) {
+ return;
+ }
Review Comment:
I guess that is not likely to happen. New items are rejected if we are not the leader.
But maybe it could happen if the leader is flapping?
- t1: we are not leader, we clear the queue
- t2: become leader
- t3: add a request
- t4: stop being leader
- t1: we are not leader, return
the request added by t3 will thus remain stuck in the queue, until we become leader again and process it. The future will timeout after 5 minutes.
A simpler alternative to all of this can be to have the leadership check done for each batch separately in `processBatch` (this check should probably exist anyway). If not leader, fail it there.
The differences would be:
- even non-leaders would keep polling the queue
- requests would wait until their due time to be failed
Please let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1036742760
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -446,6 +449,140 @@ public LockResult tryLock(final Task task, final LockRequest request)
}
}
+ /**
+ * Attempts to allocate segments for the given requests. Each request contains
+ * a {@link Task} and a {@link SegmentAllocateAction}. This method tries to
+ * acquire the task locks on the required intervals/segments and then performs
+ * a batch allocation of segments. It is possible that some requests succeed
+ * successfully and others failed. In that case, only the failed ones should be
+ * retried.
+ *
+ * @param requests List of allocation requests
+ * @param dataSource Datasource for which segment is to be allocated.
+ * @param interval Interval for which segment is to be allocated.
+ * @param skipSegmentLineageCheck Whether lineage check is to be skipped
+ * (this is true for streaming ingestion)
+ * @param lockGranularity Granularity of task lock
+ * @return List of allocation results in the same order as the requests.
+ */
+ public List<SegmentAllocateResult> allocateSegments(
+ List<SegmentAllocateRequest> requests,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ LockGranularity lockGranularity
+ )
+ {
+ log.info("Allocating [%d] segments for datasource [%s], interval [%s]", requests.size(), dataSource, interval);
+ final boolean isTimeChunkLock = lockGranularity == LockGranularity.TIME_CHUNK;
+
+ final AllocationHolderList holderList = new AllocationHolderList(requests, interval);
+ holderList.getPending().forEach(this::verifyTaskIsActive);
+
+ giant.lock();
+ try {
+ if (isTimeChunkLock) {
+ holderList.getPending().forEach(holder -> acquireTaskLock(holder, true));
+ allocateSegmentIds(dataSource, interval, skipSegmentLineageCheck, holderList.getPending());
+ } else {
+ allocateSegmentIds(dataSource, interval, skipSegmentLineageCheck, holderList.getPending());
+ holderList.getPending().forEach(holder -> acquireTaskLock(holder, false));
+ }
+
+ // TODO: for failed allocations, cleanup newly created locks from the posse map
Review Comment:
I need to verify if this is an actual issue. Also, this behaviour is the same in the old flow too.
I think I should tackle this in an immediate follow up PR, so that this PR just adds the new flow without modifying any behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1036742760
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskLockbox.java:
##########
@@ -446,6 +449,140 @@ public LockResult tryLock(final Task task, final LockRequest request)
}
}
+ /**
+ * Attempts to allocate segments for the given requests. Each request contains
+ * a {@link Task} and a {@link SegmentAllocateAction}. This method tries to
+ * acquire the task locks on the required intervals/segments and then performs
+ * a batch allocation of segments. It is possible that some requests succeed
+ * successfully and others failed. In that case, only the failed ones should be
+ * retried.
+ *
+ * @param requests List of allocation requests
+ * @param dataSource Datasource for which segment is to be allocated.
+ * @param interval Interval for which segment is to be allocated.
+ * @param skipSegmentLineageCheck Whether lineage check is to be skipped
+ * (this is true for streaming ingestion)
+ * @param lockGranularity Granularity of task lock
+ * @return List of allocation results in the same order as the requests.
+ */
+ public List<SegmentAllocateResult> allocateSegments(
+ List<SegmentAllocateRequest> requests,
+ String dataSource,
+ Interval interval,
+ boolean skipSegmentLineageCheck,
+ LockGranularity lockGranularity
+ )
+ {
+ log.info("Allocating [%d] segments for datasource [%s], interval [%s]", requests.size(), dataSource, interval);
+ final boolean isTimeChunkLock = lockGranularity == LockGranularity.TIME_CHUNK;
+
+ final AllocationHolderList holderList = new AllocationHolderList(requests, interval);
+ holderList.getPending().forEach(this::verifyTaskIsActive);
+
+ giant.lock();
+ try {
+ if (isTimeChunkLock) {
+ holderList.getPending().forEach(holder -> acquireTaskLock(holder, true));
+ allocateSegmentIds(dataSource, interval, skipSegmentLineageCheck, holderList.getPending());
+ } else {
+ allocateSegmentIds(dataSource, interval, skipSegmentLineageCheck, holderList.getPending());
+ holderList.getPending().forEach(holder -> acquireTaskLock(holder, false));
+ }
+
+ // TODO: for failed allocations, cleanup newly created locks from the posse map
Review Comment:
I need to verify if this is an actual issue. Also, this behaviour is the same in the old flow too.
I think I should tackle this in a follow up PR, so that this PR just adds the new flow without modifying any behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1035524934
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,694 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.BlockingDeque;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private static final int MAX_QUEUE_SIZE = 2000;
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final AtomicBoolean isLeader = new AtomicBoolean(false);
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final BlockingDeque<AllocateRequestBatch> processingQueue = new LinkedBlockingDeque<>(MAX_QUEUE_SIZE);
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ ServiceEmitter emitter,
+ ScheduledExecutorFactory executorFactory
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = executorFactory.create(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Initializing segment allocation queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Tearing down segment allocation queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Gets the number of batches currently in the queue.
+ */
+ public int size()
+ {
+ return processingQueue.size();
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!isLeader.get()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ AllocateRequestBatch newBatch = new AllocateRequestBatch(key);
+ requestFuture.set(newBatch.add(request));
+ return addBatchToQueue(newBatch) ? newBatch : null;
+ } else {
+ requestFuture.set(existingBatch.add(request));
+ return existingBatch;
+ }
+ });
+
+ return requestFuture.get();
+ }
+
+ /**
+ * Tries to add the given batch to the processing queue. If the queue is full,
+ * marks the batch as completed failing all the pending requests.
+ */
+ private boolean addBatchToQueue(AllocateRequestBatch batch)
+ {
+ batch.resetQueueTime();
+ if (processingQueue.offer(batch)) {
+ log.debug("Added a new batch [%s] to queue.", batch.key);
+ return true;
+ } else {
+ log.warn("Cannot add batch [%s] as queue is full. Failing [%d] requests.", batch.key, batch.size());
Review Comment:
It can also happen when things keep getting retried, due to frequent updates to the set of used segments. It would be up to the client to retry in such cases.
The action can be to look at the emitted metrics `batch/runTime` and `batch/attempts`. A greater value of the first would be caused by metadata slowness, the second is retries. The user can then determine if the metadata store needs resizing. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #13369:
URL: https://github.com/apache/druid/pull/13369#issuecomment-1337170838
This pull request **introduces 2 alerts** when merging 88c4cca01effcbb2a0bdb1e23d77bd18c57b8301 into 91774196285ea51b823883394ef1bfe8f2417892 - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-d53d5ef09e91d5fc905ec490285754424ee9d3fd)
**new alerts:**
* 2 for User\-controlled data in numeric cast
---
**Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December**, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out [our post on the GitHub blog](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027604544
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/LocalTaskActionClient.java:
##########
@@ -76,11 +76,34 @@ public <RetType> RetType submit(TaskAction<RetType> taskAction)
}
final long performStartTime = System.currentTimeMillis();
- final RetType result = taskAction.perform(task, toolbox);
+ final RetType result;
+ if (isBatchAllocateAction(taskAction)) {
+ result = performBatchAllocateAction(taskAction);
+ } else {
+ result = taskAction.perform(task, toolbox);
+ }
emitTimerMetric("task/action/run/time", taskAction, System.currentTimeMillis() - performStartTime);
return result;
}
+ private <R> boolean isBatchAllocateAction(TaskAction<R> taskAction)
+ {
+ return toolbox.canBatchSegmentAllocation()
+ && taskAction instanceof SegmentAllocateAction;
Review Comment:
Yeah, that is what I wanted to do originally. Will update this.
Ideally, we shouldn't even be calling `future.get()` here, rather we should do an `HttpRequest.startAsync` in the `OverlordResource` itself.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
Review Comment:
> shouldn't this be future.completeExceptionally?
No, this is in order to retain the existing behaviour.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
Review Comment:
Thanks, will handle this. If the batch couldn't be added to the queue (due to the queue being full), I suppose we don't have a choice but to just complete the request with a `null` return value.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
Review Comment:
Thanks, will update this.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
Review Comment:
I am not too sure about this one. An exception being caught here means that there is probably some bug in the code. Retrying wouldn't help. We could maybe raise an alert here? Let me know what you think.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
Review Comment:
`index` is returned from the binary search. I will rephrase the comment.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
Review Comment:
Shouldn't be a lot. If there are too many tasks running, chances are they would end up sharing a batch.
This log information could be useful when not looking at metrics and trying to verify if batches are getting added to the queue. But I guess it can be a debug log line.
Maybe I should just add a log line in the processing method which prints the current queue size.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
+ requestToFuture.keySet().forEach(
+ request -> emitTaskMetric("task/action/failed/count", 1L, request)
+ );
+ requestToFuture.clear();
+ }
+ }
+
+ synchronized void handleResult(SegmentAllocateResult result, SegmentAllocateRequest request)
+ {
+ request.incrementAttempts();
+
+ if (result.isSuccess()) {
+ emitTaskMetric("task/action/success/count", 1L, request);
+ requestToFuture.remove(request).complete(result.getSegmentId());
+ return;
+ }
+
+ log.info("Failed to allocate segment for action [%s]: %s", request.getAction(), result.getErrorMessage());
+ if (request.canRetry()) {
+ log.debug(
+ "Can requeue action [%s] after [%d] failed attempts.",
+ request.getAction(),
+ request.getAttempts()
+ );
+ } else {
+ emitTaskMetric("task/action/failed/count", 1L, request);
+ log.error(
+ "Removing allocation action [%s] from batch after [%d] failed attempts.",
+ request.getAction(),
+ request.getAttempts()
+ );
+ requestToFuture.remove(request).complete(null);
+ }
+ }
+
+ synchronized boolean isEmpty()
+ {
+ return requestToFuture.isEmpty();
+ }
+
+ synchronized int size()
+ {
+ return requestToFuture.size();
+ }
+
+ void resetQueueTime()
+ {
+ queueTimeMillis = System.currentTimeMillis();
+ }
+
+ long getQueueTime()
+ {
+ return queueTimeMillis;
+ }
+
+ boolean isDue()
+ {
+ return System.currentTimeMillis() - queueTimeMillis > maxWaitTimeMillis;
+ }
+ }
+
+ /**
+ * Key to identify a batch of allocation requests.
+ */
+ private static class AllocateRequestKey
+ {
+ private final boolean unique;
+
+ private final String dataSource;
+ private final String groupId;
+ private final Interval preferredAllocationInterval;
+ private final Granularity preferredSegmentGranularity;
+
+ private final boolean skipSegmentLineageCheck;
+ private final LockGranularity lockGranularity;
+
+ private final boolean useNonRootGenPartitionSpace;
+
+ private final int hash;
+ private final String serialized;
Review Comment:
> Have you thought about possible GC impact of storing these requests as batches?
Thanks for the reminder, I do need to do some basic GC evaluation.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
Review Comment:
> invoking future calls can also result in an exception. Ideally, the 4 lines below should be done first. Or you can wrap the future.complete in a try/catch block.
Oh, I didn't realize. Are you sure? I am just completing the future here and setting a new value in it.
The only buggy scenario I can think is trying to complete an already completed future, but it doesn't throw an exception even then.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
Review Comment:
That cannot happen, the batch due time is nothing but queuedAtTime + maxWaitTime.
So the batch at the head of the queue has to be the one that will be processed first.
When requeueing a batch, it's queue time is reset.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
Review Comment:
Yeah, that makes sense. We can complete the pending futures here with a `null` so that the tasks can retry with the new leader.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
+ requestToFuture.keySet().forEach(
+ request -> emitTaskMetric("task/action/failed/count", 1L, request)
+ );
+ requestToFuture.clear();
+ }
+ }
+
+ synchronized void handleResult(SegmentAllocateResult result, SegmentAllocateRequest request)
+ {
+ request.incrementAttempts();
+
+ if (result.isSuccess()) {
+ emitTaskMetric("task/action/success/count", 1L, request);
+ requestToFuture.remove(request).complete(result.getSegmentId());
+ return;
+ }
+
+ log.info("Failed to allocate segment for action [%s]: %s", request.getAction(), result.getErrorMessage());
Review Comment:
No, this queue is only for batching `SegmentAllocateAction`s only.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
+ requestToFuture.keySet().forEach(
+ request -> emitTaskMetric("task/action/failed/count", 1L, request)
+ );
+ requestToFuture.clear();
+ }
+ }
+
+ synchronized void handleResult(SegmentAllocateResult result, SegmentAllocateRequest request)
+ {
+ request.incrementAttempts();
+
+ if (result.isSuccess()) {
+ emitTaskMetric("task/action/success/count", 1L, request);
+ requestToFuture.remove(request).complete(result.getSegmentId());
+ return;
+ }
+
+ log.info("Failed to allocate segment for action [%s]: %s", request.getAction(), result.getErrorMessage());
+ if (request.canRetry()) {
+ log.debug(
+ "Can requeue action [%s] after [%d] failed attempts.",
+ request.getAction(),
+ request.getAttempts()
+ );
+ } else {
+ emitTaskMetric("task/action/failed/count", 1L, request);
+ log.error(
+ "Removing allocation action [%s] from batch after [%d] failed attempts.",
+ request.getAction(),
+ request.getAttempts()
+ );
+ requestToFuture.remove(request).complete(null);
+ }
+ }
+
+ synchronized boolean isEmpty()
+ {
+ return requestToFuture.isEmpty();
+ }
+
+ synchronized int size()
+ {
+ return requestToFuture.size();
+ }
+
+ void resetQueueTime()
+ {
+ queueTimeMillis = System.currentTimeMillis();
+ }
+
+ long getQueueTime()
+ {
+ return queueTimeMillis;
+ }
+
+ boolean isDue()
+ {
+ return System.currentTimeMillis() - queueTimeMillis > maxWaitTimeMillis;
+ }
+ }
+
+ /**
+ * Key to identify a batch of allocation requests.
+ */
+ private static class AllocateRequestKey
+ {
+ private final boolean unique;
+
+ private final String dataSource;
+ private final String groupId;
+ private final Interval preferredAllocationInterval;
+ private final Granularity preferredSegmentGranularity;
+
+ private final boolean skipSegmentLineageCheck;
+ private final LockGranularity lockGranularity;
+
+ private final boolean useNonRootGenPartitionSpace;
+
+ private final int hash;
+ private final String serialized;
Review Comment:
> what will happen if segment allocation is slow for some reason? would Overlord continue to take in requests?
Currently, yes. We can try to put a limit on the queue size and/or total number of requests in queue. What do you think?
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
+ requestToFuture.keySet().forEach(
+ request -> emitTaskMetric("task/action/failed/count", 1L, request)
+ );
+ requestToFuture.clear();
+ }
+ }
+
+ synchronized void handleResult(SegmentAllocateResult result, SegmentAllocateRequest request)
+ {
+ request.incrementAttempts();
+
+ if (result.isSuccess()) {
+ emitTaskMetric("task/action/success/count", 1L, request);
+ requestToFuture.remove(request).complete(result.getSegmentId());
+ return;
+ }
+
+ log.info("Failed to allocate segment for action [%s]: %s", request.getAction(), result.getErrorMessage());
+ if (request.canRetry()) {
+ log.debug(
+ "Can requeue action [%s] after [%d] failed attempts.",
+ request.getAction(),
+ request.getAttempts()
+ );
+ } else {
+ emitTaskMetric("task/action/failed/count", 1L, request);
+ log.error(
+ "Removing allocation action [%s] from batch after [%d] failed attempts.",
+ request.getAction(),
+ request.getAttempts()
+ );
+ requestToFuture.remove(request).complete(null);
+ }
+ }
+
+ synchronized boolean isEmpty()
+ {
+ return requestToFuture.isEmpty();
+ }
+
+ synchronized int size()
+ {
+ return requestToFuture.size();
+ }
+
+ void resetQueueTime()
+ {
+ queueTimeMillis = System.currentTimeMillis();
+ }
+
+ long getQueueTime()
+ {
+ return queueTimeMillis;
+ }
+
+ boolean isDue()
+ {
+ return System.currentTimeMillis() - queueTimeMillis > maxWaitTimeMillis;
+ }
+ }
+
+ /**
+ * Key to identify a batch of allocation requests.
+ */
+ private static class AllocateRequestKey
+ {
+ private final boolean unique;
+
+ private final String dataSource;
+ private final String groupId;
+ private final Interval preferredAllocationInterval;
+ private final Granularity preferredSegmentGranularity;
+
+ private final boolean skipSegmentLineageCheck;
+ private final LockGranularity lockGranularity;
+
+ private final boolean useNonRootGenPartitionSpace;
+
+ private final int hash;
+ private final String serialized;
Review Comment:
> does this string need to be stored?
No, I think we can get rid of this. Not sure why I had done this in the first place. The computations in a batch are heavy enough to not make this matter at all. 😅
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
Review Comment:
Handled in line 366. I think a test for this method would be helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027617318
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
Review Comment:
> invoking future calls can also result in an exception. Ideally, the 4 lines below should be done first. Or you can wrap the future.complete in a try/catch block.
Oh, I didn't realize. Are you sure? I am just completing the future here and setting a new value in it.
The only buggy scenario I can think of is trying to complete an already completed future, but it doesn't throw an exception even then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027632501
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
+ requestToFuture.keySet().forEach(
+ request -> emitTaskMetric("task/action/failed/count", 1L, request)
+ );
+ requestToFuture.clear();
+ }
+ }
+
+ synchronized void handleResult(SegmentAllocateResult result, SegmentAllocateRequest request)
+ {
+ request.incrementAttempts();
+
+ if (result.isSuccess()) {
+ emitTaskMetric("task/action/success/count", 1L, request);
+ requestToFuture.remove(request).complete(result.getSegmentId());
+ return;
+ }
+
+ log.info("Failed to allocate segment for action [%s]: %s", request.getAction(), result.getErrorMessage());
Review Comment:
I think I will just remove this log line and include the error message in the subsequent logs inside the if-else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #13369: Add SegmentAllocationQueue to batch allocation actions
Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #13369:
URL: https://github.com/apache/druid/pull/13369#discussion_r1027665996
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -0,0 +1,685 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.actions;
+
+import com.google.inject.Inject;
+import org.apache.druid.client.indexing.IndexingService;
+import org.apache.druid.discovery.DruidLeaderSelector;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.indexing.common.LockGranularity;
+import org.apache.druid.indexing.common.task.IndexTaskUtils;
+import org.apache.druid.indexing.common.task.Task;
+import org.apache.druid.indexing.overlord.IndexerMetadataStorageCoordinator;
+import org.apache.druid.indexing.overlord.Segments;
+import org.apache.druid.indexing.overlord.TaskLockbox;
+import org.apache.druid.indexing.overlord.config.TaskLockConfig;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutors;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.guava.Comparators;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.service.ServiceEmitter;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+import org.apache.druid.query.DruidMetrics;
+import org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.Interval;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Queue for {@link SegmentAllocateRequest}s.
+ */
+@ManageLifecycle
+public class SegmentAllocationQueue
+{
+ private static final Logger log = new Logger(SegmentAllocationQueue.class);
+
+ private final long maxWaitTimeMillis;
+ private final boolean enabled;
+
+ private final TaskLockbox taskLockbox;
+ private final ScheduledExecutorService executor;
+ private final IndexerMetadataStorageCoordinator metadataStorage;
+ private final DruidLeaderSelector leaderSelector;
+ private final ServiceEmitter emitter;
+
+ private final ConcurrentHashMap<AllocateRequestKey, AllocateRequestBatch> keyToBatch = new ConcurrentHashMap<>();
+ private final Deque<AllocateRequestBatch> processingQueue = new ConcurrentLinkedDeque<>();
+
+ @Inject
+ public SegmentAllocationQueue(
+ TaskLockbox taskLockbox,
+ TaskLockConfig taskLockConfig,
+ IndexerMetadataStorageCoordinator metadataStorage,
+ @IndexingService DruidLeaderSelector leaderSelector,
+ ServiceEmitter emitter
+ )
+ {
+ this.emitter = emitter;
+ this.taskLockbox = taskLockbox;
+ this.metadataStorage = metadataStorage;
+ this.leaderSelector = leaderSelector;
+ this.maxWaitTimeMillis = taskLockConfig.getBatchAllocationMaxWaitTime();
+ this.enabled = taskLockConfig.isBatchSegmentAllocation();
+
+ this.executor = ScheduledExecutors.fixed(1, "SegmentAllocQueue-%s");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+ log.info("Starting queue.");
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+ log.info("Stopping queue.");
+ executor.shutdownNow();
+ }
+
+ public boolean isEnabled()
+ {
+ return enabled;
+ }
+
+ private void scheduleQueuePoll(long delay)
+ {
+ executor.schedule(this::processBatchesDue, delay, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Queues a SegmentAllocateRequest. The returned future may complete successfully
+ * with a non-null value or with a non-null value.
+ */
+ public Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ if (!leaderSelector.isLeader()) {
+ throw new ISE("Cannot allocate segment if not leader.");
+ } else if (!isEnabled()) {
+ throw new ISE("Batched segment allocation is disabled.");
+ }
+
+ final AllocateRequestKey requestKey = new AllocateRequestKey(request, false);
+ final AtomicReference<Future<SegmentIdWithShardSpec>> requestFuture = new AtomicReference<>();
+
+ keyToBatch.compute(requestKey, (key, existingBatch) -> {
+ AllocateRequestBatch computedBatch = existingBatch;
+ if (computedBatch == null) {
+ computedBatch = new AllocateRequestBatch(key);
+ computedBatch.resetQueueTime();
+ processingQueue.offer(computedBatch);
+ }
+
+ // Possible race condition:
+ // t1 -> new batch is added to queue or batch already exists in queue
+ // t2 -> executor pops batch, processes all requests in it
+ // t1 -> new request is added to dangling batch and is never picked up
+ // Solution: For existing batch, call keyToBatch.remove() on the key to
+ // wait on keyToBatch.compute() to finish before proceeding with processBatch().
+ // For new batch, keyToBatch.remove() would not wait as key is not in map yet
+ // but a new batch is unlikely to be due immediately, so it won't get popped right away.
+ requestFuture.set(computedBatch.add(request));
+ return computedBatch;
+ });
+
+ return requestFuture.get();
+ }
+
+ private void requeueBatch(AllocateRequestBatch batch)
+ {
+ log.info("Requeueing [%d] failed requests in batch [%s].", batch.size(), batch.key);
+ keyToBatch.compute(batch.key, (key, existingBatch) -> {
+ if (existingBatch == null) {
+ batch.resetQueueTime();
+ processingQueue.offer(batch);
+ return batch;
+ } else {
+ // Merge requests from this batch to existing one
+ existingBatch.merge(batch);
+ return existingBatch;
+ }
+ });
+ }
+
+ private void processBatchesDue()
+ {
+ // If not leader, clear the queue and do not schedule any more rounds of processing
+ if (!leaderSelector.isLeader()) {
+ log.info("Not leader anymore. Clearing [%d] batches from queue.", processingQueue.size());
+ processingQueue.clear();
+ keyToBatch.clear();
+ return;
+ }
+
+ // Process all batches which are due
+ log.debug("Processing all batches which are due for execution.");
+ int numProcessedBatches = 0;
+
+ AllocateRequestBatch nextBatch = processingQueue.peek();
+ while (nextBatch != null && nextBatch.isDue()) {
+ processingQueue.poll();
+ boolean processed;
+ try {
+ processed = processBatch(nextBatch);
+ ++numProcessedBatches;
+ }
+ catch (Throwable t) {
+ processed = true;
+ log.error(t, "Error while processing batch [%s]", nextBatch.key);
+ }
+
+ if (processed) {
+ nextBatch.markCompleted();
+ } else {
+ requeueBatch(nextBatch);
+ }
+
+ nextBatch = processingQueue.peek();
+ }
+
+ // Schedule the next round of processing
+ final long nextScheduleDelay;
+ if (processingQueue.isEmpty()) {
+ nextScheduleDelay = maxWaitTimeMillis;
+ } else {
+ nextBatch = processingQueue.peek();
+ long timeElapsed = System.currentTimeMillis() - nextBatch.getQueueTime();
+ nextScheduleDelay = Math.max(0, maxWaitTimeMillis - timeElapsed);
+ }
+ scheduleQueuePoll(nextScheduleDelay);
+ log.debug("Processed [%d] batches, next execution in [%d ms]", numProcessedBatches, nextScheduleDelay);
+ }
+
+ /**
+ * Processes the given batch. Returns true if the batch was completely processed
+ * and should not be requeued.
+ */
+ private boolean processBatch(AllocateRequestBatch requestBatch)
+ {
+ final AllocateRequestKey requestKey = requestBatch.key;
+ keyToBatch.remove(requestKey);
+ if (requestBatch.isEmpty()) {
+ return true;
+ }
+
+ log.info(
+ "Processing [%d] requests for batch [%s], queue time [%s].",
+ requestBatch.size(),
+ requestKey,
+ requestBatch.getQueueTime()
+ );
+
+ final long startTimeMillis = System.currentTimeMillis();
+ final int batchSize = requestBatch.size();
+ emitBatchMetric("task/action/batch/size", batchSize, requestKey);
+ emitBatchMetric("task/action/batch/queueTime", (startTimeMillis - requestBatch.getQueueTime()), requestKey);
+
+ final Set<DataSegment> usedSegments = retrieveUsedSegments(requestKey);
+ final int successCount = allocateSegmentsForBatch(requestBatch, usedSegments);
+
+ emitBatchMetric("task/action/batch/retries", 1L, requestKey);
+ emitBatchMetric("task/action/batch/runTime", (System.currentTimeMillis() - startTimeMillis), requestKey);
+ log.info("Successfully processed [%d / %d] requests in batch [%s].", successCount, batchSize, requestKey);
+
+ if (requestBatch.isEmpty()) {
+ log.info("All requests in batch [%s] have been processed.", requestKey);
+ return true;
+ }
+
+ // Requeue the batch only if used segments have changed
+ log.info("There are [%d] failed requests in batch [%s].", requestBatch.size(), requestKey);
+ final Set<DataSegment> updatedUsedSegments = retrieveUsedSegments(requestKey);
+
+ if (updatedUsedSegments.equals(usedSegments)) {
+ log.error("Used segments have not changed. Not requeueing failed requests.");
+ return true;
+ } else {
+ log.info("Used segments have changed. Requeuing failed requests");
+ return false;
+ }
+ }
+
+ private Set<DataSegment> retrieveUsedSegments(AllocateRequestKey key)
+ {
+ return new HashSet<>(
+ metadataStorage.retrieveUsedSegmentsForInterval(
+ key.dataSource,
+ key.preferredAllocationInterval,
+ Segments.ONLY_VISIBLE
+ )
+ );
+ }
+
+ private int allocateSegmentsForBatch(AllocateRequestBatch requestBatch, Set<DataSegment> usedSegments)
+ {
+ int successCount = 0;
+
+ final List<SegmentAllocateRequest> allRequests = requestBatch.getRequests();
+ final Set<SegmentAllocateRequest> pendingRequests = new HashSet<>();
+
+ if (usedSegments.isEmpty()) {
+ pendingRequests.addAll(allRequests);
+ } else {
+ final Interval[] sortedUsedSegmentIntervals = getSortedIntervals(usedSegments);
+ final Map<Interval, List<SegmentAllocateRequest>> usedIntervalToRequests = new HashMap<>();
+
+ for (SegmentAllocateRequest request : allRequests) {
+ // If there is an overlapping used segment interval, that interval is
+ // the only candidate for allocation
+ Interval overlappingInterval = findOverlappingInterval(
+ request.getRowInterval(),
+ sortedUsedSegmentIntervals
+ );
+
+ if (overlappingInterval == null) {
+ pendingRequests.add(request);
+ } else if (overlappingInterval.contains(request.getRowInterval())) {
+ // Found an enclosing interval, use this for allocation
+ usedIntervalToRequests.computeIfAbsent(overlappingInterval, i -> new ArrayList<>())
+ .add(request);
+ }
+ }
+
+ // Try to allocate segments for the identified used segment intervals
+ // Do not retry the failed requests with other intervals unless the batch is requeued
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : usedIntervalToRequests.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ }
+ }
+
+ // For requests that do not overlap with a used segment, first try to allocate
+ // using the preferred granularity, then smaller granularities
+ for (Granularity granularity :
+ Granularity.granularitiesFinerThan(requestBatch.key.preferredSegmentGranularity)) {
+ Map<Interval, List<SegmentAllocateRequest>> requestsByInterval =
+ getRequestsByInterval(pendingRequests, granularity);
+
+ for (Map.Entry<Interval, List<SegmentAllocateRequest>> entry : requestsByInterval.entrySet()) {
+ List<SegmentAllocateRequest> successfulRequests = allocateSegmentsForInterval(
+ entry.getKey(),
+ entry.getValue(),
+ requestBatch
+ );
+ successCount += successfulRequests.size();
+ pendingRequests.removeAll(successfulRequests);
+ }
+ }
+
+ return successCount;
+ }
+
+ private Interval findOverlappingInterval(Interval searchInterval, Interval[] sortedIntervals)
+ {
+ int index = Arrays.binarySearch(
+ sortedIntervals,
+ searchInterval,
+ Comparators.intervalsByStartThenEnd()
+ );
+ if (index >= 0) {
+ return sortedIntervals[index];
+ }
+
+ // Key was not found, returned index is (-(insertionPoint) - 1)
+ index = -(index + 1);
+
+ // If the interval at index doesn't overlap, (index + 1) wouldn't overlap either
+ if (index < sortedIntervals.length) {
+ if (sortedIntervals[index].overlaps(searchInterval)) {
+ return sortedIntervals[index];
+ }
+ }
+
+ // If the interval at (index - 1) doesn't overlap, (index - 2) wouldn't overlap either
+ if (index > 0) {
+ if (sortedIntervals[index - 1].overlaps(searchInterval)) {
+ return sortedIntervals[index - 1];
+ }
+ }
+
+ return null;
+ }
+
+ private Interval[] getSortedIntervals(Set<DataSegment> usedSegments)
+ {
+ TreeSet<Interval> sortedSet = new TreeSet<>(Comparators.intervalsByStartThenEnd());
+ usedSegments.forEach(segment -> sortedSet.add(segment.getInterval()));
+ return sortedSet.toArray(new Interval[0]);
+ }
+
+ /**
+ * Tries to allocate segments for the given requests over the specified interval.
+ * Returns the list of requests for which segments were successfully allocated.
+ */
+ private List<SegmentAllocateRequest> allocateSegmentsForInterval(
+ Interval tryInterval,
+ List<SegmentAllocateRequest> requests,
+ AllocateRequestBatch requestBatch
+ )
+ {
+ if (requests.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final AllocateRequestKey requestKey = requestBatch.key;
+ log.info(
+ "Trying allocation for [%d] requests, interval [%s] in batch [%s]",
+ requests.size(),
+ tryInterval,
+ requestKey
+ );
+
+ final List<SegmentAllocateResult> results = taskLockbox.allocateSegments(
+ requests,
+ requestKey.dataSource,
+ tryInterval,
+ requestKey.skipSegmentLineageCheck,
+ requestKey.lockGranularity
+ );
+
+ final List<SegmentAllocateRequest> successfulRequests = new ArrayList<>();
+ for (int i = 0; i < requests.size(); ++i) {
+ SegmentAllocateRequest request = requests.get(i);
+ SegmentAllocateResult result = results.get(i);
+ if (result.isSuccess()) {
+ successfulRequests.add(request);
+ }
+
+ requestBatch.handleResult(result, request);
+ }
+
+ return successfulRequests;
+ }
+
+ private Map<Interval, List<SegmentAllocateRequest>> getRequestsByInterval(
+ Set<SegmentAllocateRequest> requests,
+ Granularity tryGranularity
+ )
+ {
+ final Map<Interval, List<SegmentAllocateRequest>> tryIntervalToRequests = new HashMap<>();
+ for (SegmentAllocateRequest request : requests) {
+ Interval tryInterval = tryGranularity.bucket(request.getAction().getTimestamp());
+ if (tryInterval.contains(request.getRowInterval())) {
+ tryIntervalToRequests.computeIfAbsent(tryInterval, i -> new ArrayList<>()).add(request);
+ }
+ }
+ return tryIntervalToRequests;
+ }
+
+ private void emitTaskMetric(String metric, long value, SegmentAllocateRequest request)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ IndexTaskUtils.setTaskDimensions(metricBuilder, request.getTask());
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ private void emitBatchMetric(String metric, long value, AllocateRequestKey key)
+ {
+ final ServiceMetricEvent.Builder metricBuilder = ServiceMetricEvent.builder();
+ metricBuilder.setDimension("taskActionType", SegmentAllocateAction.TYPE);
+ metricBuilder.setDimension(DruidMetrics.DATASOURCE, key.dataSource);
+ metricBuilder.setDimension(DruidMetrics.INTERVAL, key.preferredAllocationInterval.toString());
+ emitter.emit(metricBuilder.build(metric, value));
+ }
+
+ public void becomeLeader()
+ {
+ if (isEnabled()) {
+ // Start polling the queue
+ log.info("Elected leader. Starting queue processing.");
+ scheduleQueuePoll(maxWaitTimeMillis);
+ } else {
+ log.info("Elected leader but batched segment allocation is disabled.");
+ }
+ }
+
+ public void stopBeingLeader()
+ {
+ log.info("Not leader anymore. Stopping queue processing.");
+ }
+
+ /**
+ * A batch of segment allocation requests.
+ */
+ private class AllocateRequestBatch
+ {
+ private long queueTimeMillis;
+ private final AllocateRequestKey key;
+
+ /**
+ * This must be accessed through methods synchronized on this batch.
+ * It is to avoid races between a new request being added just when the batch
+ * is being processed.
+ */
+ private final Map<SegmentAllocateRequest, CompletableFuture<SegmentIdWithShardSpec>>
+ requestToFuture = new HashMap<>();
+
+ AllocateRequestBatch(AllocateRequestKey key)
+ {
+ log.info("Creating a new batch with key: %s", key);
+ this.key = key;
+ }
+
+ synchronized Future<SegmentIdWithShardSpec> add(SegmentAllocateRequest request)
+ {
+ log.info("Adding request to batch [%s]: %s", key, request.getAction());
+ return requestToFuture.computeIfAbsent(request, req -> new CompletableFuture<>());
+ }
+
+ synchronized void merge(AllocateRequestBatch batch)
+ {
+ requestToFuture.putAll(batch.requestToFuture);
+ batch.requestToFuture.clear();
+ }
+
+ synchronized List<SegmentAllocateRequest> getRequests()
+ {
+ return new ArrayList<>(requestToFuture.keySet());
+ }
+
+ synchronized void markCompleted()
+ {
+ if (!requestToFuture.isEmpty()) {
+ log.info("Marking [%d] requests in batch [%s] as failed.", size(), key);
+ requestToFuture.values().forEach(future -> future.complete(null));
Review Comment:
I meant that there can be an exception in a listener attached to this future. e.g.
```
future.whenComplete() {
throw some exception;
}
```
But if the callback is a simple op, its fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org