You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/12/01 11:58:20 UTC
[GitHub] [druid] adarshsanjeev opened a new pull request, #13475: Remove stray reference to fix OOM while merging sketches
adarshsanjeev opened a new pull request, #13475:
URL: https://github.com/apache/druid/pull/13475
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [x] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] rohangarg commented on a diff in pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
rohangarg commented on code in PR #13475:
URL: https://github.com/apache/druid/pull/13475#discussion_r1039377372
##########
extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/exec/WorkerSketchFetcherTest.java:
##########
@@ -113,20 +117,30 @@ public void tearDown() throws Exception
public void test_submitFetcherTask_parallelFetch_workerThrowsException_shouldCancelOtherTasks() throws Exception
{
// Store futures in a queue
- final Queue<ListenableFuture<ClusterByStatisticsSnapshot>> futureQueue = new ConcurrentLinkedQueue<>();
+ final Queue<Future<?>> futureQueue = new ConcurrentLinkedQueue<>();
final List<String> workerIds = ImmutableList.of("0", "1", "2", "3");
final CountDownLatch latch = new CountDownLatch(workerIds.size());
- target = spy(new WorkerSketchFetcher(workerClient, ClusterStatisticsMergeMode.PARALLEL, 300_000_000));
+ target = spy(
+ new WorkerSketchFetcher(
+ workerClient,
+ ClusterStatisticsMergeMode.PARALLEL,
+ 300_000_000,
+ executorService
+ )
+ );
+
+ // When submitting futures from the executor, add it to the list first.
+ doAnswer(invocation -> {
+ Future<?> future = spy((Future<?>) invocation.callRealMethod());
+ futureQueue.add(future);
+ return future;
+ }).when(executorService).submit(any(Runnable.class));
- // When fetching snapshots, return a mock and add future to queue
doAnswer(invocation -> {
- ListenableFuture<ClusterByStatisticsSnapshot> snapshotListenableFuture =
- spy(Futures.immediateFuture(mock(ClusterByStatisticsSnapshot.class)));
- futureQueue.add(snapshotListenableFuture);
latch.countDown();
latch.await();
- return snapshotListenableFuture;
+ return Futures.immediateFuture(mock(ClusterByStatisticsSnapshot.class));
}).when(workerClient).fetchClusterByStatisticsSnapshot(any(), any(), anyInt());
Review Comment:
1. shouldn't it be `eq("1")` in the first argument, since below we're adding a different mock for `eq("2")` ?
2. Can we return a custom future from here which is never done and also captures the `cancel` invocation to increment the cancel counter - that could remove the need for spying executor service and creating new testing constructors for fetcher?
3. Why are we doing `latch.await()` in every thread and using sleep in the main test thread? Didn't understand that part
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -118,22 +140,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>> inMemoryFullSketchMerging(
final int workerCount = workerTaskIds.size();
// Guarded by synchronized mergedStatisticsCollector
final Set<Integer> finishedWorkers = new HashSet<>();
+ final Queue<Future<?>> executorFutures = new ConcurrentLinkedQueue<>();
Review Comment:
is there a reason for making this `ConcurrentLinkedQueue` instead of a simple list?
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -231,6 +260,7 @@ public void submitFetchingTasksForNextTimeChunk()
stageDefinition.createResultKeyStatisticsCollector(statisticsMaxRetainedBytes);
// Guarded by synchronized mergedStatisticsCollector
Set<Integer> finishedWorkers = new HashSet<>();
+ final Queue<Future<?>> executorFutures = new ConcurrentLinkedQueue<>();
Review Comment:
same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] adarshsanjeev commented on a diff in pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
adarshsanjeev commented on code in PR #13475:
URL: https://github.com/apache/druid/pull/13475#discussion_r1039001996
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -155,7 +152,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>> inMemoryFullSketchMerging(
mergedStatisticsCollector.clear();
Review Comment:
Changed
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -155,7 +152,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>> inMemoryFullSketchMerging(
mergedStatisticsCollector.clear();
}
}
- });
+ }));
+ });
+
+ partitionFuture.whenComplete((result, exception) -> {
Review Comment:
Yes, the future that is added as a callback should still get executed if the completable future is already finished executing, though the javadoc doesn't guarantee which thread would be executing it if an executor is also not passed as an argument.
It is also not a hard necessity that we cancel every pending future but it depends on number of workers which could be up to a 1000, which would waste a lot of time, but correctness would still be present.
Also, with regard to retrying, part of this will be updated with the worker retry PR, to only fetch the statistics which are needed even if a worker restarts, so after that we would be retrying the stats collection.
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -118,22 +120,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>> inMemoryFullSketchMerging(
final int workerCount = workerTaskIds.size();
// Guarded by synchronized mergedStatisticsCollector
final Set<Integer> finishedWorkers = new HashSet<>();
+ final Set<Future<?>> futuresToCancel = ConcurrentHashMap.newKeySet();
Review Comment:
Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] rohangarg commented on a diff in pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
rohangarg commented on code in PR #13475:
URL: https://github.com/apache/druid/pull/13475#discussion_r1037165745
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -118,22 +120,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>> inMemoryFullSketchMerging(
final int workerCount = workerTaskIds.size();
// Guarded by synchronized mergedStatisticsCollector
final Set<Integer> finishedWorkers = new HashSet<>();
+ final Set<Future<?>> futuresToCancel = ConcurrentHashMap.newKeySet();
Review Comment:
also this is not really `futuresToCancel` - this is the exhaustive list of futures
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] rohangarg commented on a diff in pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
rohangarg commented on code in PR #13475:
URL: https://github.com/apache/druid/pull/13475#discussion_r1037161873
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -155,7 +152,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>> inMemoryFullSketchMerging(
mergedStatisticsCollector.clear();
}
}
- });
+ }));
+ });
+
+ partitionFuture.whenComplete((result, exception) -> {
Review Comment:
in a corner case, it could happen that `partitionFuture` finishes before all futures are queued up in the executor. To avoid that one simple solution could be to have a `CountDownLatch` with value as `workerCount`. The `whenComplete` would then wait for latch to become 0 before cancelling all the futures.
Maybe we could also have a better solution with cancellation - will update on this thread if I find any utility.
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -118,22 +120,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>> inMemoryFullSketchMerging(
final int workerCount = workerTaskIds.size();
// Guarded by synchronized mergedStatisticsCollector
final Set<Integer> finishedWorkers = new HashSet<>();
+ final Set<Future<?>> futuresToCancel = ConcurrentHashMap.newKeySet();
Review Comment:
this could be a list - not sure if equals is correctly done for futures
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -155,7 +152,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>> inMemoryFullSketchMerging(
mergedStatisticsCollector.clear();
Review Comment:
we can call this catch block only once by checking `partitionFuture.isDone`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] LakshSingla commented on a diff in pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
LakshSingla commented on code in PR #13475:
URL: https://github.com/apache/druid/pull/13475#discussion_r1037778390
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -155,7 +152,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>> inMemoryFullSketchMerging(
mergedStatisticsCollector.clear();
}
}
- });
+ }));
+ });
+
+ partitionFuture.whenComplete((result, exception) -> {
Review Comment:
Is this a failsafe condition or would this be required? From my understanding, the successful completion of partitionFuture depends on the completion of the underlying futures, and in case of unsuccessful completion, we shouldn't have any issue with prematurely cancelling any underlying future. I might have missed something though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] rohangarg commented on a diff in pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
rohangarg commented on code in PR #13475:
URL: https://github.com/apache/druid/pull/13475#discussion_r1038041864
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -155,7 +152,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>> inMemoryFullSketchMerging(
mergedStatisticsCollector.clear();
}
}
- });
+ }));
+ });
+
+ partitionFuture.whenComplete((result, exception) -> {
Review Comment:
Yes, agree that since attaching this callback is in the same thread after queuing - this shouldn't trigger without futures. Although, that raises the question that if the future finishes before this callback is attached, will this callback execute while attaching it? if so, will that callback execution happen on the same thread attaching it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] cryptoe merged pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
cryptoe merged PR #13475:
URL: https://github.com/apache/druid/pull/13475
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] rohangarg commented on a diff in pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
rohangarg commented on code in PR #13475:
URL: https://github.com/apache/druid/pull/13475#discussion_r1038043825
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -155,7 +152,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>> inMemoryFullSketchMerging(
mergedStatisticsCollector.clear();
}
}
- });
+ }));
+ });
+
+ partitionFuture.whenComplete((result, exception) -> {
Review Comment:
Also, is it necessary to cancel the pending tasks? It would be incase we'll be retrying the stats collection stage - otherwise if we're doing to fail the whole job, it is not necessary to fail them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] adarshsanjeev commented on a diff in pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
adarshsanjeev commented on code in PR #13475:
URL: https://github.com/apache/druid/pull/13475#discussion_r1038031651
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -155,7 +152,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>> inMemoryFullSketchMerging(
mergedStatisticsCollector.clear();
}
}
- });
+ }));
+ });
+
+ partitionFuture.whenComplete((result, exception) -> {
Review Comment:
I also have a question in addition to Laksh's point, since we are attaching this callback in the same thread after queueing up the futures, would this trigger without all the futures present anyway?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] rohangarg commented on a diff in pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
rohangarg commented on code in PR #13475:
URL: https://github.com/apache/druid/pull/13475#discussion_r1038043825
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -155,7 +152,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>> inMemoryFullSketchMerging(
mergedStatisticsCollector.clear();
}
}
- });
+ }));
+ });
+
+ partitionFuture.whenComplete((result, exception) -> {
Review Comment:
Also, is it necessary to cancel the pending/running futures? It would be incase we'll be retrying the stats collection stage - otherwise if we're going to fail the whole job, it is not necessary to fail them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] rohangarg commented on a diff in pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
rohangarg commented on code in PR #13475:
URL: https://github.com/apache/druid/pull/13475#discussion_r1039901091
##########
extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/exec/WorkerSketchFetcherTest.java:
##########
@@ -113,20 +117,30 @@ public void tearDown() throws Exception
public void test_submitFetcherTask_parallelFetch_workerThrowsException_shouldCancelOtherTasks() throws Exception
{
// Store futures in a queue
- final Queue<ListenableFuture<ClusterByStatisticsSnapshot>> futureQueue = new ConcurrentLinkedQueue<>();
+ final Queue<Future<?>> futureQueue = new ConcurrentLinkedQueue<>();
final List<String> workerIds = ImmutableList.of("0", "1", "2", "3");
final CountDownLatch latch = new CountDownLatch(workerIds.size());
- target = spy(new WorkerSketchFetcher(workerClient, ClusterStatisticsMergeMode.PARALLEL, 300_000_000));
+ target = spy(
+ new WorkerSketchFetcher(
+ workerClient,
+ ClusterStatisticsMergeMode.PARALLEL,
+ 300_000_000,
+ executorService
+ )
+ );
+
+ // When submitting futures from the executor, add it to the list first.
+ doAnswer(invocation -> {
+ Future<?> future = spy((Future<?>) invocation.callRealMethod());
+ futureQueue.add(future);
+ return future;
+ }).when(executorService).submit(any(Runnable.class));
- // When fetching snapshots, return a mock and add future to queue
doAnswer(invocation -> {
- ListenableFuture<ClusterByStatisticsSnapshot> snapshotListenableFuture =
- spy(Futures.immediateFuture(mock(ClusterByStatisticsSnapshot.class)));
- futureQueue.add(snapshotListenableFuture);
latch.countDown();
latch.await();
- return snapshotListenableFuture;
+ return Futures.immediateFuture(mock(ClusterByStatisticsSnapshot.class));
}).when(workerClient).fetchClusterByStatisticsSnapshot(any(), any(), anyInt());
Review Comment:
Ah, yes I missed that this method is for a part of the computation and not the whole computation itself.
Although the same concept can be still applied and we can make the computation in this mocking long timed and wait for an interruption arisen from the cancel. I tried the following approach locally : https://github.com/apache/druid/commit/c536a3f414c20dec35c2a47ef47e4d1049ed1091
The goal of the above is to minimize mocking/spying from the attributes of the class being UT-ed - but I'm ok with the any of the approaches for now since this change is planned for release 25.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] adarshsanjeev commented on pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
adarshsanjeev commented on PR #13475:
URL: https://github.com/apache/druid/pull/13475#issuecomment-1336477017
There are unit tests in WorkerSketchFetcherTest which test if tasks would be cancelled, but there aren't any tests to see if any references to large objects like key stats are leaked. I suppose we could test something like that with WeakReference?
I have reproduced the bug and functionally tested this out locally, and these changes seems to fix the bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] cryptoe commented on pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
cryptoe commented on PR #13475:
URL: https://github.com/apache/druid/pull/13475#issuecomment-1341856200
UT's and IT's passed for MSQ. As the worker retry PR #13353 is redoing `WorkerSketchFetcher`, I guess its okay to make an exception for less code coverage on workerSketchFetcher.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] rohangarg commented on a diff in pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
rohangarg commented on code in PR #13475:
URL: https://github.com/apache/druid/pull/13475#discussion_r1038043825
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -155,7 +152,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>> inMemoryFullSketchMerging(
mergedStatisticsCollector.clear();
}
}
- });
+ }));
+ });
+
+ partitionFuture.whenComplete((result, exception) -> {
Review Comment:
Also, is it necessary to cancel the pending/running futures? It would be incase we'll be retrying the stats collection stage - otherwise if we're doing to fail the whole job, it is not necessary to fail them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] rohangarg commented on pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
rohangarg commented on PR #13475:
URL: https://github.com/apache/druid/pull/13475#issuecomment-1337097474
> There are unit tests in WorkerSketchFetcherTest which test if tasks would be cancelled, but there aren't any tests to see if any references to large objects like key stats are leaked. I suppose we could test something like that with WeakReference?
I have reproduced the bug and functionally tested this out locally, and these changes seems to fix the bug.
I think doing the functional test is ok for now, maybe in future one way to test it could be to generate multiple small sketches in a test and write them to file. Then we can setup the fetcher for those sketches and check if the fetcher runs stably - it will automatically OOM incase it ends up taking a lot of memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] adarshsanjeev commented on a diff in pull request #13475: Remove stray reference to fix OOM while merging sketches
Posted by GitBox <gi...@apache.org>.
adarshsanjeev commented on code in PR #13475:
URL: https://github.com/apache/druid/pull/13475#discussion_r1039718396
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerSketchFetcher.java:
##########
@@ -118,22 +140,17 @@ CompletableFuture<Either<Long, ClusterByPartitions>> inMemoryFullSketchMerging(
final int workerCount = workerTaskIds.size();
// Guarded by synchronized mergedStatisticsCollector
final Set<Integer> finishedWorkers = new HashSet<>();
+ final Queue<Future<?>> executorFutures = new ConcurrentLinkedQueue<>();
Review Comment:
Discussed offline. Changed to a simple list as this would be enough
##########
extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/exec/WorkerSketchFetcherTest.java:
##########
@@ -113,20 +117,30 @@ public void tearDown() throws Exception
public void test_submitFetcherTask_parallelFetch_workerThrowsException_shouldCancelOtherTasks() throws Exception
{
// Store futures in a queue
- final Queue<ListenableFuture<ClusterByStatisticsSnapshot>> futureQueue = new ConcurrentLinkedQueue<>();
+ final Queue<Future<?>> futureQueue = new ConcurrentLinkedQueue<>();
final List<String> workerIds = ImmutableList.of("0", "1", "2", "3");
final CountDownLatch latch = new CountDownLatch(workerIds.size());
- target = spy(new WorkerSketchFetcher(workerClient, ClusterStatisticsMergeMode.PARALLEL, 300_000_000));
+ target = spy(
+ new WorkerSketchFetcher(
+ workerClient,
+ ClusterStatisticsMergeMode.PARALLEL,
+ 300_000_000,
+ executorService
+ )
+ );
+
+ // When submitting futures from the executor, add it to the list first.
+ doAnswer(invocation -> {
+ Future<?> future = spy((Future<?>) invocation.callRealMethod());
+ futureQueue.add(future);
+ return future;
+ }).when(executorService).submit(any(Runnable.class));
- // When fetching snapshots, return a mock and add future to queue
doAnswer(invocation -> {
- ListenableFuture<ClusterByStatisticsSnapshot> snapshotListenableFuture =
- spy(Futures.immediateFuture(mock(ClusterByStatisticsSnapshot.class)));
- futureQueue.add(snapshotListenableFuture);
latch.countDown();
latch.await();
- return snapshotListenableFuture;
+ return Futures.immediateFuture(mock(ClusterByStatisticsSnapshot.class));
}).when(workerClient).fetchClusterByStatisticsSnapshot(any(), any(), anyInt());
Review Comment:
Changed the answers to be more concise and clear instead of layering mocks.
I didn't quite understand the second point, if the executorService future is cancelled, it does not call cancel on the custom future, since it is unaware of it entirely. It instead throws an InterruptedException if the task has already started (which is find and handled properly in our code), however, this our custom mock would not know that it was interrupted.
The sleep in the main thread was to resolve a rare test failure where the asserts were happening before the futures could be cancelled, causing the test to fail prematurely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org