You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "svalaskevicius (via GitHub)" <gi...@apache.org> on 2026/02/13 10:27:00 UTC
[PR] Allow stopping thread pools manually [iceberg]
svalaskevicius opened a new pull request, #15312:
URL: https://github.com/apache/iceberg/pull/15312
- allows to stop thread pools manually, to avoid leaks in hot-reload environments
- allows to opt-out of the standard shutdown mechanism to manage graceful service stops (and commit the last pending files)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2804806236
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
+ ExecutorService item;
+ Queue<ExecutorService> invoked = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ invoked.add(item);
+ }
+ while ((item = invoked.poll()) != null) {
+ try {
+ item.awaitTermination(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
Review Comment:
ah! I see what you mean. ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4333562905
@rdblue
Hi, thanks for the feedback. I've updated the PR description - I hope it's clearer.
Please can you explain what do you mean regarding the default thread pools? Having them created via MoreExecutors is exactly the problem, or do you mean to simply revert the commit 1bc41ccda ?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4023140056
I have played around with reforactoring the tests, so we don't need to access the manger directly. See: https://github.com/pvary/iceberg/commit/32e9d280f5b8ed33d262693ce85c4198448e5344
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2904832297
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +153,89 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
Review Comment:
```suggestion
/**
* Shuts down all thread pools registered via {@link #newExitingWorkerPool(String, int)} or {@link
* #newExitingScheduledPool(String, int, Duration)} and removes the JVM shutdown hook.
*
* <p>This method is useful for:
*
* <ul>
* <li>Preventing thread pool leaks in hot-reload environments
* <li>Managing graceful application shutdown when the application needs to handle its own
* shutdown hooks (e.g. to commit pending files before exiting)
* </ul>
*
* <p>Only call this method at the end of the intended usage of the library. Calling it earlier
* will stop thread pools required for normal library workflows.
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4330983298
@svalaskevicius can you please update the description with exactly what this is changing and why? I think it is a red flag that this is touching the static worker pool, but doesn't specifically call that out in the PR description and also doesn't explain why that is needed.
If this PR is about how to manage threadpools coming from factory methods in `ThreadPools`, then what this is trying to accomplish should be more clear.
I also am skeptical of this kind of update in general. This class was intended to hold a single threadpool, not to provide threadpools. As it grew over time, people wanted to have threadpools configured the same way. But changing the behavior of the default pools in order to make other uses of the convenience methods more generic is not the right path forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2878160945
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -148,9 +152,99 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* ensure the pool terminates when the JVM exits. This is suitable for long-lived thread pools
* that should be automatically cleaned up on JVM shutdown.
*/
- public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ public static synchronized ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(new ExecutorServiceWithTimeout(service, DEFAULT_SHUTDOWN_TIMEOUT));
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static synchronized void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : THREAD_POOLS_TO_SHUTDOWN) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ THREAD_POOLS_TO_SHUTDOWN.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
Review Comment:
i.e. the error I am trying to solve for our application is that on shutdown, iceberg library starts throwing all sorts of exceptions how all threadpools are already shut down. JVM does not allow to control the order of shutdown hooks, so it's either this change in iceberg or runtime magic within JVM...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4884289972
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2904864959
##########
core/src/test/java/org/apache/iceberg/util/TestThreadPools.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.junit.jupiter.api.Test;
+
+public class TestThreadPools {
+
+ @Test
+ public void testRemoveShutdownHook() {
+ try {
+ assertThat(ThreadPools.isShutdownHookRegistered()).isTrue();
+ ThreadPools.removeShutdownHook();
+ assertThat(ThreadPools.isShutdownHookRegistered()).isFalse();
+ } finally {
+ ThreadPools.initShutdownHook();
+ }
+ }
+
+ @Test
+ public void testThreadPoolManagerAddAndShutdown() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ ExecutorService testExecutor = Executors.newFixedThreadPool(2);
+
+ Duration timeout = Duration.ofSeconds(5);
+ manager.addThreadPool(testExecutor, timeout);
+
+ manager.shutdownAll();
+
+ assertThat(testExecutor.isShutdown()).isTrue();
+ }
+
+ @Test
+ public void testThreadPoolManagerMultipleShutdowns() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ ExecutorService executor1 = Executors.newFixedThreadPool(1);
+ ExecutorService executor2 = Executors.newFixedThreadPool(1);
+
+ Duration timeout = Duration.ofSeconds(5);
+ manager.addThreadPool(executor1, timeout);
+ manager.addThreadPool(executor2, timeout);
+
+ manager.shutdownAll();
+
+ assertThat(executor1.isShutdown()).isTrue();
+ assertThat(executor2.isShutdown()).isTrue();
+ }
+
+ @Test
+ public void testThreadPoolManagerShutdownNowCalled() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ final AtomicBoolean interrupted = new AtomicBoolean(false);
+ ExecutorService slowExecutor = Executors.newFixedThreadPool(1);
+
+ manager.addThreadPool(slowExecutor, Duration.ofMillis(50));
Review Comment:
Use scheduled pool to test this
```
// Use a short termination timeout so shutdownThreadPools will call shutdownNow
ExecutorService slowPool =
ThreadPools.newExitingScheduledPool("test-slow-pool", 1, Duration.ofMillis(50));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3115817017
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +153,89 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Shuts down all thread pools registered via {@link #newExitingWorkerPool(String, int)} or {@link
+ * #newExitingScheduledPool(String, int, Duration)} and removes the JVM shutdown hook.
+ *
+ * <p>This method is useful for:
+ *
+ * <ul>
+ * <li>Preventing thread pool leaks in hot-reload environments
+ * <li>Managing graceful application shutdown when the application needs to handle its own
+ * shutdown hooks (e.g. to commit pending files before exiting)
+ * </ul>
+ *
+ * <p>Only call this method at the end of the intended usage of the library. Calling it earlier
+ * will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ @VisibleForTesting
+ static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ () -> {
+ shutdownHook = null;
+ shutdownThreadPools();
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
Review Comment:
```suggestion
* Remove the shutdown hook for the thread pools created via the {@link
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3136904860
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -123,12 +132,7 @@ public static ExecutorService newWorkerPool(String namePrefix) {
}
/**
- * Creates a fixed-size thread pool that uses daemon threads. The pool is wrapped with {@link
- * MoreExecutors#getExitingExecutorService(ThreadPoolExecutor)}, which registers a shutdown hook
- * to ensure the pool terminates when the JVM exits. <b>Important:</b> Even if the pool is
- * explicitly shut down using {@link ExecutorService#shutdown()}, the shutdown hook is <i>not</i>
- * removed. This can lead to accumulation of shutdown hooks if this method is used repeatedly for
- * short-lived thread pools.
+ * Creates a fixed-size thread pool that uses daemon threads.
Review Comment:
actually it's right there:
> The pool will be stopped by the
> * shutdown hook to ensure the pool terminates when the JVM exits - unless `removeShutdownHook` is
> * called, in which case it is user's responsibility to stop the threadpools using the
> * `shutdownThreadPools` method.
or do you have something else in mind?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4339444427
> I've updated the PR description - I hope it's clearer.
If I understand correctly from #15031, it looks like the issue is that Flink uses new classloaders and the shutdown hook is never called because the JVM never exits. So we need use threadpools tied to Flink's lifecycle. That's reasonable, but we definitely do NOT want to give "users" control over threadpool lifecycles.
I think this implementation goes a bit beyond what we need to do. Assuming that we want to have a way to shut down these pools, that doesn't mean that we should remove the shutdown hook. The contract of `newExitingWorkerPool` is that it will be shut down by a hook. Why not update the shutdown hook so that it is a noop if the pool is already shut down?
Also, it's debatable whether we need to track all pools created by the factory methods here. This class is responsible for its own static pools and its methods are reused elsewhere. Having a way to close all threadpools created by this class seems very risky to me. I'm not convinced that we want to expose a method to shut down a single static threadpool directly, let alone one that will shut down pools created and managed by other classes. The reason for static pools is to ensure one is available. Allowing them to be closed allows one class to break functionality for another. Allowing all of them to be closed is even more risk.
I think the solution is not to expose these methods. Instead, I propose two things:
1. Deprecate and remove `newExitingWorkerPool` and fix the 3 uses of it. `HadoopFileIO` can manage the lifecycle of its thread pool.
2. Lazily create the pools managed by `ThreadPools`. The APIs that use the static pools support passing in an executor service with a lifecycle managed by the client. This means that a Flink application can manage its own threadpools (which is why we added the API) and shut them down appropriately. If Flink isn't using the static pools, then they will never start and we don't need a shutdown method.
I think if we do these two things then we will fix the problem and be better off. I'd also be okay with an option to prevent these pools from being started if we need more guarantees. I think that's safer and better than using them but shutting them down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2890901492
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
Review Comment:
Why not private?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2895473063
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +152,88 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
+ if (shutdownHook != null) {
+
+ try {
+ Runtime.getRuntime().removeShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot remove the shutdown hook for thread pools clean up", e);
+ }
+
+ shutdownHook = null;
+ }
+ }
+
+ /**
+ * Check if the shutdown hook has been registered.
+ *
+ * @return true if the shutdown hook is registered, false otherwise
+ */
+ public static synchronized boolean isShutdownHookRegistered() {
Review Comment:
only used in the test ATM, not real benefit - if you prefer, happy to get rid of it. Any recommendation for the test in that case though?
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +152,88 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
+ if (shutdownHook != null) {
+
+ try {
+ Runtime.getRuntime().removeShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot remove the shutdown hook for thread pools clean up", e);
+ }
+
+ shutdownHook = null;
+ }
+ }
+
+ /**
+ * Check if the shutdown hook has been registered.
+ *
+ * @return true if the shutdown hook is registered, false otherwise
+ */
+ public static synchronized boolean isShutdownHookRegistered() {
Review Comment:
only used in the test ATM, no real benefit - if you prefer, happy to get rid of it. Any recommendation for the test in that case though?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2895000357
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
Review Comment:
I think we have `@VisibleForTesting` for classes as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2895502040
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +152,88 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
+ if (shutdownHook != null) {
+
+ try {
+ Runtime.getRuntime().removeShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot remove the shutdown hook for thread pools clean up", e);
+ }
+
+ shutdownHook = null;
+ }
+ }
+
+ /**
+ * Check if the shutdown hook has been registered.
+ *
+ * @return true if the shutdown hook is registered, false otherwise
+ */
+ public static synchronized boolean isShutdownHookRegistered() {
Review Comment:
I suppose I can make it non-public too. will do this for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2878232437
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -148,9 +152,99 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* ensure the pool terminates when the JVM exits. This is suitable for long-lived thread pools
* that should be automatically cleaned up on JVM shutdown.
*/
- public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ public static synchronized ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(new ExecutorServiceWithTimeout(service, DEFAULT_SHUTDOWN_TIMEOUT));
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static synchronized void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : THREAD_POOLS_TO_SHUTDOWN) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ THREAD_POOLS_TO_SHUTDOWN.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
Review Comment:
to explain better. we write a new iceberg file periodically - and create a new one for the current window. if our application gets restarted (shutdown hooks get triggered due to SIGTERM), we want to write (commit / flush) the currently open window.
so we have a handler for the shutdown hook, that completes the writes for the iceberg library (and adds the new file to aws glue catalog). but if jvm has killed all the threadpools - we cannot.
we the idea is that in our use case, we DO NOT want iceberg library to use any shutdown hook (because of undeterministic nature of them, no ability to control the sequence). instead, we will unregister the default behaviour (opt out) and invoke the shutting down only when our application is ready to do it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4533166681
hi. I think this is an important issue and the static shutdown hook based approach as implemented currently raises various problems that I tried to address (tactically) in the PR.
having this problem (the family of problems) solved one way or another would definitely receive thanks from me, and I suspect other users who have similar issues - be it to be able to unload the iceberg exporter in dynamic environments, or make sure it has not stopped itself before the main application finishes exporting the files on shutdown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2878179224
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -148,9 +152,99 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* ensure the pool terminates when the JVM exits. This is suitable for long-lived thread pools
* that should be automatically cleaned up on JVM shutdown.
*/
- public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ public static synchronized ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(new ExecutorServiceWithTimeout(service, DEFAULT_SHUTDOWN_TIMEOUT));
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static synchronized void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : THREAD_POOLS_TO_SHUTDOWN) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ THREAD_POOLS_TO_SHUTDOWN.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
Review Comment:
Why do you need to call this method? You could simply call `shutdownThreadPools()` which will also remove the shutdown hook.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2878232437
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -148,9 +152,99 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* ensure the pool terminates when the JVM exits. This is suitable for long-lived thread pools
* that should be automatically cleaned up on JVM shutdown.
*/
- public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ public static synchronized ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(new ExecutorServiceWithTimeout(service, DEFAULT_SHUTDOWN_TIMEOUT));
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static synchronized void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : THREAD_POOLS_TO_SHUTDOWN) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ THREAD_POOLS_TO_SHUTDOWN.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
Review Comment:
to explain better. we write a new iceberg file periodically - and create a new one for the current window. if our application gets restarted, we want to write (commit / flush) the currently open window.
so we have a handler for the shutdown hook, that completes the writes for the iceberg library (and adds the new file to aws glue catalog). but if jvm has killed all the threadpools - we cannot.
we the idea is that in our use case, we DO NOT want iceberg library to use any shutdown hook. instead, we will unregister the default behaviour (opt out) and invoke the shutting down only when our application is ready to do it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4000897865
hi @mxm I've refactored it a little and added some tests - what do you think?
I've also confirmed that the self-published library helps for my use case and I can commit all files on app shutdown, and then manually invoke the thread cleanup when the app is ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2841704066
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +166,98 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ ExecutorService item;
+ Queue<ExecutorService> pendingShutdown = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ pendingShutdown.add(item);
+ }
+ while ((item = pendingShutdown.poll()) != null) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = SHUTDOWN_TIMEOUT.toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static void removeShutdownHook() {
+ if (shutdownHook != null) {
+ try {
+ Runtime.getRuntime().removeShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot remove the shutdown hook for thread pools clean up", e);
+ }
+ shutdownHook = null;
Review Comment:
This is not safe when called concurrently and can produce `NullPointerException` when removing the shutdown hook if two threads execute this at the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2845758223
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +166,98 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ ExecutorService item;
+ Queue<ExecutorService> pendingShutdown = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ pendingShutdown.add(item);
+ }
+ while ((item = pendingShutdown.poll()) != null) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = SHUTDOWN_TIMEOUT.toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static void removeShutdownHook() {
+ if (shutdownHook != null) {
+ try {
+ Runtime.getRuntime().removeShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot remove the shutdown hook for thread pools clean up", e);
+ }
+ shutdownHook = null;
Review Comment:
actually even simpler - changed these two methods to synchronized, as they are not intended to be performant - intended to be called once, not on hotpath
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +166,98 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ ExecutorService item;
+ Queue<ExecutorService> pendingShutdown = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ pendingShutdown.add(item);
+ }
+ while ((item = pendingShutdown.poll()) != null) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = SHUTDOWN_TIMEOUT.toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static void removeShutdownHook() {
+ if (shutdownHook != null) {
+ try {
+ Runtime.getRuntime().removeShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot remove the shutdown hook for thread pools clean up", e);
+ }
+ shutdownHook = null;
Review Comment:
actually even simpler - changed these two methods to synchronized, as they are not intended to be performant - intended to be called once, not on hotpath, please check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2905079658
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ synchronized void addThreadPool(ExecutorService service, Duration timeout) {
+ threadPoolsToShutdown.add(new ExecutorServiceWithTimeout(service, timeout));
+ }
+
+ /** Shut down all registered thread pools. */
+ synchronized void shutdownAll() {
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : threadPoolsToShutdown) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ threadPoolsToShutdown.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
Review Comment:
We usually don't add a new line if the enclosing block is also closed, so 318 is not strictly required. Also for symmetry the 308 is not required either
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2895467032
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -42,8 +45,12 @@ private ThreadPools() {}
public static final String WORKER_THREAD_POOL_SIZE_PROP =
SystemConfigs.WORKER_THREAD_POOL_SIZE.propertyKey();
+ private static final Duration DEFAULT_SHUTDOWN_TIMEOUT = Duration.ofSeconds(120);
Review Comment:
it preserves the behaviour - 120 seconds used to come from guava: https://github.com/google/guava/blob/6e87e5148e7d6d81ee3b774fd9ab8345cd27148c/guava/src/com/google/common/util/concurrent/MoreExecutors.java#L249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2891045175
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +152,86 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ static synchronized void initShutdownHook() {
Review Comment:
this is used in tests - so that they could access it to test.. do you have alternative recommendation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2878144054
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -148,9 +152,99 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* ensure the pool terminates when the JVM exits. This is suitable for long-lived thread pools
* that should be automatically cleaned up on JVM shutdown.
*/
- public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ public static synchronized ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(new ExecutorServiceWithTimeout(service, DEFAULT_SHUTDOWN_TIMEOUT));
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static synchronized void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : THREAD_POOLS_TO_SHUTDOWN) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ THREAD_POOLS_TO_SHUTDOWN.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
Review Comment:
this is for the use case I mentioned we need - to unregister the automatic shutdown hook, so that _our code_ could run on shutdown hook first, commit and flush anything iceberg, and then we'll invoke the service shutdown manually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4066568234
Hi @pvary , @nastra ,
Please can I ask what's the next step with this PR and if there is anything I can do to help this PR to progress further?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2894985703
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +152,88 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
+ if (shutdownHook != null) {
+
+ try {
+ Runtime.getRuntime().removeShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot remove the shutdown hook for thread pools clean up", e);
+ }
+
+ shutdownHook = null;
+ }
+ }
+
+ /**
+ * Check if the shutdown hook has been registered.
+ *
+ * @return true if the shutdown hook is registered, false otherwise
+ */
+ public static synchronized boolean isShutdownHookRegistered() {
Review Comment:
What is the intended use for this? The `initShutdownHook` is reentrant, so it is ok to call it multiple times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2904843745
##########
core/src/test/java/org/apache/iceberg/util/TestThreadPools.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.junit.jupiter.api.Test;
+
+public class TestThreadPools {
+
+ @Test
+ public void testRemoveShutdownHook() {
+ try {
+ assertThat(ThreadPools.isShutdownHookRegistered()).isTrue();
+ ThreadPools.removeShutdownHook();
+ assertThat(ThreadPools.isShutdownHookRegistered()).isFalse();
+ } finally {
+ ThreadPools.initShutdownHook();
+ }
Review Comment:
Use this instead of having a try/catch every place:
```
@AfterEach
void restoreShutdownHook() {
ThreadPools.initShutdownHook();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3101848302
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +153,89 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Shuts down all thread pools registered via {@link #newExitingWorkerPool(String, int)} or {@link
+ * #newExitingScheduledPool(String, int, Duration)} and removes the JVM shutdown hook.
+ *
+ * <p>This method is useful for:
+ *
+ * <ul>
+ * <li>Preventing thread pool leaks in hot-reload environments
+ * <li>Managing graceful application shutdown when the application needs to handle its own
+ * shutdown hooks (e.g. to commit pending files before exiting)
+ * </ul>
+ *
+ * <p>Only call this method at the end of the intended usage of the library. Calling it earlier
+ * will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ @VisibleForTesting
+ static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ () -> {
+ shutdownHook = null;
Review Comment:
I don't think this is technically thread safe because the runnable doesn't run within the synchronized context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4268542120
I'll take a closer look at this next week
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3115765107
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java:
##########
@@ -112,14 +109,8 @@ class S3OutputStream extends PositionOutputStream {
synchronized (S3OutputStream.class) {
if (executorService == null) {
executorService =
- MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor)
- Executors.newFixedThreadPool(
- s3FileIOProperties.multipartUploadThreads(),
- new ThreadFactoryBuilder()
- .setDaemon(true)
- .setNameFormat("iceberg-s3fileio-upload-%d")
- .build()));
+ ThreadPools.newExitingWorkerPool(
Review Comment:
why are we updating only this place and not other places as well? Not saying that we should update all places in this PR but I'm trying to understand why only this file is updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3118074606
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +153,89 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Shuts down all thread pools registered via {@link #newExitingWorkerPool(String, int)} or {@link
+ * #newExitingScheduledPool(String, int, Duration)} and removes the JVM shutdown hook.
+ *
+ * <p>This method is useful for:
+ *
+ * <ul>
+ * <li>Preventing thread pool leaks in hot-reload environments
+ * <li>Managing graceful application shutdown when the application needs to handle its own
+ * shutdown hooks (e.g. to commit pending files before exiting)
+ * </ul>
+ *
+ * <p>Only call this method at the end of the intended usage of the library. Calling it earlier
+ * will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ @VisibleForTesting
+ static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ () -> {
+ shutdownHook = null;
Review Comment:
this thread only executed as a shutdown hook - when jvm is exiting, and there can be only one shutdown hook registered. As such, this particular line should be both harmless, and also harmless to remove.
do you have any preferences for a change here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3118118216
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java:
##########
@@ -112,14 +109,8 @@ class S3OutputStream extends PositionOutputStream {
synchronized (S3OutputStream.class) {
if (executorService == null) {
executorService =
- MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor)
- Executors.newFixedThreadPool(
- s3FileIOProperties.multipartUploadThreads(),
- new ThreadFactoryBuilder()
- .setDaemon(true)
- .setNameFormat("iceberg-s3fileio-upload-%d")
- .build()));
+ ThreadPools.newExitingWorkerPool(
Review Comment:
I was about to reply that this is the only place I found - but in fact this is the only place that had been stopped by `MoreExecutors` while I was testing it (my application still required it to continue until our own shutdown hook was invoked so it could stop gracefully)
the other places are these:
```
rg getExitingExecutorService | grep -v 'src/test' | grep -v 'src/integration'
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/procedures/BaseProcedure.java: MoreExecutors.getExitingExecutorService(
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/BaseProcedure.java: MoreExecutors.getExitingExecutorService(
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: return MoreExecutors.getExitingExecutorService(
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: return MoreExecutors.getExitingExecutorService(
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java: return MoreExecutors.getExitingExecutorService(
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java: return MoreExecutors.getExitingExecutorService(
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/procedures/BaseProcedure.java: MoreExecutors.getExitingExecutorService(
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: return MoreExecutors.getExitingExecutorService(
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java: return MoreExecutors.getExitingExecutorService(
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/BaseProcedure.java: MoreExecutors.getExitingExecutorService(
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java: return MoreExecutors.getExitingExecutorService(
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java: return MoreExecutors.getExitingExecutorService(
```
I'm not running spark so I cannot test them. Any preferences/suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3136478523
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java:
##########
@@ -112,14 +109,8 @@ class S3OutputStream extends PositionOutputStream {
synchronized (S3OutputStream.class) {
if (executorService == null) {
executorService =
- MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor)
- Executors.newFixedThreadPool(
- s3FileIOProperties.multipartUploadThreads(),
- new ThreadFactoryBuilder()
- .setDaemon(true)
- .setNameFormat("iceberg-s3fileio-upload-%d")
- .build()));
+ ThreadPools.newExitingWorkerPool(
Review Comment:
I think we should be updating this file and other places as well in a separate PR, otherwise it's difficult to justify changing one place without updating other places
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3136914202
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -123,12 +132,7 @@ public static ExecutorService newWorkerPool(String namePrefix) {
}
/**
- * Creates a fixed-size thread pool that uses daemon threads. The pool is wrapped with {@link
- * MoreExecutors#getExitingExecutorService(ThreadPoolExecutor)}, which registers a shutdown hook
- * to ensure the pool terminates when the JVM exits. <b>Important:</b> Even if the pool is
- * explicitly shut down using {@link ExecutorService#shutdown()}, the shutdown hook is <i>not</i>
- * removed. This can lead to accumulation of shutdown hooks if this method is used repeatedly for
- * short-lived thread pools.
+ * Creates a fixed-size thread pool that uses daemon threads.
Review Comment:
@svalaskevicius that javadoc is on the other `newWorkerPool()` method but we removed most of the javadoc on this methd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-3991006171
> > I'm a bit wary of the nondeterministic state of this PR/task - esp with the new parallel PR out there :) would it be possible to have a think and set out a single list of changes pending before it can be considered complete/good enough?
>
> I understand that this feels uncertain, but as long as we discover open issues, we need to address them. It's a good idea to make a plan and update it over time. Both contributor and reviewer can do that. Let me try to formulate the acceptance criteria
thanks for this reply - it is reassuring, and I agree wrt the new uncovered issues. I'll respond below inline too:
> 1. Centrally manage all static thread pools and allow them to be shut down manually
:+1: I believe this is done (subject to review ofc!)
> 2. Keep the existing behavior in place to shut down thread pools on JVM shutdown
:+1: I believe this is done, with an exception the the shutdown duration changed in `AuthRefreshPoolHolder` (which I copied from the other PR)
> 3. Clean code and tests
:+1: I believe this is done -- for the code part
> > also, what tests do you have in mind?
>
> There are several things we can test:
>
> * Verify static thread pools are registered
ok, I can add this
> * Verify shutdown works as intended
>
> * Verify shutdown hook gets added / removed
these might need some thinking how to void stopping the static thread pools so that other tests could continue running by construction. I suppose I can extract a non static class similar to the other PR. would not be a big change I believe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2890928006
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ synchronized void addThreadPool(ExecutorService service, Duration timeout) {
+ threadPoolsToShutdown.add(new ExecutorServiceWithTimeout(service, timeout));
+ }
+
+ /** Shut down all registered thread pools. */
+ synchronized void shutdownAll() {
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : threadPoolsToShutdown) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ threadPoolsToShutdown.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+ }
+
+ static class ExecutorServiceWithTimeout {
Review Comment:
Why not private?
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
Review Comment:
and also all of the methods
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ synchronized void addThreadPool(ExecutorService service, Duration timeout) {
+ threadPoolsToShutdown.add(new ExecutorServiceWithTimeout(service, timeout));
+ }
+
+ /** Shut down all registered thread pools. */
+ synchronized void shutdownAll() {
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : threadPoolsToShutdown) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ threadPoolsToShutdown.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+ }
+
+ static class ExecutorServiceWithTimeout {
Review Comment:
and also all of the methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2891055146
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ synchronized void addThreadPool(ExecutorService service, Duration timeout) {
+ threadPoolsToShutdown.add(new ExecutorServiceWithTimeout(service, timeout));
+ }
+
+ /** Shut down all registered thread pools. */
+ synchronized void shutdownAll() {
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : threadPoolsToShutdown) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ threadPoolsToShutdown.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
Review Comment:
can log. IIRC this was taken from guava as it was, but sure - we can do better :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2894969199
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -98,16 +111,11 @@ public static ScheduledExecutorService authRefreshPool() {
private static class AuthRefreshPoolHolder {
private static final ScheduledExecutorService INSTANCE =
ThreadPools.newExitingScheduledPool(
- "auth-session-refresh", AUTH_REFRESH_THREAD_POOL_SIZE, Duration.ZERO);
+ "auth-session-refresh", AUTH_REFRESH_THREAD_POOL_SIZE, Duration.ofSeconds(10));
Review Comment:
Why is this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3115893170
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +153,89 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Shuts down all thread pools registered via {@link #newExitingWorkerPool(String, int)} or {@link
+ * #newExitingScheduledPool(String, int, Duration)} and removes the JVM shutdown hook.
+ *
+ * <p>This method is useful for:
+ *
+ * <ul>
+ * <li>Preventing thread pool leaks in hot-reload environments
+ * <li>Managing graceful application shutdown when the application needs to handle its own
+ * shutdown hooks (e.g. to commit pending files before exiting)
+ * </ul>
+ *
+ * <p>Only call this method at the end of the intended usage of the library. Calling it earlier
+ * will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ @VisibleForTesting
+ static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ () -> {
+ shutdownHook = null;
+ shutdownThreadPools();
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
+ if (shutdownHook != null) {
+ try {
+ Runtime.getRuntime().removeShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot remove the shutdown hook for thread pools clean up", e);
+ }
+
+ shutdownHook = null;
+ }
+ }
+
+ /**
+ * Check if the shutdown hook has been registered.
+ *
+ * @return true if the shutdown hook is registered, false otherwise
+ */
+ @VisibleForTesting
+ static synchronized boolean isShutdownHookRegistered() {
Review Comment:
is this for testing only?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4348265412
@pvary, let me outline my assumptions so that my rationale is clear. First, I'm assuming that we are only worrying about the exiting pools. The other pools should have their own lifecycle management. The non-exiting pools are what I would expect `FileIO` instances to use, unless the pools are shared. Next, I see `newExistingScheduledPool` has 2 outside uses, `S3FileIO` and `GCSFileIO`, and `newExitingWorkerPool` has 3 outside uses in `HadoopFileIO`, `RESTMetricsReporter`, and `AuthSessionCache`. 5 uses is not a huge issue, but this could be a larger problem if the actual issue is that `FileIO` don't properly manage pool lifecycles.
However, I don't think the `FileIO` situation has much influence over the decision here. This PR proposes that we add:
1. A way to remove shutdown hooks from all thread pools
2. A global shutdown for all thread pools created by `ThreadPools`
I think those are both poor solutions. For the first one, if you don't want a threadpool that is configured like the static pools, the right answer is to use your own threadpool and manage its lifecycle. Customizing the lifecycle of the static pools opens the possibility of misconfiguration. The reasonable way to allow using your own threadpool (and not worrying about the shutdown hooks) is to lazily start the static pools. But if you can already shut down the static pools, then we don't really need even that.
For the second one, I think it is a bad idea to have a global shutdown for the static pools, let along all of the pools created by the factory methods. This is a drastic behavior change that can be misused. It may allow us to avoid fixing pools created by `FileIO`, but we should not be creating dangerous ways to work around the right solution. And if I understand correctly, the problem you're saying we need to solve is that the pools created for `FileIO` have inconsistently managed lifecycles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4516869035
@svalaskevicius, @rdblue: Shall we discuss this on the next community Sync?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4352158562
@rdblue: Thanks for the detailed reply. This helps clarify where our views differ.
> First, I'm assuming that we are only worrying about the exiting pools.
Agreed.
> The other pools should have their own lifecycle management.
Fully agree.
> The non-exiting pools are what I would expect FileIO instances to use, unless the pools are shared.
In practice, all `FileIO` implementations currently rely on static exiting worker pools. The pattern is that these pools are created on first use and then shared across instances. That approach has two advantages:
- The threads are used only occasionally, so sharing them keeps the overall thread footprint small.
- The pool size naturally bounds the level of outgoing parallelism.
> This PR proposes that we add:
> - A way to remove shutdown hooks from all thread pools
> - A global shutdown for all thread pools created by ThreadPools
The naming may indeed need improvement, but the proposed shutdown only targets the **exiting** pools created by `newExitingWorkerPool` and `newExitingScheduledPool`. These are exactly the pools where we currently lack direct control over shutdown.
> the problem you're saying we need to solve is that the pools created for FileIO have inconsistently managed lifecycles.
From my perspective, the pattern across `FileIO` implementations is actually quite consistent: they require a global shared pool per `FileIO` type, reused across instances and normally closed only on exit. The issue is that “exit” is not always equivalent to a JVM shutdown. In cases like hot replacement, we need a way to release those resources even while the JVM keeps running.
Based on this discussion, I see a few possible directions:
1. Adopt the previously mentioned (but somewhat invasive) change to the `FileIO API`.
2. Introduce a `ThreadPoolManager` outside of `ThreadPools` and migrate `FileIO` implementations to use managed pools.
3. Keep the `ThreadPoolManager` within `ThreadPools`, but don’t let it manage the standard pools (`WORKER_POOL`, `DELETE_WORKER_POOL`, `AuthRefreshPoolHolder.INSTANCE`).
Independently of which path we take, I think we should ensure that standard pools are lazily initialized, and that every place using them (using the methods: `getWorkerPool`, `getDeleteWorkerPool`, `authRefreshPool`) allows injecting an external pool as a substitute.
I would prefer 2 or 3. Do you think it would be reasonable? Do you have another suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4125775623
Hi @nastra, @rdblue, @RussellSpitzer,
Sorry to be a bother and I appreciate you're busy, but it's been some time since any update on this PR so I pinged you just in case you missed the earlier messages.
We're planning to adopt iceberg and are currently testing it before the release having done the integration work, but we have an issue that stops us using the library in any real environment. This is our only blocker and it would be wonderful to have it resolved!
Can you please have a look and let us know if there are any changes still needed, or if it can be merged to main?
Thanks,
Sarunas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2805134809
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
+ ExecutorService item;
+ Queue<ExecutorService> invoked = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ invoked.add(item);
+ }
+ while ((item = invoked.poll()) != null) {
+ try {
+ item.awaitTermination(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ }
+ }
+
+ /**
+ * Initialise a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>NOTE: it is normally started automatically, and does not need to be invoked manually, unless
+ * it has been stopped via the {@link #removeShutdownHook()}
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static void initShutdownHook() {
+ if (shutDownHook == null) {
+ shutDownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutDownStartedThreadPools();
+ }
+ });
+
+ try {
+ shutDownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ // OK if we can't set the name in this environment.
+ }
Review Comment:
logging added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #15312: Core, AWS: Allow stopping thread pools manually
URL: https://github.com/apache/iceberg/pull/15312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4031503005
by the way, what is the policy wrt setting the above review items as "Resolved" - should I have done that when applying or the reviewers - to denote they are happy with how it was applied? as I think all of them are resolved by now but there is quite a few left unmarked as such :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4031677872
> by the way, what is the policy wrt setting the above review items as "Resolved" - should I have done that when making changes or the reviewers - to denote they are happy with how it was changed? as I think all of them are resolved by now but there is quite a few left unmarked as such :)
AFAIK there is no official policy, but I think it is good practice for the PR author to resolve them as they get addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2904839890
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +265,80 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ @VisibleForTesting
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ synchronized void addThreadPool(ExecutorService service, Duration timeout) {
+ threadPoolsToShutdown.add(new ExecutorServiceWithTimeout(service, timeout));
+ }
+
+ /** Shut down all registered thread pools. */
+ synchronized void shutdownAll() {
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+
+ for (ExecutorServiceWithTimeout item : threadPoolsToShutdown) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+
+ threadPoolsToShutdown.clear();
+
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ LOG.warn("Interrupted while shutting down, ignoring", e);
+ }
+
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+ }
+
+ private static class ExecutorServiceWithTimeout {
+ private ExecutorService service;
+ private Duration timeout;
+
+ private ExecutorServiceWithTimeout(ExecutorService service, Duration timeout) {
+ this.service = service;
+ this.timeout = timeout;
+ }
+
+ private ExecutorService getService() {
+ return service;
+ }
+
+ private Duration getTimeout() {
+ return timeout;
+ }
+ }
Review Comment:
```suggestion
private record ExecutorServiceWithTimeout(ExecutorService service, Duration timeout) {}
```
Make this a record and adjust the calls accordingly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2804814950
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
+ ExecutorService item;
+ Queue<ExecutorService> invoked = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ invoked.add(item);
+ }
+ while ((item = invoked.poll()) != null) {
+ try {
+ item.awaitTermination(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ }
+ }
+
+ /**
+ * Initialise a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>NOTE: it is normally started automatically, and does not need to be invoked manually, unless
+ * it has been stopped via the {@link #removeShutdownHook()}
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static void initShutdownHook() {
+ if (shutDownHook == null) {
+ shutDownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutDownStartedThreadPools();
+ }
+ });
+
+ try {
+ shutDownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ // OK if we can't set the name in this environment.
+ }
Review Comment:
this try catch was .... copied from the prev implementation in the library, and it didnt have it in the addShutdownHook, so I just replicated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-3990934943
>I'm a bit wary of the nondeterministic state of this PR/task - esp with the new parallel PR out there :) would it be possible to have a think and set out a single list of changes pending before it can be considered complete/good enough?
I understand that this feels uncertain, but as long as we discover open issues, we need to address them. It's a good idea to make a plan and update it over time. Both contributor and reviewer can do that. Let me try to formulate the acceptance criteria
1. Centrally manage all static thread pools and allow them to be shut down manually
2. Keep the existing behavior in place to shut down thread pools on JVM shutdown
3. Clean code and tests
>also, what tests do you have in mind?
There are several things we can test:
- Verify static thread pools are registered
- Verify shutdown works as intended
- Verify shutdown hook gets added / removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2878212605
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -148,9 +152,99 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* ensure the pool terminates when the JVM exits. This is suitable for long-lived thread pools
* that should be automatically cleaned up on JVM shutdown.
*/
- public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ public static synchronized ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(new ExecutorServiceWithTimeout(service, DEFAULT_SHUTDOWN_TIMEOUT));
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static synchronized void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : THREAD_POOLS_TO_SHUTDOWN) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ THREAD_POOLS_TO_SHUTDOWN.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
Review Comment:
but I cannot shut them down - I still need iceberg library to work! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2873605065
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
Review Comment:
I still think it is a bit unexpected that this can be called concurrently. What is the benefit really?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2873623657
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -44,6 +50,9 @@ private ThreadPools() {}
public static final int WORKER_THREAD_POOL_SIZE = SystemConfigs.WORKER_THREAD_POOL_SIZE.value();
+ private static final ConcurrentLinkedDeque<ExecutorService> THREAD_POOLS_TO_SHUTDOWN =
+ new ConcurrentLinkedDeque<>();
Review Comment:
I would prefer a simpler solution here with a simple `List<ExecutorService>`. I don't think it hurts to synchronize on updates to the list and on the shutdown logic. Let's make this simple. There is no need to allow for multiple threads to issue the shutdown at the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2891048142
##########
core/src/test/java/org/apache/iceberg/util/TestThreadPools.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.junit.jupiter.api.Test;
+
+public class TestThreadPools {
+
+ @Test
+ public void testRemoveShutdownHook() {
+ try {
+ assertThat(ThreadPools.isShutdownHookRegistered()).isTrue();
+ ThreadPools.removeShutdownHook();
+ assertThat(ThreadPools.isShutdownHookRegistered()).isFalse();
+ } finally {
+ ThreadPools.initShutdownHook();
+ }
+ }
+
+ @Test
+ public void testThreadPoolManagerAddAndShutdown() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ ExecutorService testExecutor = Executors.newFixedThreadPool(2);
+
+ Duration timeout = Duration.ofSeconds(5);
+ manager.addThreadPool(testExecutor, timeout);
+
+ manager.shutdownAll();
+
+ assertThat(testExecutor.isShutdown()).isTrue();
+ }
+
+ @Test
+ public void testThreadPoolManagerMultipleShutdowns() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ ExecutorService executor1 = Executors.newFixedThreadPool(1);
+ ExecutorService executor2 = Executors.newFixedThreadPool(1);
+
+ Duration timeout = Duration.ofSeconds(5);
+ manager.addThreadPool(executor1, timeout);
+ manager.addThreadPool(executor2, timeout);
+
+ manager.shutdownAll();
+
+ assertThat(executor1.isShutdown()).isTrue();
+ assertThat(executor2.isShutdown()).isTrue();
+ }
+
+ @Test
+ public void testThreadPoolManagerShutdownNowCalled() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ final AtomicBoolean interrupted = new AtomicBoolean(false);
+ ExecutorService slowExecutor = Executors.newFixedThreadPool(1);
+
+ manager.addThreadPool(slowExecutor, Duration.ofMillis(50));
+
+ CountDownLatch threadStarted = new CountDownLatch(1);
+ CountDownLatch threadInterrupted = new CountDownLatch(1);
+
+ slowExecutor.submit(
+ () -> {
+ try {
+ threadStarted.countDown();
+ Thread.sleep(60_000);
Review Comment:
yeah, will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2891082232
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ synchronized void addThreadPool(ExecutorService service, Duration timeout) {
+ threadPoolsToShutdown.add(new ExecutorServiceWithTimeout(service, timeout));
+ }
+
+ /** Shut down all registered thread pools. */
+ synchronized void shutdownAll() {
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : threadPoolsToShutdown) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ threadPoolsToShutdown.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
Review Comment:
I suppose I'll add new lines before try and after closing of that block - so it's visually separate... (?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2891092036
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ synchronized void addThreadPool(ExecutorService service, Duration timeout) {
+ threadPoolsToShutdown.add(new ExecutorServiceWithTimeout(service, timeout));
+ }
+
+ /** Shut down all registered thread pools. */
+ synchronized void shutdownAll() {
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : threadPoolsToShutdown) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ threadPoolsToShutdown.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+ }
+
+ static class ExecutorServiceWithTimeout {
Review Comment:
yep. will change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3115887023
##########
core/src/test/java/org/apache/iceberg/util/TestThreadPools.java:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+@Timeout(value = 30, unit = TimeUnit.SECONDS)
+class TestThreadPools {
+
+ @AfterEach
+ void restoreShutdownHook() {
+ ThreadPools.initShutdownHook();
+ }
+
+ @Test
+ void testRemoveShutdownHook() {
Review Comment:
let's omit `test` prefix in new tests as those don't add any value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3115913355
##########
core/src/test/java/org/apache/iceberg/util/TestThreadPools.java:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+@Timeout(value = 30, unit = TimeUnit.SECONDS)
+class TestThreadPools {
Review Comment:
this should also verify that WORKER_POOL and DELETE_WORKER_POOL are shut down
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3115849947
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +153,89 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Shuts down all thread pools registered via {@link #newExitingWorkerPool(String, int)} or {@link
+ * #newExitingScheduledPool(String, int, Duration)} and removes the JVM shutdown hook.
+ *
+ * <p>This method is useful for:
+ *
+ * <ul>
+ * <li>Preventing thread pool leaks in hot-reload environments
+ * <li>Managing graceful application shutdown when the application needs to handle its own
+ * shutdown hooks (e.g. to commit pending files before exiting)
+ * </ul>
+ *
+ * <p>Only call this method at the end of the intended usage of the library. Calling it earlier
Review Comment:
I think we should use some other wording than `library`. Also we should be more explicit about when this should be used, because calling this at the wrong time will cause unwanted side effects and errors that are hard to debug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-3926778735
@mxm
> It works, but it is quite nondeterministic how the shutdown is executed. With the current code, the shutdown timeout can be reset multiple times if there are multiple calls to shutdownThreadPools().
yeah, but there are no linear types in java, so the best I could do is to throw an exception if it's called again. but if the library is used after calling it the first time there should be quite a few exceptions anw.
IMO at best, I could update the javadoc to word it somehow more seriously - to only call this at the end of the intended usage of the library, and never before. Also, it is safe to call it multiple times -- but only AFTER the client code is sure that it's not longer using the library
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2827301341
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +166,85 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutdownStartedThreadPools() {
+ long startTime = System.nanoTime();
+ ExecutorService item;
+ Queue<ExecutorService> invoked = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ invoked.add(item);
+ }
+ while ((item = invoked.poll()) != null) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = SHUTDOWN_TIMEOUT_SECONDS * 1_000_000_000L - timeElapsed;
+ if (remainingTime > 0) {
Review Comment:
hmm we still need to calculate the remainingTime to pass to the function to wait. but if this comment is about the `break` instead of `if` I can add it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2894979580
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +152,88 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
+ if (shutdownHook != null) {
+
Review Comment:
nit: If we do anything else, please remove this new line
Not important and really just some personal preference
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2895700535
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +264,82 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ @VisibleForTesting
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ @VisibleForTesting
Review Comment:
Since the class is annotated, we don't need to duplicate it on the method. Sorry for the confusion, I have done a sequential review of the code, and did not collect the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2895011199
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -42,8 +45,12 @@ private ThreadPools() {}
public static final String WORKER_THREAD_POOL_SIZE_PROP =
SystemConfigs.WORKER_THREAD_POOL_SIZE.propertyKey();
+ private static final Duration DEFAULT_SHUTDOWN_TIMEOUT = Duration.ofSeconds(120);
Review Comment:
This is a new addition. Can you please help me understand why do we want an extra delay in shutdown when we have the possibility to handle the shutdown manually?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2877125089
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -44,6 +50,9 @@ private ThreadPools() {}
public static final int WORKER_THREAD_POOL_SIZE = SystemConfigs.WORKER_THREAD_POOL_SIZE.value();
+ private static final ConcurrentLinkedDeque<ExecutorService> THREAD_POOLS_TO_SHUTDOWN =
+ new ConcurrentLinkedDeque<>();
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4023291919
thanks @pvary - especially for the commit I could cherry pick! this has been applied now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2904852365
##########
core/src/test/java/org/apache/iceberg/util/TestThreadPools.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.junit.jupiter.api.Test;
+
+public class TestThreadPools {
+
+ @Test
+ public void testRemoveShutdownHook() {
+ try {
+ assertThat(ThreadPools.isShutdownHookRegistered()).isTrue();
+ ThreadPools.removeShutdownHook();
+ assertThat(ThreadPools.isShutdownHookRegistered()).isFalse();
+ } finally {
+ ThreadPools.initShutdownHook();
+ }
+ }
+
+ @Test
+ public void testThreadPoolManagerAddAndShutdown() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ ExecutorService testExecutor = Executors.newFixedThreadPool(2);
+
+ Duration timeout = Duration.ofSeconds(5);
+ manager.addThreadPool(testExecutor, timeout);
+
+ manager.shutdownAll();
+
+ assertThat(testExecutor.isShutdown()).isTrue();
+ }
+
+ @Test
+ public void testThreadPoolManagerMultipleShutdowns() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ ExecutorService executor1 = Executors.newFixedThreadPool(1);
+ ExecutorService executor2 = Executors.newFixedThreadPool(1);
+
+ Duration timeout = Duration.ofSeconds(5);
+ manager.addThreadPool(executor1, timeout);
+ manager.addThreadPool(executor2, timeout);
+
+ manager.shutdownAll();
+
+ assertThat(executor1.isShutdown()).isTrue();
+ assertThat(executor2.isShutdown()).isTrue();
+ }
Review Comment:
```suggestion
@Test
void testMultipleExitingPoolsShutdown() {
ExecutorService pool1 = ThreadPools.newExitingWorkerPool("test-pool-1", 1);
ExecutorService pool2 = ThreadPools.newExitingWorkerPool("test-pool-2", 1);
ThreadPools.shutdownThreadPools();
assertThat(pool1.isShutdown()).isTrue();
assertThat(pool2.isShutdown()).isTrue();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2904861801
##########
core/src/test/java/org/apache/iceberg/util/TestThreadPools.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.junit.jupiter.api.Test;
+
+public class TestThreadPools {
+
+ @Test
+ public void testRemoveShutdownHook() {
+ try {
+ assertThat(ThreadPools.isShutdownHookRegistered()).isTrue();
+ ThreadPools.removeShutdownHook();
+ assertThat(ThreadPools.isShutdownHookRegistered()).isFalse();
+ } finally {
+ ThreadPools.initShutdownHook();
+ }
+ }
+
+ @Test
+ public void testThreadPoolManagerAddAndShutdown() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ ExecutorService testExecutor = Executors.newFixedThreadPool(2);
+
+ Duration timeout = Duration.ofSeconds(5);
+ manager.addThreadPool(testExecutor, timeout);
+
+ manager.shutdownAll();
+
+ assertThat(testExecutor.isShutdown()).isTrue();
+ }
+
+ @Test
+ public void testThreadPoolManagerMultipleShutdowns() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ ExecutorService executor1 = Executors.newFixedThreadPool(1);
+ ExecutorService executor2 = Executors.newFixedThreadPool(1);
+
+ Duration timeout = Duration.ofSeconds(5);
+ manager.addThreadPool(executor1, timeout);
+ manager.addThreadPool(executor2, timeout);
+
+ manager.shutdownAll();
+
+ assertThat(executor1.isShutdown()).isTrue();
+ assertThat(executor2.isShutdown()).isTrue();
+ }
+
Review Comment:
Add a test scheduled pool shutdown
```
@Test
void testExitingScheduledPoolShutdown() {
ExecutorService scheduled =
ThreadPools.newExitingScheduledPool("test-scheduled", 1, Duration.ofSeconds(5));
ThreadPools.shutdownThreadPools();
assertThat(scheduled.isShutdown()).isTrue();
}
``
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3115849947
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +153,89 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Shuts down all thread pools registered via {@link #newExitingWorkerPool(String, int)} or {@link
+ * #newExitingScheduledPool(String, int, Duration)} and removes the JVM shutdown hook.
+ *
+ * <p>This method is useful for:
+ *
+ * <ul>
+ * <li>Preventing thread pool leaks in hot-reload environments
+ * <li>Managing graceful application shutdown when the application needs to handle its own
+ * shutdown hooks (e.g. to commit pending files before exiting)
+ * </ul>
+ *
+ * <p>Only call this method at the end of the intended usage of the library. Calling it earlier
Review Comment:
I think we should use some other wording than `library`. this should probably say that this should be only used with caution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3136484038
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -123,12 +132,7 @@ public static ExecutorService newWorkerPool(String namePrefix) {
}
/**
- * Creates a fixed-size thread pool that uses daemon threads. The pool is wrapped with {@link
- * MoreExecutors#getExitingExecutorService(ThreadPoolExecutor)}, which registers a shutdown hook
- * to ensure the pool terminates when the JVM exits. <b>Important:</b> Even if the pool is
- * explicitly shut down using {@link ExecutorService#shutdown()}, the shutdown hook is <i>not</i>
- * removed. This can lead to accumulation of shutdown hooks if this method is used repeatedly for
- * short-lived thread pools.
+ * Creates a fixed-size thread pool that uses daemon threads.
Review Comment:
unresolving as this hasn't been addressed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3136879658
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -123,12 +132,7 @@ public static ExecutorService newWorkerPool(String namePrefix) {
}
/**
- * Creates a fixed-size thread pool that uses daemon threads. The pool is wrapped with {@link
- * MoreExecutors#getExitingExecutorService(ThreadPoolExecutor)}, which registers a shutdown hook
- * to ensure the pool terminates when the JVM exits. <b>Important:</b> Even if the pool is
- * explicitly shut down using {@link ExecutorService#shutdown()}, the shutdown hook is <i>not</i>
- * removed. This can lead to accumulation of shutdown hooks if this method is used repeatedly for
- * short-lived thread pools.
+ * Creates a fixed-size thread pool that uses daemon threads.
Review Comment:
hmm I did add it _somewhere_. sorry - checking again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3136941239
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -102,12 +114,10 @@ private static class AuthRefreshPoolHolder {
}
/**
- * Creates a fixed-size thread pool that uses daemon threads. The pool is wrapped with {@link
- * MoreExecutors#getExitingExecutorService(ThreadPoolExecutor)}, which registers a shutdown hook
- * to ensure the pool terminates when the JVM exits. <b>Important:</b> Even if the pool is
- * explicitly shut down using {@link ExecutorService#shutdown()}, the shutdown hook is <i>not</i>
- * removed. This can lead to accumulation of shutdown hooks if this method is used repeatedly for
- * short-lived thread pools.
+ * Creates a fixed-size thread pool that uses daemon threads. The pool will be stopped by the
+ * shutdown hook to ensure the pool terminates when the JVM exits - unless `removeShutdownHook` is
Review Comment:
please use proper javadoc format when referring to methods here and in the other javadoc comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3136935693
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -123,12 +132,7 @@ public static ExecutorService newWorkerPool(String namePrefix) {
}
/**
- * Creates a fixed-size thread pool that uses daemon threads. The pool is wrapped with {@link
- * MoreExecutors#getExitingExecutorService(ThreadPoolExecutor)}, which registers a shutdown hook
- * to ensure the pool terminates when the JVM exits. <b>Important:</b> Even if the pool is
- * explicitly shut down using {@link ExecutorService#shutdown()}, the shutdown hook is <i>not</i>
- * removed. This can lead to accumulation of shutdown hooks if this method is used repeatedly for
- * short-lived thread pools.
+ * Creates a fixed-size thread pool that uses daemon threads.
Review Comment:
aha! got lost in diff :) updated this and also the main, non-deprecated function to use the same wording
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3136911244
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java:
##########
@@ -112,14 +109,8 @@ class S3OutputStream extends PositionOutputStream {
synchronized (S3OutputStream.class) {
if (executorService == null) {
executorService =
- MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor)
- Executors.newFixedThreadPool(
- s3FileIOProperties.multipartUploadThreads(),
- new ThreadFactoryBuilder()
- .setDaemon(true)
- .setNameFormat("iceberg-s3fileio-upload-%d")
- .build()));
+ ThreadPools.newExitingWorkerPool(
Review Comment:
reverted this commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4342379909
@rdblue
> Allowing them to be closed allows one class to break functionality for another.
Technically, nothing prevents users from shutting down the static pools already, for example:
```
ThreadPools.getWorkerPool().shutdown();
```
Today we mostly rely on reviews to prevent this, and in some cases it is not necessarily obvious. If the goal is to guarantee that users cannot shut these pools down, that could be addressed more directly (e.g., by wrapping the returned executor), but that safeguard does not exist today.
Changing all usages of `newExitingWorkerPool` / `newExitingScheduledPool` would also be fairly invasive. These thread pools are used across several `FileIO` implementations (e.g., `HadoopFileIO` → `newExitingWorkerPool`, `S3FileIO` / `GCSFileIO` → `newExitingScheduledPool`, `ADLSFileIO` → `getWorkerPool`).
Do you think a better long‑term direction would be to change `FileIO.initialize(Map<String, String> properties)` to something like:
```
FileIO.initialize(
Map<String, String> properties,
Function<Class<? extends FileIO>, ScheduledExecutorService> executorFactory)
```
and propagate this through the `FileIO` implementations?
In practice, `FileIO` instances are often created by catalogs, which suggest that we change the `Catalog.initialize(String name, Map<String, String> properties)` to something like:
```
Catalog.initialize(
String name,
Map<String, String> properties,
Function<Class<? extends FileIO>, ScheduledExecutorService> executorFactory)
```
and propagate this through the `Catalog` implementations and change the `CatalogUtil.loadCatalog(String impl, String catalogName, Map<String, String> properties, Object hadoopConf)` to allow setting the factory.
The FileIO implementations can't manage the lifecycle for the pools as they don't know if they should keep the shared resources open or they should close them too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4345082752
NOTE: updated the description again to add a section about the non-deterministic nature of the JVM shutdown hooks and how it prevents clean iceberg file commits when the application is exiting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2904834141
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +153,89 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ @VisibleForTesting
+ static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
Review Comment:
```suggestion
.newThread(
() -> {
shutdownHook = null;
shutdownThreadPools();
});
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2890304053
##########
core/src/test/java/org/apache/iceberg/util/TestThreadPools.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.junit.jupiter.api.Test;
+
+public class TestThreadPools {
+
+ @Test
+ public void testRemoveShutdownHook() {
+ try {
+ assertThat(ThreadPools.isShutdownHookRegistered()).isTrue();
+ ThreadPools.removeShutdownHook();
+ assertThat(ThreadPools.isShutdownHookRegistered()).isFalse();
+ } finally {
+ ThreadPools.initShutdownHook();
+ }
+ }
+
+ @Test
+ public void testThreadPoolManagerAddAndShutdown() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ ExecutorService testExecutor = Executors.newFixedThreadPool(2);
+
+ Duration timeout = Duration.ofSeconds(5);
+ manager.addThreadPool(testExecutor, timeout);
+
+ manager.shutdownAll();
+
+ assertThat(testExecutor.isShutdown()).isTrue();
+ }
+
+ @Test
+ public void testThreadPoolManagerMultipleShutdowns() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ ExecutorService executor1 = Executors.newFixedThreadPool(1);
+ ExecutorService executor2 = Executors.newFixedThreadPool(1);
+
+ Duration timeout = Duration.ofSeconds(5);
+ manager.addThreadPool(executor1, timeout);
+ manager.addThreadPool(executor2, timeout);
+
+ manager.shutdownAll();
+
+ assertThat(executor1.isShutdown()).isTrue();
+ assertThat(executor2.isShutdown()).isTrue();
+ }
+
+ @Test
+ public void testThreadPoolManagerShutdownNowCalled() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ final AtomicBoolean interrupted = new AtomicBoolean(false);
+ ExecutorService slowExecutor = Executors.newFixedThreadPool(1);
+
+ manager.addThreadPool(slowExecutor, Duration.ofMillis(50));
+
+ CountDownLatch threadStarted = new CountDownLatch(1);
+ CountDownLatch threadInterrupted = new CountDownLatch(1);
+
+ slowExecutor.submit(
+ () -> {
+ try {
+ threadStarted.countDown();
+ Thread.sleep(60_000);
Review Comment:
Should we add a comment that we're not actually waiting 60 seconds here?
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ synchronized void addThreadPool(ExecutorService service, Duration timeout) {
+ threadPoolsToShutdown.add(new ExecutorServiceWithTimeout(service, timeout));
+ }
+
+ /** Shut down all registered thread pools. */
+ synchronized void shutdownAll() {
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : threadPoolsToShutdown) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ threadPoolsToShutdown.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
Review Comment:
NIT newline after blocks (Iceberg style)
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ synchronized void addThreadPool(ExecutorService service, Duration timeout) {
+ threadPoolsToShutdown.add(new ExecutorServiceWithTimeout(service, timeout));
+ }
+
+ /** Shut down all registered thread pools. */
+ synchronized void shutdownAll() {
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : threadPoolsToShutdown) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ threadPoolsToShutdown.clear();
Review Comment:
NIT newline after blocks (Iceberg style)
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +152,86 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
+ if (shutdownHook != null) {
+ try {
+ Runtime.getRuntime().removeShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot remove the shutdown hook for thread pools clean up", e);
+ }
+ shutdownHook = null;
Review Comment:
NIT newline after blocks (Iceberg style)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2890881921
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +152,86 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ static synchronized void initShutdownHook() {
Review Comment:
Why is this package private? Why not private?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2894975123
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +152,86 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ static synchronized void initShutdownHook() {
Review Comment:
In this case use `@VisibleForTesting` showing that it should not be used anywhere else just in tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2904844979
##########
core/src/test/java/org/apache/iceberg/util/TestThreadPools.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.junit.jupiter.api.Test;
+
+public class TestThreadPools {
Review Comment:
Make the test class and methods package private
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2804808121
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
+ ExecutorService item;
+ Queue<ExecutorService> invoked = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ invoked.add(item);
+ }
+ while ((item = invoked.poll()) != null) {
+ try {
+ item.awaitTermination(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ }
+ }
+
+ /**
+ * Initialise a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>NOTE: it is normally started automatically, and does not need to be invoked manually, unless
+ * it has been stopped via the {@link #removeShutdownHook()}
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static void initShutdownHook() {
Review Comment:
true, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2827360407
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +166,85 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutdownStartedThreadPools() {
+ long startTime = System.nanoTime();
+ ExecutorService item;
+ Queue<ExecutorService> invoked = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ invoked.add(item);
+ }
+ while ((item = invoked.poll()) != null) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = SHUTDOWN_TIMEOUT_SECONDS * 1_000_000_000L - timeElapsed;
+ if (remainingTime > 0) {
Review Comment:
actually added an _else_ branch instead to invoke `shutdownNow` (timeout has expired)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4267928791
@nastra: Could you please take a look? This issue comes up from time-to-time and makes the restart logic for long running engines problematic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2878160945
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -148,9 +152,99 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* ensure the pool terminates when the JVM exits. This is suitable for long-lived thread pools
* that should be automatically cleaned up on JVM shutdown.
*/
- public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ public static synchronized ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(new ExecutorServiceWithTimeout(service, DEFAULT_SHUTDOWN_TIMEOUT));
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static synchronized void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : THREAD_POOLS_TO_SHUTDOWN) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ THREAD_POOLS_TO_SHUTDOWN.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
Review Comment:
i.e. the error I am trying to solve for our application is that on shutdown, iceberg library starts throwing all sorts of exceptions how all threadpools are already shut down. JVM does not allow to control the order of shutdown hooks, so it's either this change in iceberg or runtime magic within JVM... (at least I could only come up with these two options, happy to hear any more ideas)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2878160945
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -148,9 +152,99 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* ensure the pool terminates when the JVM exits. This is suitable for long-lived thread pools
* that should be automatically cleaned up on JVM shutdown.
*/
- public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ public static synchronized ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(new ExecutorServiceWithTimeout(service, DEFAULT_SHUTDOWN_TIMEOUT));
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static synchronized void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : THREAD_POOLS_TO_SHUTDOWN) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ THREAD_POOLS_TO_SHUTDOWN.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
Review Comment:
i.e. the error I am trying to solve for our application is that on shutdown, iceberg library starts throwing all sorts of exceptions how all threadpools are already shut down. JVM does not allow to control the order of shutdown hooks, so it's either this change in iceberg of runtime magic within JVM...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2878232437
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -148,9 +152,99 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* ensure the pool terminates when the JVM exits. This is suitable for long-lived thread pools
* that should be automatically cleaned up on JVM shutdown.
*/
- public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ public static synchronized ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(new ExecutorServiceWithTimeout(service, DEFAULT_SHUTDOWN_TIMEOUT));
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static synchronized void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : THREAD_POOLS_TO_SHUTDOWN) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ THREAD_POOLS_TO_SHUTDOWN.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
Review Comment:
to explain better. we write a new iceberg file periodically - and create a new one for the current window. if our application gets restarted (shutdown hooks get triggered due to SIGTERM), we want to write (commit / flush) the currently open window.
so we have a handler for the shutdown hook, that completes the writes for the iceberg library (and adds the new file to aws glue catalog). but if jvm has killed all the threadpools - we cannot.
we the idea is that in our use case, we DO NOT want iceberg library to use any shutdown hook. instead, we will unregister the default behaviour (opt out) and invoke the shutting down only when our application is ready to do it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2879298511
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -148,9 +152,99 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* ensure the pool terminates when the JVM exits. This is suitable for long-lived thread pools
* that should be automatically cleaned up on JVM shutdown.
*/
- public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ public static synchronized ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(new ExecutorServiceWithTimeout(service, DEFAULT_SHUTDOWN_TIMEOUT));
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static synchronized void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : THREAD_POOLS_TO_SHUTDOWN) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ THREAD_POOLS_TO_SHUTDOWN.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
Review Comment:
cool, thanks :)
yeah, as I mentioned in the referenced issue my use case can be seen as a very closely related one and we could go for the win-win setup - instead of me creating a new issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2879234267
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -148,9 +152,99 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* ensure the pool terminates when the JVM exits. This is suitable for long-lived thread pools
* that should be automatically cleaned up on JVM shutdown.
*/
- public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ public static synchronized ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(new ExecutorServiceWithTimeout(service, DEFAULT_SHUTDOWN_TIMEOUT));
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static synchronized void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : THREAD_POOLS_TO_SHUTDOWN) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ THREAD_POOLS_TO_SHUTDOWN.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
Review Comment:
That makes sense. It goes a bit beyond the original idea of #15031, but it makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2842869418
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +166,98 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ ExecutorService item;
+ Queue<ExecutorService> pendingShutdown = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ pendingShutdown.add(item);
+ }
+ while ((item = pendingShutdown.poll()) != null) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = SHUTDOWN_TIMEOUT.toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static void removeShutdownHook() {
+ if (shutdownHook != null) {
+ try {
+ Runtime.getRuntime().removeShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot remove the shutdown hook for thread pools clean up", e);
+ }
+ shutdownHook = null;
Review Comment:
happy to change this to atomic ref tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2877121992
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
Review Comment:
ok, I can make it synchronized too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4555684484
> hi. I think this is an important issue and the static shutdown hook based approach as implemented currently raises various problems that I tried to address (tactically) in the PR.
>
> having this problem (the family of problems) solved one way or another would definitely receive thanks from me, and I suspect other users who have similar issues - be it to be able to unload the iceberg exporter in dynamic environments, or make sure it has not stopped itself before the main application finishes exporting the files on shutdown.
The next community sync is on 3rd of June 10, 9:00 AM PT. You should add the topic to the [Iceberg Community Sync](https://docs.google.com/document/d/1YuGhUdukLP5gGiqCbk0A5_Wifqe2CZWgOd3TbhY3UQg) docs, and come to the meeting, so we can talk about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4814399975
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2878497196
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -148,9 +152,99 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* ensure the pool terminates when the JVM exits. This is suitable for long-lived thread pools
* that should be automatically cleaned up on JVM shutdown.
*/
- public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ public static synchronized ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(new ExecutorServiceWithTimeout(service, DEFAULT_SHUTDOWN_TIMEOUT));
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static synchronized void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : THREAD_POOLS_TO_SHUTDOWN) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ THREAD_POOLS_TO_SHUTDOWN.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
Review Comment:
@mxm please can you confirm you're ok with the above?
because the ability to keep the iceberg library working (threads running) while our application is already running a shutdown hook handler is crucial for us and the main reason why I am working on this PR (at the same time solving the other use case which asks to terminate threads w/o JVM exit)
ofc I would also be happy if you want to propose an alternative way to solve this problem, although I think having this method public solves it in an easy way and is compatible with the overall improvements
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2804346455
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
Review Comment:
I think this needs to be synchronized to work in a multi-threaded environment.
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
+ ExecutorService item;
+ Queue<ExecutorService> invoked = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ invoked.add(item);
+ }
+ while ((item = invoked.poll()) != null) {
+ try {
+ item.awaitTermination(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ }
+ }
+
+ /**
+ * Initialise a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>NOTE: it is normally started automatically, and does not need to be invoked manually, unless
+ * it has been stopped via the {@link #removeShutdownHook()}
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static void initShutdownHook() {
Review Comment:
```suggestion
private static void initShutdownHook() {
```
This should either be private or be `synchronized` to avoid adding the shutdown hook multiple times.
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
+ ExecutorService item;
+ Queue<ExecutorService> invoked = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ invoked.add(item);
+ }
+ while ((item = invoked.poll()) != null) {
+ try {
+ item.awaitTermination(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ }
+ }
+
+ /**
+ * Initialise a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>NOTE: it is normally started automatically, and does not need to be invoked manually, unless
+ * it has been stopped via the {@link #removeShutdownHook()}
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static void initShutdownHook() {
+ if (shutDownHook == null) {
+ shutDownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutDownStartedThreadPools();
+ }
+ });
+
+ try {
+ shutDownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ // OK if we can't set the name in this environment.
+ }
Review Comment:
Print a log message to warn the user? What about adding the shutdown hook itself? Do we need try/catch for that as well?
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -56,6 +62,14 @@ private ThreadPools() {}
public static final int AUTH_REFRESH_THREAD_POOL_SIZE =
SystemConfigs.AUTH_REFRESH_THREAD_POOL_SIZE.value();
+ private static final int SHUTDOWN_TIMEOUT_SECONDS = 120;
+
+ private static Thread shutDownHook = null;
Review Comment:
No need to initialize with null.
```suggestion
private static Thread shutDownHook;
```
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
+ ExecutorService item;
+ Queue<ExecutorService> invoked = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ invoked.add(item);
+ }
+ while ((item = invoked.poll()) != null) {
+ try {
+ item.awaitTermination(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
Review Comment:
I think we want to do that in a concurrent fashion. Waiting one by one would mean waiting for `N * SHUTDOWN_TIMEOUT_SECONDS` seconds of time. Probably we need to collect a list of futures and wait on their completion for max `SHUTDOWN_TIMEOUT_SECONDS`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2804803288
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
+ ExecutorService item;
+ Queue<ExecutorService> invoked = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ invoked.add(item);
+ }
+ while ((item = invoked.poll()) != null) {
+ try {
+ item.awaitTermination(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
Review Comment:
this is exactly what is being done :) the shutdown is invoked just above, this loop just waits for their completion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2805147989
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
+ ExecutorService item;
+ Queue<ExecutorService> invoked = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ invoked.add(item);
+ }
+ while ((item = invoked.poll()) != null) {
+ try {
+ item.awaitTermination(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ }
+ }
+
+ /**
+ * Initialise a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>NOTE: it is normally started automatically, and does not need to be invoked manually, unless
+ * it has been stopped via the {@link #removeShutdownHook()}
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static void initShutdownHook() {
+ if (shutDownHook == null) {
+ shutDownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutDownStartedThreadPools();
+ }
+ });
+
+ try {
+ shutDownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ // OK if we can't set the name in this environment.
+ }
Review Comment:
same with shutdown hook related security exceptions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4067120433
@rdblue, @RussellSpitzer: Any thoughts around this? I think this change is good to go, but I would value your input as it touches on a fundamental part of Iceberg.
I think it would be a good addition as I have seen many issues with stopping threadpools at the end of the JVM and this allows users to overwrite the default behavior and still doesn't change it for the ones not interested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3118207954
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +153,89 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Shuts down all thread pools registered via {@link #newExitingWorkerPool(String, int)} or {@link
+ * #newExitingScheduledPool(String, int, Duration)} and removes the JVM shutdown hook.
+ *
+ * <p>This method is useful for:
+ *
+ * <ul>
+ * <li>Preventing thread pool leaks in hot-reload environments
+ * <li>Managing graceful application shutdown when the application needs to handle its own
+ * shutdown hooks (e.g. to commit pending files before exiting)
+ * </ul>
+ *
+ * <p>Only call this method at the end of the intended usage of the library. Calling it earlier
+ * will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ @VisibleForTesting
+ static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ () -> {
+ shutdownHook = null;
+ shutdownThreadPools();
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
+ if (shutdownHook != null) {
+ try {
+ Runtime.getRuntime().removeShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot remove the shutdown hook for thread pools clean up", e);
+ }
+
+ shutdownHook = null;
+ }
+ }
+
+ /**
+ * Check if the shutdown hook has been registered.
+ *
+ * @return true if the shutdown hook is registered, false otherwise
+ */
+ @VisibleForTesting
+ static synchronized boolean isShutdownHookRegistered() {
Review Comment:
yes that was the intention. happy to remove the annotation though if you prefer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3136913122
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +153,89 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Shuts down all thread pools registered via {@link #newExitingWorkerPool(String, int)} or {@link
+ * #newExitingScheduledPool(String, int, Duration)} and removes the JVM shutdown hook.
+ *
+ * <p>This method is useful for:
+ *
+ * <ul>
+ * <li>Preventing thread pool leaks in hot-reload environments
+ * <li>Managing graceful application shutdown when the application needs to handle its own
+ * shutdown hooks (e.g. to commit pending files before exiting)
+ * </ul>
+ *
+ * <p>Only call this method at the end of the intended usage of the library. Calling it earlier
+ * will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ @VisibleForTesting
+ static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ () -> {
+ shutdownHook = null;
+ shutdownThreadPools();
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
+ if (shutdownHook != null) {
+ try {
+ Runtime.getRuntime().removeShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot remove the shutdown hook for thread pools clean up", e);
+ }
+
+ shutdownHook = null;
+ }
+ }
+
+ /**
+ * Check if the shutdown hook has been registered.
+ *
+ * @return true if the shutdown hook is registered, false otherwise
+ */
+ @VisibleForTesting
+ static synchronized boolean isShutdownHookRegistered() {
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2890913227
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ synchronized void addThreadPool(ExecutorService service, Duration timeout) {
+ threadPoolsToShutdown.add(new ExecutorServiceWithTimeout(service, timeout));
+ }
+
+ /** Shut down all registered thread pools. */
+ synchronized void shutdownAll() {
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : threadPoolsToShutdown) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ threadPoolsToShutdown.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
Review Comment:
I don't know where should be a newline here
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ synchronized void addThreadPool(ExecutorService service, Duration timeout) {
+ threadPoolsToShutdown.add(new ExecutorServiceWithTimeout(service, timeout));
+ }
+
+ /** Shut down all registered thread pools. */
+ synchronized void shutdownAll() {
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : threadPoolsToShutdown) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ threadPoolsToShutdown.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
Review Comment:
Shall we at least log?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2891041517
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
Review Comment:
hmm this is used in tests - so that they could access it to test..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2895696578
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +152,86 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ static synchronized void initShutdownHook() {
Review Comment:
Please add the `@VisibleForTesting` annotation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2827135929
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +166,85 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutdownStartedThreadPools() {
+ long startTime = System.nanoTime();
Review Comment:
the reason I chose nanoTime instead is that it is monotonic, and would not suffer big jumps (when system clock changes) - IIRC currentTimeMillis is related to clock and could cause issues if server clock is adjusted during the shutdown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2895005589
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ synchronized void addThreadPool(ExecutorService service, Duration timeout) {
+ threadPoolsToShutdown.add(new ExecutorServiceWithTimeout(service, timeout));
+ }
+
+ /** Shut down all registered thread pools. */
+ synchronized void shutdownAll() {
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : threadPoolsToShutdown) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ threadPoolsToShutdown.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
Review Comment:
For me the newline in 308 and 316 are unnecessary and even hurts my eyes a bit 😄
But these are not really important stuff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2895498687
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -42,8 +45,12 @@ private ThreadPools() {}
public static final String WORKER_THREAD_POOL_SIZE_PROP =
SystemConfigs.WORKER_THREAD_POOL_SIZE.propertyKey();
+ private static final Duration DEFAULT_SHUTDOWN_TIMEOUT = Duration.ofSeconds(120);
Review Comment:
Thanks for clarifying!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4351980799
Hi @rdblue,
Just to clarify:
> Next, I see newExistingScheduledPool has 2 outside uses, S3FileIO and GCSFileIO, and newExitingWorkerPool has 3 outside uses in HadoopFileIO, RESTMetricsReporter, and AuthSessionCache.
The problem covers all thread pools currently managed by MoreExecutors - including both worker pools here.
> 2. A global shutdown for all thread pools created by ThreadPools
A global shutdown exists already via MoreExecutors, just that it is deferred to the JVM shutdown hook and there is no control as to when exactly will it be invoked.
---
> I think those are both poor solutions. For the first one, if you don't want a threadpool that is configured like the static pools, the right answer is to use your own threadpool and manage its lifecycle <..>
I agree with this. If iceberg had thread pools injected via constructors (and without relying on the global JVM shutdown hooks) it would address the issue cleanly. At this point, however, I was looking for a pragmatic solution. We have an application that uses the iceberg library, and cannot commit/flush files on exit because iceberg has already killed its threads and is producing a lot of errors how the ThreadPools have been stopped.
Given that, and that there is an open issue related to thread stopping, my reasoning was that these can be combined to a single change to solve both problems - this is the resulting PR.
I suppose the question is - is this an acceptable interim solution until the threadpool (factory?) injection is implemented, or should this PR be closed and converted to an open issue (for the uncontrolled/non-deterministic shutdown behaviour)?
Do you have any thoughts as to when could the proper fix be expected?
Regards,
Sarunas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2805076480
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
+ ExecutorService item;
+ Queue<ExecutorService> invoked = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ invoked.add(item);
+ }
+ while ((item = invoked.poll()) != null) {
+ try {
+ item.awaitTermination(SHUTDOWN_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
Review Comment:
I've started going about this with CompletableFuture but it needs it's own ExecutorService and grew in complexity...
instead, I opted for a simpler implementation to ensure the same - please have a look (just checking how much wait time can subsequent calls have)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2891041517
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
Review Comment:
hmm this is used in tests - so that they could access it to test.. do you have alternative recommendation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2895888374
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ synchronized void addThreadPool(ExecutorService service, Duration timeout) {
+ threadPoolsToShutdown.add(new ExecutorServiceWithTimeout(service, timeout));
+ }
+
+ /** Shut down all registered thread pools. */
+ synchronized void shutdownAll() {
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : threadPoolsToShutdown) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ threadPoolsToShutdown.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
Review Comment:
happy to update but no longer sure what to update to :D imo if I add one to 316, one on 308 is only logical, but happy with any combination
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2891115976
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +261,74 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ synchronized void addThreadPool(ExecutorService service, Duration timeout) {
+ threadPoolsToShutdown.add(new ExecutorServiceWithTimeout(service, timeout));
+ }
+
+ /** Shut down all registered thread pools. */
+ synchronized void shutdownAll() {
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : threadPoolsToShutdown) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ threadPoolsToShutdown.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
Review Comment:
done, I think it _looks_ reasonable, but please check :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2895477429
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -98,16 +111,11 @@ public static ScheduledExecutorService authRefreshPool() {
private static class AuthRefreshPoolHolder {
private static final ScheduledExecutorService INSTANCE =
ThreadPools.newExitingScheduledPool(
- "auth-session-refresh", AUTH_REFRESH_THREAD_POOL_SIZE, Duration.ZERO);
+ "auth-session-refresh", AUTH_REFRESH_THREAD_POOL_SIZE, Duration.ofSeconds(10));
Review Comment:
this was added from the other PR for the same ticket - happy to revert.. tbh I will
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2904839890
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -180,15 +265,80 @@ public static ScheduledExecutorService newScheduledPool(String namePrefix, int p
* is suitable for long-lived thread pools that should be automatically cleaned up on JVM
* shutdown.
*/
- public static ScheduledExecutorService newExitingScheduledPool(
+ public static synchronized ScheduledExecutorService newExitingScheduledPool(
String namePrefix, int poolSize, Duration terminationTimeout) {
- return MoreExecutors.getExitingScheduledExecutorService(
- (ScheduledThreadPoolExecutor) newScheduledPool(namePrefix, poolSize),
- terminationTimeout.toMillis(),
- TimeUnit.MILLISECONDS);
+ ScheduledExecutorService service =
+ Executors.unconfigurableScheduledExecutorService(newScheduledPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, terminationTimeout);
+ return service;
}
private static ThreadFactory newDaemonThreadFactory(String namePrefix) {
return new ThreadFactoryBuilder().setDaemon(true).setNameFormat(namePrefix + "-%d").build();
}
+
+ /** Manages the lifecycle of thread pools that need to be shut down gracefully. */
+ @VisibleForTesting
+ static class ThreadPoolManager {
+ private final List<ExecutorServiceWithTimeout> threadPoolsToShutdown = Lists.newArrayList();
+
+ /**
+ * Add an executor service to the list of thread pools to be shut down.
+ *
+ * @param service the executor service to add
+ * @param timeout the timeout for shutdown operations
+ */
+ synchronized void addThreadPool(ExecutorService service, Duration timeout) {
+ threadPoolsToShutdown.add(new ExecutorServiceWithTimeout(service, timeout));
+ }
+
+ /** Shut down all registered thread pools. */
+ synchronized void shutdownAll() {
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+
+ for (ExecutorServiceWithTimeout item : threadPoolsToShutdown) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+
+ threadPoolsToShutdown.clear();
+
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException e) {
+ LOG.warn("Interrupted while shutting down, ignoring", e);
+ }
+
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+ }
+
+ private static class ExecutorServiceWithTimeout {
+ private ExecutorService service;
+ private Duration timeout;
+
+ private ExecutorServiceWithTimeout(ExecutorService service, Duration timeout) {
+ this.service = service;
+ this.timeout = timeout;
+ }
+
+ private ExecutorService getService() {
+ return service;
+ }
+
+ private Duration getTimeout() {
+ return timeout;
+ }
+ }
Review Comment:
```suggestion
private record ExecutorServiceWithTimeout(ExecutorService service, Duration timeout) {}
```
Make this a record and adjust the accessor usage accordingly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2904870365
##########
core/src/test/java/org/apache/iceberg/util/TestThreadPools.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.junit.jupiter.api.Test;
+
+public class TestThreadPools {
+
+ @Test
+ public void testRemoveShutdownHook() {
+ try {
+ assertThat(ThreadPools.isShutdownHookRegistered()).isTrue();
+ ThreadPools.removeShutdownHook();
+ assertThat(ThreadPools.isShutdownHookRegistered()).isFalse();
+ } finally {
+ ThreadPools.initShutdownHook();
+ }
+ }
+
+ @Test
+ public void testThreadPoolManagerAddAndShutdown() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ ExecutorService testExecutor = Executors.newFixedThreadPool(2);
+
+ Duration timeout = Duration.ofSeconds(5);
+ manager.addThreadPool(testExecutor, timeout);
+
+ manager.shutdownAll();
+
+ assertThat(testExecutor.isShutdown()).isTrue();
+ }
+
+ @Test
+ public void testThreadPoolManagerMultipleShutdowns() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ ExecutorService executor1 = Executors.newFixedThreadPool(1);
+ ExecutorService executor2 = Executors.newFixedThreadPool(1);
+
+ Duration timeout = Duration.ofSeconds(5);
+ manager.addThreadPool(executor1, timeout);
+ manager.addThreadPool(executor2, timeout);
+
+ manager.shutdownAll();
+
+ assertThat(executor1.isShutdown()).isTrue();
+ assertThat(executor2.isShutdown()).isTrue();
+ }
+
+ @Test
+ public void testThreadPoolManagerShutdownNowCalled() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
Review Comment:
Get rid of the `ThreadPoolManager` calls, use the `ThreadPools` API.
After that we can make `ThreadPoolManager` and all of the methods private
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2904851061
##########
core/src/test/java/org/apache/iceberg/util/TestThreadPools.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.junit.jupiter.api.Test;
+
+public class TestThreadPools {
+
+ @Test
+ public void testRemoveShutdownHook() {
+ try {
+ assertThat(ThreadPools.isShutdownHookRegistered()).isTrue();
+ ThreadPools.removeShutdownHook();
+ assertThat(ThreadPools.isShutdownHookRegistered()).isFalse();
+ } finally {
+ ThreadPools.initShutdownHook();
+ }
+ }
+
+ @Test
+ public void testThreadPoolManagerAddAndShutdown() throws Exception {
+ ThreadPools.ThreadPoolManager manager = new ThreadPools.ThreadPoolManager();
+
+ ExecutorService testExecutor = Executors.newFixedThreadPool(2);
+
+ Duration timeout = Duration.ofSeconds(5);
+ manager.addThreadPool(testExecutor, timeout);
+
+ manager.shutdownAll();
+
+ assertThat(testExecutor.isShutdown()).isTrue();
+ }
Review Comment:
We can always test through the `ThreadPools` API. It is better than relying in an implementation detail
```suggestion
@Test
void testNewExitingWorkerPoolShutdown() {
ExecutorService pool = ThreadPools.newExitingWorkerPool("test-exiting-pool", 2);
ThreadPools.shutdownThreadPools();
assertThat(pool.isShutdown()).isTrue();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2822842091
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
Review Comment:
I suppose it is fine if multiple components issue a shutdown. I was thinking to guard this by a lock, e.g. on the class file, but seems like it is fine to call this concurrently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2822676861
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +166,85 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutdownStartedThreadPools() {
+ long startTime = System.nanoTime();
Review Comment:
```suggestion
long startTimeMs = System.currentTimeMillis();
```
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +166,85 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutdownStartedThreadPools() {
+ long startTime = System.nanoTime();
+ ExecutorService item;
+ Queue<ExecutorService> invoked = new ArrayDeque<>();
Review Comment:
```suggestion
Queue<ExecutorService> pendingShutdown = new ArrayDeque<>();
```
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -56,6 +65,14 @@ private ThreadPools() {}
public static final int AUTH_REFRESH_THREAD_POOL_SIZE =
SystemConfigs.AUTH_REFRESH_THREAD_POOL_SIZE.value();
+ private static final int SHUTDOWN_TIMEOUT_SECONDS = 120;
Review Comment:
```suggestion
private static final Duration SHUTDOWN_TIMEOUT_SECONDS = Duration.ofSeconds(120);
```
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +166,85 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutdownStartedThreadPools() {
+ long startTime = System.nanoTime();
+ ExecutorService item;
+ Queue<ExecutorService> invoked = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ invoked.add(item);
+ }
+ while ((item = invoked.poll()) != null) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = SHUTDOWN_TIMEOUT_SECONDS * 1_000_000_000L - timeElapsed;
+ if (remainingTime > 0) {
Review Comment:
```suggestion
if (System.currentTimeMillis() - startTimeMs > SHUTDOWN_TIME_OUT.millis()) {
break;
}
```
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +166,85 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutdownStartedThreadPools() {
Review Comment:
```suggestion
public static void shutdownThreadPools() {
```
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +166,85 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutdownStartedThreadPools() {
Review Comment:
I think first thing we should remove the corresponding shutdown hook.
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
Review Comment:
It works, but it is quite nondeterministic how the shutdown is executed. With the current code, the shutdown timeout can be reset multiple times if there are multiple calls to shutdownThreadPools().
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +166,85 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutdownStartedThreadPools() {
+ long startTime = System.nanoTime();
+ ExecutorService item;
+ Queue<ExecutorService> invoked = new ArrayDeque<>();
+ while ((item = THREAD_POOLS_TO_SHUTDOWN.poll()) != null) {
+ item.shutdown();
+ invoked.add(item);
+ }
+ while ((item = invoked.poll()) != null) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = SHUTDOWN_TIMEOUT_SECONDS * 1_000_000_000L - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ item.awaitTermination(remainingTime, TimeUnit.NANOSECONDS);
Review Comment:
Should we also try this?
```suggestion
if (!item.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
item.snapshotNow();
}
```
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +166,85 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutdownStartedThreadPools() {
+ long startTime = System.nanoTime();
Review Comment:
Probably sufficient to use System.currentTimeMillis() here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2878121063
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -148,9 +152,99 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* ensure the pool terminates when the JVM exits. This is suitable for long-lived thread pools
* that should be automatically cleaned up on JVM shutdown.
*/
- public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ public static synchronized ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(new ExecutorServiceWithTimeout(service, DEFAULT_SHUTDOWN_TIMEOUT));
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ *
+ * <p>This method allows: (1) to stop thread pools manually, to avoid leaks in hot-reload
+ * environments; (2) opt-out of the standard shutdown mechanism to manage graceful service stops
+ * (and commit the last pending files, if the client application needs to react to shutdown hooks
+ * on its own).
+ *
+ * <p>Please only call this method at the end of the intended usage of the library, and NEVER
+ * before, as this method will stop thread pools required for normal library workflows.
+ */
+ public static synchronized void shutdownThreadPools() {
+ removeShutdownHook();
+ long startTime = System.nanoTime();
+ List<ExecutorServiceWithTimeout> pendingShutdown = Lists.newArrayList();
+ for (ExecutorServiceWithTimeout item : THREAD_POOLS_TO_SHUTDOWN) {
+ item.getService().shutdown();
+ pendingShutdown.add(item);
+ }
+ THREAD_POOLS_TO_SHUTDOWN.clear();
+ for (ExecutorServiceWithTimeout item : pendingShutdown) {
+ long timeElapsed = System.nanoTime() - startTime;
+ long remainingTime = item.getTimeout().toNanos() - timeElapsed;
+ if (remainingTime > 0) {
+ try {
+ if (!item.service.awaitTermination(remainingTime, TimeUnit.NANOSECONDS)) {
+ item.getService().shutdownNow();
+ }
+ } catch (InterruptedException ignored) {
+ // We're shutting down anyway, so just ignore.
+ }
+ } else {
+ item.getService().shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ private static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ new Runnable() {
+ @Override
+ public void run() {
+ shutdownHook = null;
+ shutdownThreadPools();
+ }
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
Review Comment:
Why is this method `public`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#issuecomment-3990192231
@mxm updated, please check.
I had missed `AuthRefreshPoolHolder` and indeed the whole scheduled executors side - which is now added as well.
also, what tests do you have in mind?
this is still all pretty much static code and invoking the shutdown in test affects a lot. I suppose one option would be to extract functions for the more complicated, non-obvious features, and test them in isolation - do you have any preferences?
I'm a bit wary of the nondeterministic state of this PR/task - esp with the new parallel PR out there :) would it be possible to have a think and set out a single list of changes pending before it can be considered complete/good enough?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r2804800382
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +163,75 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOLS_TO_SHUTDOWN.add(service);
+ return service;
+ }
+
+ /**
+ * Force manual shutdown of the thread pools created via the {@link #newExitingWorkerPool(String,
+ * int)}.
+ */
+ public static void shutDownStartedThreadPools() {
Review Comment:
the idea is that it uses `ConcurrentLinkedDeque` to safely drain the threadpools and shut them down, so should be safe to be called concurrently?
happy to add synchronized if you still think it is needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3115778011
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -123,12 +132,7 @@ public static ExecutorService newWorkerPool(String namePrefix) {
}
/**
- * Creates a fixed-size thread pool that uses daemon threads. The pool is wrapped with {@link
- * MoreExecutors#getExitingExecutorService(ThreadPoolExecutor)}, which registers a shutdown hook
- * to ensure the pool terminates when the JVM exits. <b>Important:</b> Even if the pool is
- * explicitly shut down using {@link ExecutorService#shutdown()}, the shutdown hook is <i>not</i>
- * removed. This can lead to accumulation of shutdown hooks if this method is used repeatedly for
- * short-lived thread pools.
+ * Creates a fixed-size thread pool that uses daemon threads.
Review Comment:
I think the javadoc should still mention when the pool terminates
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3118643394
##########
core/src/test/java/org/apache/iceberg/util/TestThreadPools.java:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.util;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Duration;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+@Timeout(value = 30, unit = TimeUnit.SECONDS)
+class TestThreadPools {
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "svalaskevicius (via GitHub)" <gi...@apache.org>.
svalaskevicius commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3118652771
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +153,89 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Shuts down all thread pools registered via {@link #newExitingWorkerPool(String, int)} or {@link
+ * #newExitingScheduledPool(String, int, Duration)} and removes the JVM shutdown hook.
+ *
+ * <p>This method is useful for:
+ *
+ * <ul>
+ * <li>Preventing thread pool leaks in hot-reload environments
+ * <li>Managing graceful application shutdown when the application needs to handle its own
+ * shutdown hooks (e.g. to commit pending files before exiting)
+ * </ul>
+ *
+ * <p>Only call this method at the end of the intended usage of the library. Calling it earlier
+ * will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ @VisibleForTesting
+ static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ () -> {
+ shutdownHook = null;
Review Comment:
I've removed this line and called the underlying method directly (synchronised)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, AWS: Allow stopping thread pools manually [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15312:
URL: https://github.com/apache/iceberg/pull/15312#discussion_r3136540411
##########
core/src/main/java/org/apache/iceberg/util/ThreadPools.java:
##########
@@ -149,8 +153,89 @@ public static ExecutorService newWorkerPool(String namePrefix, int poolSize) {
* that should be automatically cleaned up on JVM shutdown.
*/
public static ExecutorService newExitingWorkerPool(String namePrefix, int poolSize) {
- return MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) newFixedThreadPool(namePrefix, poolSize));
+ ExecutorService service =
+ Executors.unconfigurableExecutorService(newFixedThreadPool(namePrefix, poolSize));
+ THREAD_POOL_MANAGER.addThreadPool(service, DEFAULT_SHUTDOWN_TIMEOUT);
+ return service;
+ }
+
+ /**
+ * Shuts down all thread pools registered via {@link #newExitingWorkerPool(String, int)} or {@link
+ * #newExitingScheduledPool(String, int, Duration)} and removes the JVM shutdown hook.
+ *
+ * <p>This method is useful for:
+ *
+ * <ul>
+ * <li>Preventing thread pool leaks in hot-reload environments
+ * <li>Managing graceful application shutdown when the application needs to handle its own
+ * shutdown hooks (e.g. to commit pending files before exiting)
+ * </ul>
+ *
+ * <p>Only call this method at the end of the intended usage of the library. Calling it earlier
+ * will stop thread pools required for normal library workflows.
+ */
+ public static void shutdownThreadPools() {
+ THREAD_POOL_MANAGER.shutdownAll();
+ removeShutdownHook();
+ }
+
+ /**
+ * Initialize a shutdown hook to stop the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ */
+ @SuppressWarnings("ShutdownHook")
+ @VisibleForTesting
+ static synchronized void initShutdownHook() {
+ if (shutdownHook == null) {
+ shutdownHook =
+ Executors.defaultThreadFactory()
+ .newThread(
+ () -> {
+ shutdownHook = null;
+ shutdownThreadPools();
+ });
+
+ try {
+ shutdownHook.setName("DelayedShutdownHook-iceberg");
+ } catch (SecurityException e) {
+ LOG.warn("Cannot set thread name for the shutdown hook", e);
+ }
+
+ try {
+ Runtime.getRuntime().addShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot install a shutdown hook for thread pools clean up", e);
+ }
+ }
+ }
+
+ /**
+ * Stop the shutdown hook for the thread pools created via the {@link
+ * #newExitingWorkerPool(String, int)}.
+ *
+ * <p>Thread pools can still be stopped manually via the {@link #shutdownThreadPools()} method.
+ */
+ @SuppressWarnings("ShutdownHook")
+ public static synchronized void removeShutdownHook() {
+ if (shutdownHook != null) {
+ try {
+ Runtime.getRuntime().removeShutdownHook(shutdownHook);
+ } catch (SecurityException e) {
+ LOG.warn("Cannot remove the shutdown hook for thread pools clean up", e);
+ }
+
+ shutdownHook = null;
+ }
+ }
+
+ /**
+ * Check if the shutdown hook has been registered.
+ *
+ * @return true if the shutdown hook is registered, false otherwise
+ */
+ @VisibleForTesting
+ static synchronized boolean isShutdownHookRegistered() {
Review Comment:
I think I would probably remove this method and I feel like testing shutdown/termination and re-init is usually enough instead of actually verifying that the `shutdownHook` itself was initialized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org