You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "qiujun8023 (via GitHub)" <gi...@apache.org> on 2026/10/28 02:13:20 UTC
[I] ExecutorServiceReloader recreates executor on every trigger when jobExecutorThreadPoolSizeProviderType is not set, leaking shutdown hooks [shardingsphere-elasticjob]
qiujun8023 opened a new issue, #2561:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2561
## Bug Report
### Which version of ElasticJob did you use?
3.0.5 (`elasticjob-bootstrap`). The same code is still on `master`. 3.0.4 is not affected.
### Expected behavior
If `jobExecutorThreadPoolSizeProviderType` is not set, the default `CPU` provider is used. The documentation lists it as optional with default `CPU`. The job executor thread pool should be created once and reused for every trigger.
### Actual behavior
When `jobExecutorThreadPoolSizeProviderType` is not set, `ExecutorServiceReloader.reloadIfNecessary` shuts down the executor and creates a new one on every job trigger. This affects the Java API, the Spring Boot starter (unset property is `null`) and `@ElasticJobConfiguration` (`JobAnnotationBuilder` converts the default `""` to `null`).
`ElasticJobExecutorService.createExecutorService()` wraps each new pool with `MoreExecutors.getExitingExecutorService`, which registers a JVM shutdown hook. `shutdownNow()` does not remove that hook, so `ApplicationShutdownHooks` keeps every old `ThreadPoolExecutor`, its unstarted hook `Thread`, thread factories, queues and locks reachable. Heap usage therefore grows without bound.
In our production service, several jobs run every few seconds. After about 4.5 days, a heap histogram showed about 490,000 instances each of `ThreadPoolExecutor`, `Thread`, `BasicThreadFactory` and `MoreExecutors$Application$$Lambda`, while the process had only about 475 live threads. The heap was 93% full, and G1 concurrent marking plus GC threads used about half of the process CPU time. CPU usage kept rising linearly from about 30% to over 60%.
### Reason analyze (If you can)
`init` stores the **resolved** provider type, but `reloadIfNecessary` compares it with the **raw** configured value:
```java
public synchronized void reloadIfNecessary(final JobConfiguration jobConfig) {
if (jobExecutorThreadPoolSizeProviderType.equals(jobConfig.getJobExecutorThreadPoolSizeProviderType())) {
return;
}
executorService.shutdownNow();
init(jobConfig);
}
private void init(final JobConfiguration jobConfig) {
JobExecutorThreadPoolSizeProvider jobExecutorThreadPoolSizeProvider = TypedSPILoader.getService(JobExecutorThreadPoolSizeProvider.class, jobConfig.getJobExecutorThreadPoolSizeProviderType());
jobExecutorThreadPoolSizeProviderType = jobExecutorThreadPoolSizeProvider.getType();
...
}
```
When the configured value is `null`, `TypedSPILoader` falls back to the default provider, so the field becomes `"CPU"`. After that, `"CPU".equals(null)` is always false, so the executor is rebuilt on every trigger.
In 3.0.4, `ExecutorServiceReloadable` normalized empty values to the default before comparing, so this did not happen:
```java
String newJobExecutorServiceHandlerType = Strings.isNullOrEmpty(jobConfig.getJobExecutorServiceHandlerType())
? JobExecutorServiceHandlerFactory.DEFAULT_HANDLER
: jobConfig.getJobExecutorServiceHandlerType();
```
`ExecutorServiceReloaderTest` only covers explicitly configured types, so the unset case is not tested.
Possible fix: resolve the configured type the same way before comparing, for example with `TypedSPILoader.getService(...).getType()` or by treating `null` as the default type. Separately, `ElasticJobExecutorService` may be better off not using `MoreExecutors.getExitingExecutorService` for executors that can be replaced at runtime, because each replaced executor leaks a shutdown hook.
Workaround: set `jobExecutorThreadPoolSizeProviderType("CPU")` explicitly.
### Steps to reproduce the behavior.
1. Create a `JobConfiguration` without `jobExecutorThreadPoolSizeProviderType`.
2. Schedule it with `ScheduleJobBootstrap` using a short cron, for example every second.
3. Observe that the executor in `ExecutorServiceReloader` changes on every trigger, and that the number of `ThreadPoolExecutor` instances in `jmap -histo` / `jcmd <pid> GC.class_histogram` keeps growing.
### Example codes for reproduce this issue (such as a github link).
```java
@Test
void assertUnnecessaryToReloadWhenTypeIsNotSet() {
JobConfiguration jobConfig = JobConfiguration.newBuilder("job", 1).build();
try (ExecutorServiceReloader executorServiceReloader = new ExecutorServiceReloader(jobConfig)) {
ExecutorService expected = executorServiceReloader.getExecutorService();
executorServiceReloader.reloadIfNecessary(jobConfig);
// fails on 3.0.5: a new executor is created and the old one is shut down
assertThat(executorServiceReloader.getExecutorService(), is(expected));
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ExecutorServiceReloader recreates executor on every trigger when jobExecutorThreadPoolSizeProviderType is not set, leaking shutdown hooks [shardingsphere-elasticjob]
Posted by "linghengqian (via GitHub)" <gi...@apache.org>.
linghengqian commented on issue #2561:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2561#issuecomment-6514402395
- See https://github.com/apache/shardingsphere-elasticjob/pull/2562 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ExecutorServiceReloader recreates executor on every trigger when jobExecutorThreadPoolSizeProviderType is not set, leaking shutdown hooks [shardingsphere-elasticjob]
Posted by "adityaanikam (via GitHub)" <gi...@apache.org>.
adityaanikam commented on issue #2561:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2561#issuecomment-6476339130
I read the code on the default branch and the cause matches this report.
`ExecutorServiceReloader.init()` stores `jobExecutorThreadPoolSizeProvider.getType()`, which is the resolved type (`"CPU"` for the default). `reloadIfNecessary()` compares it with `jobConfig.getJobExecutorThreadPoolSizeProviderType()`. The `JobConfiguration` builder has no default for that field, so it is `null` when unset, `"CPU".equals(null)` is false, and `ElasticJobExecutor.execute()` rebuilds the executor on every trigger. `ElasticJobExecutorService.createExecutorService()` wraps each pool with `MoreExecutors.getExitingExecutorService`, so each rebuild also leaves a shutdown hook behind. The existing `assertUnnecessaryToReload` only covers the case where the type is set explicitly.
I would like to send a PR with a test for the unset case. One question first: `JobErrorHandlerReloader.reloadIfNecessary()` has the same shape (it compares `jobErrorHandler.getType()` with `jobConfig.getJobErrorHandlerType()`, which is also `null` when unset), so I expect it to recreate its handler on every trigger as well, which is cheaper but unnecessary. Do you want both fixed in one PR, or only the executor reloader here?
AI disclosure: an AI assistant (Claude) helped me investigate this and draft this comment. The code references were checked against the default branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org