You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sling.apache.org by "Nageshwari Elango (Jira)" <ji...@apache.org> on 2026/08/10 14:11:00 UTC
[jira] [Updated] (SLING-13298) Summary: OakMockSlingRepository.deactivate() shuts down executors before repository.shutdown(), causing RejectedExecutionException from Oak's BackgroundObserver during test teardown
[ https://issues.apache.org/jira/browse/SLING-13298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nageshwari Elango updated SLING-13298:
--------------------------------------
Description:
The source confirms everything in the bug report. Now I have all the details needed to give you a precise, ready-to-file JIRA ticket.
----
Here's the complete JIRA ticket you can file at [https://issues.apache.org/jira/projects/SLING/issues:]
----
Project: Sling Component: Testing Issue Type: Bug Priority: Minor _(tests still pass — annotation noise only)_ Affects Version: sling-mock-oak (current master) Labels: _(optional)_ {{{}sling-mock{}}}, {{{}oak{}}}, {{test-infrastructure}}
----
Summary: sling-mock-oak: OakMockSlingRepository.deactivate() terminates executors before repository.shutdown(), causing RejectedExecutionException from BackgroundObserver's completion handler during test teardown
----
Description:
* h2. Problem
In +OakMockSlingRepository.deactivate()+, all three executors are terminated *before* the Oak JCR repository is shut down:
{code:java}
// Current order in deactivate() — verified from source
executor.shutdownNow();
scheduledExecutor.shutdownNow();
shutdownExecutorService(repository, "scheduledExecutor"); // reflects into repository object
((JackrabbitRepository) repository).shutdown();
{code}
When +repository.shutdown()+ is subsequently called, it triggers +BackgroundObserver.close()+ on registered observers. +close()+ itself is straightforward — it clears the queue and adds a STOP sentinel:
{code:java}
// BackgroundObserver.close() — verified from source
public synchronized void close() {
queue.clear();
queue.add(STOP);
stopped = true;
}
{code}
The problem is not in +close()+ itself. The problem arises when a task that was *already in-flight* at shutdown time completes. Its completion handler is registered via +currentTask.onComplete(completionHandler)+, and that handler does:
{code:java}
// completionHandler.run() — verified from source
public void run() {
currentTask = new NotifyingFutureTask(task);
executor.execute(currentTask); // throws RejectedExecutionException — executor already dead
}
{code}
Because all executors have already been terminated via +shutdownNow()+ before +repository.shutdown()+ is called, every such chained task submission throws +RejectedExecutionException+:
{noformat}
Task org.apache.jackrabbit.oak.commons.concurrent.NotifyingFutureTask@1748c62
[Not completed, task = org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1@54b0813]
rejected from java.util.concurrent.ThreadPoolExecutor@2545113e
[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0]
{noformat}
h2. Impact
* Affects all unit tests using +ResourceResolverType.JCR_OAK+ — not specific to acs-aem-commons.
* All tests still pass. This is teardown noise only.
* GitHub Actions surfaces these exceptions as +failure+-level CI annotations, making them visually indistinguishable from real test failures. Contributors must manually inspect every annotation to confirm none are genuine.
* Observed consistently on *Windows JDK 17 and JDK 21*; intermittently on macOS/Linux (timing-dependent).
* Confirmed pre-existing across multiple runs — seen in [acs-aem-commons PR #3760|https://github.com/Adobe-Consulting-Services/acs-aem-commons/pull/3760] (merged June 2026), unrelated to any code change in that repo.
* No impact on production Oak or production AEM — exclusively in the test framework teardown path.
h2. Root Cause
+OakMockSlingRepository+ creates the executors and passes them into Oak via +Oak.with(executor)+ and +Jcr.with(executor)+. The owner should outlive its dependents during shutdown. Currently it does not: all three executors are force-killed before +repository.shutdown()+ is called, so any Oak background task still in-flight at that moment fails to chain its next step.
h2. Proposed Fix
Call +repository.shutdown()+ first, while the executors are still alive, then terminate them:
{code:java}
// Proposed order
// Shut down Oak JCR repository first so any in-flight BackgroundObserver
// tasks can complete their chaining before the executor is terminated
((JackrabbitRepository) repository).shutdown();
executor.shutdownNow();
scheduledExecutor.shutdownNow();
shutdownExecutorService(repository, "scheduledExecutor");
{code}
The existing intent — force-kill executors without waiting, since this is unit-test context — is preserved. Only the ordering changes.
*Points for maintainers to evaluate:*
* Whether reversing the order alone is sufficient, or whether a short +awaitTermination()+ between +repository.shutdown()+ and +executor.shutdownNow()+ is needed to allow in-flight chained tasks to complete cleanly before the executor is killed.
* Whether any Oak internal components beyond +BackgroundObserver+ depend on these executors during +repository.shutdown()+.
h2. Steps to Reproduce
# Add a dependency on +sling-mock-oak+ and write a test using +ResourceResolverType.JCR_OAK+.
# Run the test suite.
# Observe +RejectedExecutionException+ from +org.apache.jackrabbit.oak.spi.commit.BackgroundObserver+ in teardown output.
Example project: [acs-aem-commons|https://github.com/Adobe-Consulting-Services/acs-aem-commons] — [CI run #30953046297|https://github.com/enageshwari/acs-aem-commons/actions/runs/30953046297]
h2. Source References
* [OakMockSlingRepository.java|https://github.com/apache/sling-org-apache-sling-testing-sling-mock-oak/blob/master/src/main/java/org/apache/sling/testing/mock/sling/oak/OakMockSlingRepository.java]
* [BackgroundObserver.java|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-store-spi/src/main/java/org/apache/jackrabbit/oak/spi/commit/BackgroundObserver.java]
* Related (different issue): [SLING-12250|https://issues.apache.org/jira/browse/SLING-12250]
was:
The source confirms everything in the bug report. Now I have all the details needed to give you a precise, ready-to-file JIRA ticket.
----
Here's the complete JIRA ticket you can file at https://issues.apache.org/jira/projects/SLING/issues:
----
Project: Sling Component: Testing Issue Type: Bug Priority: Minor _(tests still pass — annotation noise only)_ Affects Version: sling-mock-oak (current master) Labels: _(optional)_ {{{}sling-mock{}}}, {{{}oak{}}}, {{test-infrastructure}}
----
Summary: {{OakMockSlingRepository.deactivate()}} shuts down executors before {{{}repository.shutdown(){}}}, causing {{RejectedExecutionException}} from Oak's {{BackgroundObserver}} during test teardown
----
Description:
Problem
In {{{}OakMockSlingRepository.deactivate(){}}}, the two {{ExecutorService}} instances are terminated _before_ the Oak JCR repository is shut down:
java
{{// Current order — brokenexecutor.shutdownNow();scheduledExecutor.shutdownNow();shutdownExecutorService(repository, "scheduledExecutor");((JackrabbitRepository) repository).shutdown(); // ← executor already dead}}
When {{repository.shutdown()}} is called, it triggers {{BackgroundObserver.close()}} on all registered Oak observers. {{BackgroundObserver}} uses a chained task pattern: each step that drains its internal queue calls {{executor.execute(nextTask)}} to schedule the next step. Because the executor is already terminated, every such submission throws {{{}RejectedExecutionException{}}}:
{{Task org.apache.jackrabbit.oak.commons.concurrent.NotifyingFutureTask@1748c62[Not completed, task = org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1@54b0813]rejected from java.util.concurrent.ThreadPoolExecutor@2545113e[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0]}}
Impact
* All unit tests using {{ResourceResolverType.JCR_OAK}} are affected (not just acs-aem-commons).
* All tests still pass — this is exclusively teardown noise.
* GitHub Actions surfaces these exceptions as {{{}failure{}}}-level CI annotations, making them visually indistinguishable from real test failures. Reviewers must manually inspect every annotation to confirm none are genuine.
* Observed consistently on Windows JDK 17 and JDK 21; intermittently on macOS/Linux (timing-dependent).
* Confirmed pre-existing across multiple CI runs in acs-aem-commons PR #3760 (merged June 2026), unrelated to any recent code change in that repo.
Root Cause
The dependency direction is: {{OakMockSlingRepository}} creates the executor and passes it _into_ Oak. The owner should outlive its dependents during shutdown. Currently it does not — the executors are killed first, leaving Oak's {{BackgroundObserver}} with a dead executor when it tries to drain its queue during {{{}repository.shutdown(){}}}.
Proposed Fix
Reverse the shutdown order — call {{repository.shutdown()}} first while the executors are still alive, then terminate the executors:
java
{{// Proposed order — correct// Shut down Oak JCR repository first so BackgroundObserver can complete// its shutdown sequence while the executor is still alive((JackrabbitRepository) repository).shutdown();executor.shutdownNow();scheduledExecutor.shutdownNow();shutdownExecutorService(repository, "scheduledExecutor");}}
The existing intent (force-kill executors without waiting, since this is a unit test) is fully preserved — only the ordering changes.
Notes for maintainers to evaluate:
* Whether reversing the order alone is sufficient, or whether an {{awaitTermination()}} call between {{repository.shutdown()}} and {{executor.shutdownNow()}} is needed to let the STOP sentinel complete cleanly (with the trade-off of added wait time).
* Whether any other Oak internal components beyond {{BackgroundObserver}} depend on the executor during {{{}repository.shutdown(){}}}.
Steps to Reproduce
# Add a dependency on {{sling-mock-oak}} with {{{}ResourceResolverType.JCR_OAK{}}}.
# Run any unit test that uses a Sling mock Oak-backed resource resolver.
# Observe {{RejectedExecutionException}} from {{BackgroundObserver}} in teardown output / CI annotations.
Example project where observed: acs-aem-commons — CI run #30953046297.
References
* {{OakMockSlingRepository.java}} — {{deactivate()}} method
* {{BackgroundObserver.java}} — Oak trunk
* Related (different issue): SLING-12250
> Summary: OakMockSlingRepository.deactivate() shuts down executors before repository.shutdown(), causing RejectedExecutionException from Oak's BackgroundObserver during test teardown
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SLING-13298
> URL: https://issues.apache.org/jira/browse/SLING-13298
> Project: Sling
> Issue Type: Bug
> Components: Apache Sling Testing Clients
> Reporter: Nageshwari Elango
> Priority: Minor
>
> The source confirms everything in the bug report. Now I have all the details needed to give you a precise, ready-to-file JIRA ticket.
> ----
> Here's the complete JIRA ticket you can file at [https://issues.apache.org/jira/projects/SLING/issues:]
> ----
> Project: Sling Component: Testing Issue Type: Bug Priority: Minor _(tests still pass — annotation noise only)_ Affects Version: sling-mock-oak (current master) Labels: _(optional)_ {{{}sling-mock{}}}, {{{}oak{}}}, {{test-infrastructure}}
> ----
> Summary: sling-mock-oak: OakMockSlingRepository.deactivate() terminates executors before repository.shutdown(), causing RejectedExecutionException from BackgroundObserver's completion handler during test teardown
> ----
> Description:
> * h2. Problem
> In +OakMockSlingRepository.deactivate()+, all three executors are terminated *before* the Oak JCR repository is shut down:
> {code:java}
> // Current order in deactivate() — verified from source
> executor.shutdownNow();
> scheduledExecutor.shutdownNow();
> shutdownExecutorService(repository, "scheduledExecutor"); // reflects into repository object
> ((JackrabbitRepository) repository).shutdown();
> {code}
> When +repository.shutdown()+ is subsequently called, it triggers +BackgroundObserver.close()+ on registered observers. +close()+ itself is straightforward — it clears the queue and adds a STOP sentinel:
> {code:java}
> // BackgroundObserver.close() — verified from source
> public synchronized void close() {
> queue.clear();
> queue.add(STOP);
> stopped = true;
> }
> {code}
> The problem is not in +close()+ itself. The problem arises when a task that was *already in-flight* at shutdown time completes. Its completion handler is registered via +currentTask.onComplete(completionHandler)+, and that handler does:
> {code:java}
> // completionHandler.run() — verified from source
> public void run() {
> currentTask = new NotifyingFutureTask(task);
> executor.execute(currentTask); // throws RejectedExecutionException — executor already dead
> }
> {code}
> Because all executors have already been terminated via +shutdownNow()+ before +repository.shutdown()+ is called, every such chained task submission throws +RejectedExecutionException+:
> {noformat}
> Task org.apache.jackrabbit.oak.commons.concurrent.NotifyingFutureTask@1748c62
> [Not completed, task = org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1@54b0813]
> rejected from java.util.concurrent.ThreadPoolExecutor@2545113e
> [Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0]
> {noformat}
> h2. Impact
> * Affects all unit tests using +ResourceResolverType.JCR_OAK+ — not specific to acs-aem-commons.
> * All tests still pass. This is teardown noise only.
> * GitHub Actions surfaces these exceptions as +failure+-level CI annotations, making them visually indistinguishable from real test failures. Contributors must manually inspect every annotation to confirm none are genuine.
> * Observed consistently on *Windows JDK 17 and JDK 21*; intermittently on macOS/Linux (timing-dependent).
> * Confirmed pre-existing across multiple runs — seen in [acs-aem-commons PR #3760|https://github.com/Adobe-Consulting-Services/acs-aem-commons/pull/3760] (merged June 2026), unrelated to any code change in that repo.
> * No impact on production Oak or production AEM — exclusively in the test framework teardown path.
> h2. Root Cause
> +OakMockSlingRepository+ creates the executors and passes them into Oak via +Oak.with(executor)+ and +Jcr.with(executor)+. The owner should outlive its dependents during shutdown. Currently it does not: all three executors are force-killed before +repository.shutdown()+ is called, so any Oak background task still in-flight at that moment fails to chain its next step.
> h2. Proposed Fix
> Call +repository.shutdown()+ first, while the executors are still alive, then terminate them:
> {code:java}
> // Proposed order
> // Shut down Oak JCR repository first so any in-flight BackgroundObserver
> // tasks can complete their chaining before the executor is terminated
> ((JackrabbitRepository) repository).shutdown();
> executor.shutdownNow();
> scheduledExecutor.shutdownNow();
> shutdownExecutorService(repository, "scheduledExecutor");
> {code}
> The existing intent — force-kill executors without waiting, since this is unit-test context — is preserved. Only the ordering changes.
> *Points for maintainers to evaluate:*
> * Whether reversing the order alone is sufficient, or whether a short +awaitTermination()+ between +repository.shutdown()+ and +executor.shutdownNow()+ is needed to allow in-flight chained tasks to complete cleanly before the executor is killed.
> * Whether any Oak internal components beyond +BackgroundObserver+ depend on these executors during +repository.shutdown()+.
> h2. Steps to Reproduce
> # Add a dependency on +sling-mock-oak+ and write a test using +ResourceResolverType.JCR_OAK+.
> # Run the test suite.
> # Observe +RejectedExecutionException+ from +org.apache.jackrabbit.oak.spi.commit.BackgroundObserver+ in teardown output.
> Example project: [acs-aem-commons|https://github.com/Adobe-Consulting-Services/acs-aem-commons] — [CI run #30953046297|https://github.com/enageshwari/acs-aem-commons/actions/runs/30953046297]
> h2. Source References
> * [OakMockSlingRepository.java|https://github.com/apache/sling-org-apache-sling-testing-sling-mock-oak/blob/master/src/main/java/org/apache/sling/testing/mock/sling/oak/OakMockSlingRepository.java]
> * [BackgroundObserver.java|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-store-spi/src/main/java/org/apache/jackrabbit/oak/spi/commit/BackgroundObserver.java]
> * Related (different issue): [SLING-12250|https://issues.apache.org/jira/browse/SLING-12250]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)