[jira] [Created] (SLING-13259) OakDiscoveryService can block framework shutdown when Oak CommitQueue is slow during PropertyProvider unbind

["Daniel Iancu (Jira)" <ji...@apache.org>]

Daniel Iancu created SLING-13259:
------------------------------------

             Summary: OakDiscoveryService can block framework shutdown when Oak CommitQueue is slow during PropertyProvider unbind 
                 Key: SLING-13259
                 URL: https://issues.apache.org/jira/browse/SLING-13259
             Project: Sling
          Issue Type: Bug
          Components: Discovery
            Reporter: Daniel Iancu


h2. Description

h3. What happens

During OSGi framework shutdown, the Felix start-level thread can block for many minutes — potentially indefinitely — inside {{OakDiscoveryService.doUpdateProperties()}}, waiting for a JCR commit that never completes. The stuck thread prevents the framework from stopping any further bundles, so the JVM only exits when forcibly killed by the surrounding process manager. During that window the instance is half-dead but still visible to the rest of the cluster, delaying topology convergence on the surviving members.

h3. Stack trace

{noformat} FelixStartLevel at jdk.internal.misc.Unsafe.park (Native Method) - parking to wait for <0x…> (a java.util.concurrent.CountDownLatch
Entry.await (CommitQueue.java:346) at org.apache.jackrabbit.oak.plugins.document.CommitQueue.waitUntilHeadOfQueue (CommitQueue.java:285) … at org.apache.sling.jcr.resource.internal.helper.jcr.JcrResourceProvider.commit at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.commit at org.apache.sling.discovery.oak.OakDiscoveryService.doUpdateProperties (OakDiscoveryService.java:550) at org.apache.sling.discovery.oak.OakDiscoveryService.bindPropertyProviderInteral at org.apache.sling.discovery.oak.OakDiscoveryService.updatedPropertyProvider {noformat}

h3. Root cause

{{OakDiscoveryService}} declares {{PropertyProvider}} as a {{MULTIPLE}}, {{DYNAMIC}} reference. Every bind / unbind / update callback triggers a JCR write via {{doUpdateProperties()}} → {{ResourceResolver.commit()}} in order to keep the properties node at {{//properties}} in sync with the currently registered providers.

At shutdown, application bundles (higher start levels) typically stop before infrastructure bundles like {{discovery.oak}} (low start level). When an application bundle stops, its {{PropertyProvider}} services are unregistered, and Declarative Services invokes {{unbindPropertyProvider}} / {{updatedPropertyProvider}} on {{OakDiscoveryService}} synchronously on the Felix start-level thread. That callback then issues an Oak commit.

If the Oak {{CommitQueue}} is slow at that moment — a pending entry ahead of us, repository under load, backlog on the cluster leader — the commit parks on a {{CountDownLatch}} and does not return. The Felix start-level thread is now stuck inside a callback for a bundle it hasn't finished stopping. It cannot advance to stop any further bundles, and in particular it cannot reach {{discovery.oak}}'s own bundle to trigger {{OakDiscoveryService.deactivate()}}. Shutdown stalls entirely.

h3. Why the existing {{activated}} guard doesn't help

{{OakDiscoveryService.activated}} is set to {{false}} inside {{deactivate()}}, several lines into the method, under {{viewStateManagerLock}}. In the scenario above {{deactivate()}} is never reached, so {{activated}} remains {{true}} and every unbind callback proceeds into {{doUpdateProperties()}} and into {{commit()}}.

Even in cases where {{deactivate()}} does run, the {{activated}} flag is written under a different lock ({{viewStateManagerLock}}) from the one guarding bind/unbind ({{lock}}), so the two are not mutually observable without racing.

h3. Why the property update is unnecessary at shutdown

Once this instance is shutting down, rewriting the properties node has no value — no other cluster member will meaningfully act on the updated contents before this instance is removed from the topology anyway. The write is pure overhead, and during shutdown it is actively harmful because it holds the framework thread.

h2. Impact

Framework shutdown can hang for the full duration of the surrounding process manager's grace period before the JVM is forcibly killed.
The stalled instance remains visible in the discovery topology throughout, delaying leader re-election and topology convergence for the rest of the cluster.
Any operational tooling that assumes clean JVM exit (health checks, rolling restarts, orchestrated deployments) is affected.
h2. Steps to reproduce

Start a Sling instance with {{discovery.oak}} active and at least one {{PropertyProvider}} implementation registered by a bundle at a higher start level than {{discovery.oak}}.
Induce Oak {{CommitQueue}} slowness — for example by holding an open write session on another thread, or by loading the repository with concurrent writers that outpace the commit rate.
Trigger a clean framework shutdown (e.g. {{bin/stop}} or SIGTERM).
Observe: shutdown does not complete. A thread dump shows {{FelixStartLevel}} parked in {{CommitQueue$Entry.await}} under {{doUpdateProperties}}. The JVM only exits on forcible termination.
h2. Expected behaviour

Once the framework has begun shutting down, {{OakDiscoveryService}} should skip all property-update writes and return immediately from {{doUpdateProperties()}}. Framework shutdown should proceed at its normal pace regardless of the state of the Oak {{CommitQueue}}.

h2. Proposed fix (summary)

Introduce a {{volatile boolean deactivating}} flag that short-circuits {{doUpdateProperties()}} once shutdown has begun. The flag must be set before the first blocking commit can start, which means it cannot rely solely on {{@Deactivate}} being invoked — because in the observed scenario {{deactivate()}} is never reached. The trigger should be one of:

A {{FrameworkListener}} on {{FrameworkEvent.STOPPING}}, registered in {{@Activate}}, that sets the flag as soon as the framework begins shutting down. or
An inline check of {{bundleContext.getBundle(0).getState() >= Bundle.STOPPING}} at the top of {{doUpdateProperties()}}.
The flag is also set at the first line of {{deactivate()}} as a belt-and-braces measure for the case where {{discovery.oak}} is deactivated independently of a full framework stop.

The fix must not interrupt an already-in-progress commit — that risks corrupting the Oak session. It only prevents new commits from starting once shutdown has begun.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)