[jira] [Assigned] (SLING-13395) DistributionSubscriber imports the same package twice when it is restarted during a topology change

["Christian Schneider (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SLING-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Christian Schneider reassigned SLING-13395:
-------------------------------------------

    Assignee: Christian Schneider

> DistributionSubscriber imports the same package twice when it is restarted during a topology change
> ---------------------------------------------------------------------------------------------------
>
>                 Key: SLING-13395
>                 URL: https://issues.apache.org/jira/browse/SLING-13395
>             Project: Sling
>          Issue Type: Bug
>          Components: Content Distribution
>            Reporter: Christian Schneider
>            Assignee: Christian Schneider
>            Priority: Major
>             Fix For: Content Distribution Journal Core 0.6.0
>
>
> On a clustered instance, every topology change restarts all journal subscribers on the leader, even when the leader does not change. If a package import is still running at that moment, the restarted subscriber imports the same package again in parallel.
> How it happens:
> * OnlyOnLeader unregisters its service on every TOPOLOGY_CHANGING event and registers it again on TOPOLOGY_CHANGED. DistributionSubscriber has a static reference to OnlyOnLeader (SLING-12639), so it is deactivated and activated again on every topology change.
> * DistributionSubscriber.deactivate() does not interrupt the poller thread (SLING-9340) and does not wait for it, so an import that is already running continues.
> * The new subscriber instance starts at the last committed offset + 1, which is the offset still being imported by the old thread. The same package is then imported twice in parallel.
> * Both imports write the BookKeeper state for the same subscriber, so one of them can fail with an OakState0001 conflict on the subscriber store.
> The effect is significant when imports are slow (for example, an import that takes several minutes because of an expensive package builder) and the topology changes frequently (for example, instances joining and leaving the cluster). Several overlapping imports of the same offset were observed, each running for minutes.
> Proposed fix:
> * OnlyOnLeader: don't unregister on TOPOLOGY_CHANGING; only update the registration on TOPOLOGY_INIT/TOPOLOGY_CHANGED, and unregister only when the local instance is no longer the leader.
> * DistributionSubscriber.deactivate(): wake up a pending retry back-off and wait, bounded (30 s), for an import that is still running before returning. No locking between subscriber instances, to avoid hanging processing; a duplicate import remains possible when an import takes longer than the wait or when leadership really changes.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)