You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sling.apache.org by "Christian Schneider (Jira)" <ji...@apache.org> on 2026/10/22 17:43:00 UTC

[jira] [Created] (SLING-13396) Journal subscriber: batch install SPI for slow package builders

Christian Schneider created SLING-13396:
-------------------------------------------

             Summary: Journal subscriber: batch install SPI for slow package builders
                 Key: SLING-13396
                 URL: https://issues.apache.org/jira/browse/SLING-13396
             Project: Sling
          Issue Type: Improvement
          Components: Content Distribution
            Reporter: Christian Schneider


The journal subscriber installs one package at a time. When the install step of the package builder is slow, for example because it calls a remote system for every package, a backlog of many small and independent packages is drained slowly. A typical case is a content change that is automatically propagated to many locations, which produces one distribution package per path.

SLING-13021 proposes importing packages concurrently in the subscriber. That drops the ordering between packages (e.g. a delete of /a and an update of /a/b imported at the same time).

Proposal: keep the subscriber single-threaded, but let the package builder install several consecutive packages in one call. The builder knows the paths and can install only non-overlapping packages concurrently.

* New exported SPI {{org.apache.sling.distribution.journal.spi.BatchDistributionPackageBuilder}} (extends {{DistributionPackageBuilder}}) with {{installPackages(ResourceResolver, List<DistributionPackage>)}}. Packages are passed in journal order; the result must be the same as installing them one by one.
* New subscriber setting {{maxBatchSize}} (default 1 = current behaviour). Only applies to non-editable agents with {{packageHandling=Off}} and a package builder implementing the SPI; otherwise a warning is logged.
* The poller hands packages to an importer thread through a bounded queue. The importer takes the packages that are already queued (up to {{maxBatchSize}}), so single packages get no extra latency.
* Consecutive ADD/DELETE packages accepted by the precondition are batched. Any other package ends the batch and is processed as today.
* The offset of the last package is stored in the same commit as the batch. If the batch fails, no offset is stored and the packages are imported one by one with the existing retry and error queue handling (at-least-once, installs must be idempotent).
* New metrics {{imported_batch_size}} and {{imported_batch_duration}}.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)