[jira] [Created] (SLING-9078) Keep skipped offsets in memory

["Timothée Maret (Jira)" <ji...@apache.org>]

Timothée Maret created SLING-9078:
-------------------------------------

             Summary: Keep skipped offsets in memory
                 Key: SLING-9078
                 URL: https://issues.apache.org/jira/browse/SLING-9078
             Project: Sling
          Issue Type: Improvement
          Components: Content Distribution
    Affects Versions: Content Distribution Journal Core 0.1.8
            Reporter: Timothée Maret
             Fix For: Content Distribution Journal Core 0.1.10


In SLING-8908 we introduced a mechanism to keep track of skipped packages such that in case of journal availability flip flop, the agents would still progress. The implementation was done by persisting the offset in the repository. In environments where many skipped packages will occur (e.g. large customer with multiple agents) then those commits will start to generate a non negligible load on the repository, consuming resources and creating bottlenecks.

 

A first mitigation to this issue has been added in SLING-8965, however for very large customers the mitigation will not be effective. Increasing the grouping of skipped packages (e.g. 1/100 instead of 1/10) reduce the load on the repository but may fail to ensure progress in case of flip/flop.

 

Instead of storing the skipped packages in the repository, we should keep them in memory. A new service that does not reference the {{JournalAvailable}} service could be added. The offsets would be consolidated on demand from the stored offsets and those stored in memory. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)