[jira] [Created] (OAK-5433) System Pacing Service

["Stefan Eissing (JIRA)" <ji...@apache.org>]

Stefan Eissing created OAK-5433:
-----------------------------------

             Summary: System Pacing Service
                 Key: OAK-5433
                 URL: https://issues.apache.org/jira/browse/OAK-5433
             Project: Jackrabbit Oak
          Issue Type: New Feature
          Components: core
            Reporter: Stefan Eissing
         Attachments: obs-pacing.diff

h3. tl;dr

By adding Pacing, suitable to the application {{oak}} is running in, a system will dynamically adapt the load to its own capabilities. This effectively, in tests, keeps the system stable and responsive under stress.

h3. The Situation
During experimental Lab tests on large clusters, it became clear that the a web system based using oak is challenged by fluctuating load in relation to its own capabilities. 

When the load increases "too much" it shows the following symptoms:
* event observation queues grow
* maintenance tasks (on master) take too long
* async tasks, triggered by requests, (e.g. workflows) accumulate
and eventually
* login sessions complain about freshness
* revisions diffs are old and no longer in caches
and sometimes
* database lease times out and oak-core shuts down

This problem can arise when outside requests increased, or when local maintenance tasks occupy resources, or when available CPU diminishes due to other processes or page faults or, or, or.

Unfortunately, whenever the system becomes overburdened, the secondary effects make the system even slower and, thus, more overburdened. This can end in a vicious circle, making the system total unresponsive. Eventual recovery is an option, not a guarantee.

h3. Pacing

By _Pacing_ I mean a system behaviour that tries to balance load in relation to capabilities. If the latter one drops, the load must be reduced until the system recovers. This is related to what the {{CommitRateLimiter}} wanted to achieve
by monitoring observation queues.

The design of the {{CommitRateLimiter}}  could be very efficient, if it only know _which_ commits to delay. But it does not know the application that oak runs in. I propose replacing the Limiter by a {{PacingService}} that can be provided by the application using oak. The service will get the data about the current commit, queue length and limits. Whatever else it does remains opaque. It may raise a proper exception to indicate that the commit shall fail. But mostly, it is expected to delay those commits that would negatively affect system stability.

h3. An Example

In a proof of concept, an AEM system was blasted with endless uploads on multiple connections in order to eventually overwhelm queues. The a pacing was patched into oak-core that delayed commits from servlet requests and from certain workflows for some milliseconds until the queue length shrank again. The pacing had a maximum wait time that would make the commit fail.

The pacer was configured to trigger at 75% of maximum queue length and the system was blasted with uploads again. In the tests:
# the max queue length stayed under 80%
# no upload did reach the maximum time, all succeeded

The system adapted the external load to its capabilities successfully. 



  








--
This message was sent by Atlassian JIRA
(v6.3.4#6332)