[couchdb-documentation] 01/01: 3.x fair share scheduler documetation

[va...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

vatamane pushed a commit to branch fairshare-docs-3x
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git

commit ef6ebac734b524f2678ebf8a8f3dfce9d839724e
Author: Nick Vatamaniuc <va...@apache.org>
AuthorDate: Sun Mar 14 21:24:53 2021 -0400

    3.x fair share scheduler documetation
    
    A short description on how the algorithm works along with the
    configuration sections.
    
    Main PR: https://github.com/apache/couchdb/pull/3364
---
 src/config/replicator.rst      | 49 ++++++++++++++++++++++++++++++++++++++++++
 src/replication/replicator.rst | 45 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 94 insertions(+)

diff --git a/src/config/replicator.rst b/src/config/replicator.rst
index 9e78b59..18d3722 100644
--- a/src/config/replicator.rst
+++ b/src/config/replicator.rst
@@ -249,3 +249,52 @@ Replicator Database Configuration
 
         .. note::
              In version 2.2, the session plugin is considered experimental and is not enabled by default.
+
+    .. config:option:: usage_coeff
+
+        .. versionadded:: 3.2.0
+
+        Usage coefficient decays historic fair share usage every
+        scheduling cycle. The value must be between 0.0 and 1.0. Lower
+        values will ensure historic usage decays quicker and higher
+        values means it will be remembered longer::
+
+            [replicator]
+            usage_coeff = 0.5
+
+    .. config:option:: priority_coeff
+
+        .. versionadded:: 3.2.0
+
+       Priority coefficient decays all the job priorities such that they slowly
+       drift towards the front of the run queue. This coefficient defines a maximum
+       time window over which this algorithm would operate. For example, if this
+       value is too small (0.1), after a few cycles quite a few jobs would end up at
+       priority 0, and would render this algorithm useless. The default value of
+       0.98 is picked such that if a job ran for one scheduler cycle, then didn't
+       get to run for 7 hours, it would still have priority > 0. 7 hours was picked
+       as it was close enought to 8 hours which is the default maximum error backoff
+       interval::
+
+            [replicator]
+            priority_coeff = 0.98
+
+.. _config/replicator.shares:
+
+Fair Share Replicator Share Allocation
+======================================
+
+.. config:section:: replicator.shares :: Per-Database Fair Share Allocation
+
+    .. config:option:: $replicator_db
+
+        .. versionadded:: 3.2.0
+
+        Fair share configuration section. More shares result in a
+        higher chance that jobs from that db get to run. The default
+        value is 100, minimum is 1 and maximum is 1000. The
+        configuration may be set even if the database does not exit::
+
+            [replicator.shares]
+            _replicator_db = 100
+            $another/_replicator_db = 100
diff --git a/src/replication/replicator.rst b/src/replication/replicator.rst
index de53930..0da0ac8 100644
--- a/src/replication/replicator.rst
+++ b/src/replication/replicator.rst
@@ -21,6 +21,11 @@ Replicator Database
    anymore. There are new replication job states and new API endpoints
    ``_scheduler/jobs`` and ``_scheduler/docs``.
 
+.. versionchanged:: 3.2.0 Fair share scheduling was
+   introduced. Multiple ``_replicator`` databases get an equal chance
+   (configurable) of running their jobs. Previously replication jobs
+   were scheduled without any regard of their originating database.
+
 The ``_replicator`` database works like any other in CouchDB, but
 documents added to it will trigger replications. Create (``PUT`` or
 ``POST``) a document to start replication. ``DELETE`` a replication
@@ -539,6 +544,46 @@ After this operation, replication pulling from server X will be stopped
 and the replications in the ``_replicator`` database (pulling from
 servers A and B) will continue.
 
+Fair Share Job Scheduling
+=========================
+
+When multiple ``_replicator`` databases are used, and the total number
+of jobs on any node is greater than ``max_jobs``, replication jobs
+will be scheduled such that each of the ``_replicator`` databases by
+default get an equal chance of running their jobs.
+
+This is accomplished by assigning a number of "shares" to each
+``_replicator`` database and then automatically adjusting the
+proportion of running jobs to match each database's proportion of
+shares. By default, each ``_replicator`` database is assigned 100
+shares. It is possible to alter the share assignments for each
+individual ``_replicator`` database in the :ref:`[replicator.shares]
+<config/replicator.shares>` configuration section.
+
+The fair share behavior is perhaps easier described with a sets of
+examples. Each example assumes the default of ``max_jobs = 500``, and
+two replicator dbs: ``_replicator`` and ``another/_replicator``.
+
+Example 1: If ``_replicator`` has 1000 jobs and
+``another/_replicator`` has 10, the scheduler will run about 490 jobs
+from ``_replicator`` and 10 jobs from ``another/_replicator``.
+
+Example 2: If ``_replicator`` has 200 jobs and ``another/_replicator``
+also has 200 jobs, all 400 jobs will get to run at the sum of all the
+jobs is less than ``max_jobs`` limit.
+
+Example 3: If both replication databases have 1000 jobs each, the
+scheduler will run about 250 jobs from each database on average.
+
+Example 4: If both replication databases have 1000 jobs each, but
+``_replicator`` was assigned 400 shares, then on average the scheduler
+would run about 400 jobs from ``_replicator`` and 100 jobs from
+``_another/replicator``.
+
+The proportions described in the examples are approximate and might
+oscillate a bit and also take anywhere from a tens of minutes to hours
+converge.
+
 Replicating the replicator database
 ===================================