You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@couchdb.apache.org by wo...@apache.org on 2018/07/23 23:57:32 UTC
[couchdb-documentation] branch 2.2.0-release-notes updated (9a4e4de
-> 2ab4aa9)
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a change to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git.
from 9a4e4de wip: leave note about recommended config
new 764ee3f revisit sharding documentation
new b23729f typo
new 3f89590 respond to feedback
new 505d4aa fix proxy auth docs, goes with https://github.com/apache/couchdb/pull/1436
new 2e555af Add troubleshooting information for FIPS mode and workaround (#1171)
new d81803d Fix trailing whitespace
new f38189a Fix typo
new c5db2ad Fix line length and phrasing
new 350b16c update sharding documentation for clarity, detail
new 976c0fe respond to feedback
new d350072 Configure cluster Erlang port range using vm.args
new 2e55191 put 2.x-relevant documentation before 1.x-relevant documentation
new 3d0dd3a feat: remove proxy docs, as they don’t work in 2.x
new 869e692 respond to additional feedback
new d005d14 add description of how quorum sizes are calculated
new f8bc900 fix linter
new 3efe764 lint and ref and multiline labels
new 2ab4aa9 [WIP] Partial 2.2 release notes
The 18 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
src/api/server/authn.rst | 4 +-
src/cluster/setup.rst | 29 +-
src/cluster/sharding.rst | 698 +++++++++++++++++++++++++++-------------
src/config/http.rst | 138 ++++----
src/config/index.rst | 1 -
src/config/proxying.rst | 94 ------
src/install/troubleshooting.rst | 28 ++
src/whatsnew/2.2.rst | 148 ++++++++-
8 files changed, 719 insertions(+), 421 deletions(-)
delete mode 100644 src/config/proxying.rst
[couchdb-documentation] 06/18: Fix trailing whitespace
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit d81803d9631c3c18c120c1e2b3b61c72cdd8413b
Author: rokek <82...@users.noreply.github.com>
AuthorDate: Mon Jul 16 16:58:56 2018 -0400
Fix trailing whitespace
---
src/install/troubleshooting.rst | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/install/troubleshooting.rst b/src/install/troubleshooting.rst
index 5d17dd4..b3758c6 100644
--- a/src/install/troubleshooting.rst
+++ b/src/install/troubleshooting.rst
@@ -279,12 +279,12 @@ replication to exclude only those documents.
FIPS mode
---------
-Operating systems can be configured to disallow the use of OpenSSL MD5 hash
-functions in order to prevent use of MD5 for cryptographic purposes. CouchDB
-makes use of MD5 hashes for verifying the integrity of data (and not for
+Operating systems can be configured to disallow the use of OpenSSL MD5 hash
+functions in order to prevent use of MD5 for cryptographic purposes. CouchDB
+makes use of MD5 hashes for verifying the integrity of data (and not for
cryptography) and will not run without the ability to use MD5 hashes.
-The message below indicates that the operating system is running in "FIPS mode,"
+The message below indicates that the operating system is running in "FIPS mode,"
which among other restrictions does not allow the use of OpenSSL's MD5 funtions:
.. code-block:: text
@@ -294,13 +294,13 @@ which among other restrictions does not allow the use of OpenSSL's MD5 funtions:
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
Aborted
-A workaround for this is provided with the ``--erlang-md5`` compile flag. Use of
-the flag results in CouchDB substituting the OpenSSL MD5 function calls with
+A workaround for this is provided with the ``--erlang-md5`` compile flag. Use of
+the flag results in CouchDB substituting the OpenSSL MD5 function calls with
equivalent calls to Erlang's built-in library ``erlang:md5.`` NOTE: there may be
a performance penalty associated with this workaround.
-Because CouchDB does not make use of MD5 hashes for cryptographic purposes, this
-workaround does not defeat the purpose of "FIPS mode," provided that the system
+Because CouchDB does not make use of MD5 hashes for cryptographic purposes, this
+workaround does not defeat the purpose of "FIPS mode," provided that the system
owner is aware of and consents to its use.
macOS Known Issues
[couchdb-documentation] 08/18: Fix line length and phrasing
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit c5db2ad4ade660c100a2c2ba24c2d037ebc57649
Author: rokek <82...@users.noreply.github.com>
AuthorDate: Tue Jul 17 10:19:52 2018 -0400
Fix line length and phrasing
---
src/install/troubleshooting.rst | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/install/troubleshooting.rst b/src/install/troubleshooting.rst
index f7933e4..ae6d6a6 100644
--- a/src/install/troubleshooting.rst
+++ b/src/install/troubleshooting.rst
@@ -285,7 +285,8 @@ makes use of MD5 hashes for verifying the integrity of data (and not for
cryptography) and will not run without the ability to use MD5 hashes.
The message below indicates that the operating system is running in "FIPS mode,"
-which among other restrictions does not allow the use of OpenSSL's MD5 functions:
+which, among other restrictions, does not allow the use of OpenSSL's
+MD5 functions:
.. code-block:: text
[couchdb-documentation] 02/18: typo
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit b23729fc2a70badbb4eaf4569e0b9c254dee4c04
Author: Diana Thayer <ga...@gmail.com>
AuthorDate: Fri Apr 6 14:51:24 2018 -0700
typo
---
src/cluster/sharding.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/cluster/sharding.rst b/src/cluster/sharding.rst
index ee536bb..d86b0f6 100644
--- a/src/cluster/sharding.rst
+++ b/src/cluster/sharding.rst
@@ -248,7 +248,7 @@ cluster!**
To add a shard to a node, add entries like this to the database
metadata's ``changelog`` attribute:
-.. code:: json1
+.. code:: json
[
"add",
[couchdb-documentation] 05/18: Add troubleshooting information for
FIPS mode and workaround (#1171)
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit 2e555aff9c49508581806a1ec002c40132a8ffce
Author: rokek <82...@users.noreply.github.com>
AuthorDate: Mon Jul 16 16:37:02 2018 -0400
Add troubleshooting information for FIPS mode and workaround (#1171)
Fix formatting for troubleshooting FIPS mode error
Fix formatting for troubleshooting FIPS mode error
Fix formatting for troubleshooting FIPS mode error
Fix formatting for troubleshooting FIPS mode error
Fix formatting for troubleshooting FIPS mode error
---
src/install/troubleshooting.rst | 27 +++++++++++++++++++++++++++
1 file changed, 27 insertions(+)
diff --git a/src/install/troubleshooting.rst b/src/install/troubleshooting.rst
index 408d607..5d17dd4 100644
--- a/src/install/troubleshooting.rst
+++ b/src/install/troubleshooting.rst
@@ -276,6 +276,33 @@ the relevant CouchDB and then compact prior to replicating.
Alternatively, if the number of documents impacted is small, use filtered
replication to exclude only those documents.
+FIPS mode
+---------
+
+Operating systems can be configured to disallow the use of OpenSSL MD5 hash
+functions in order to prevent use of MD5 for cryptographic purposes. CouchDB
+makes use of MD5 hashes for verifying the integrity of data (and not for
+cryptography) and will not run without the ability to use MD5 hashes.
+
+The message below indicates that the operating system is running in "FIPS mode,"
+which among other restrictions does not allow the use of OpenSSL's MD5 funtions:
+
+.. code-block:: text
+
+ md5_dgst.c(82): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
+ [os_mon] memory supervisor port (memsup): Erlang has closed
+ [os_mon] cpu supervisor port (cpu_sup): Erlang has closed
+ Aborted
+
+A workaround for this is provided with the ``--erlang-md5`` compile flag. Use of
+the flag results in CouchDB substituting the OpenSSL MD5 function calls with
+equivalent calls to Erlang's built-in library ``erlang:md5.`` NOTE: there may be
+a performance penalty associated with this workaround.
+
+Because CouchDB does not make use of MD5 hashes for cryptographic purposes, this
+workaround does not defeat the purpose of "FIPS mode," provided that the system
+owner is aware of and consents to its use.
+
macOS Known Issues
====================
undefined error, exit_status 134
[couchdb-documentation] 04/18: fix proxy auth docs,
goes with https://github.com/apache/couchdb/pull/1436
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit 505d4aadd4008141445e7f739d496ed9bbe73a28
Author: Jan Lehnardt <ja...@apache.org>
AuthorDate: Fri Jul 13 17:27:14 2018 +0200
fix proxy auth docs, goes with https://github.com/apache/couchdb/pull/1436
---
src/api/server/authn.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/api/server/authn.rst b/src/api/server/authn.rst
index 040b571..e33a78d 100644
--- a/src/api/server/authn.rst
+++ b/src/api/server/authn.rst
@@ -274,13 +274,13 @@ Proxy Authentication
.. note::
To use this authentication method make sure that the
- ``{couch_httpd_auth, proxy_authentication_handler}`` value in added to the
+ ``{chttpd_auth, proxy_authentication_handler}`` value in added to the
list of the active :config:option:`chttpd/authentication_handlers`:
.. code-block:: ini
[chttpd]
- authentication_handlers = {chttpd_auth, cookie_authentication_handler}, {couch_httpd_auth, proxy_authentication_handler}, {chttpd_auth, default_authentication_handler}
+ authentication_handlers = {chttpd_auth, cookie_authentication_handler}, {chttpd_auth, proxy_authentication_handler}, {chttpd_auth, default_authentication_handler}
`Proxy authentication` is very useful in case your application already uses
some external authentication service and you don't want to duplicate users and
[couchdb-documentation] 07/18: Fix typo
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit f38189afabab0a4a5118341b1f052f966c1c2353
Author: rokek <82...@users.noreply.github.com>
AuthorDate: Mon Jul 16 17:08:46 2018 -0400
Fix typo
---
src/install/troubleshooting.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/install/troubleshooting.rst b/src/install/troubleshooting.rst
index b3758c6..f7933e4 100644
--- a/src/install/troubleshooting.rst
+++ b/src/install/troubleshooting.rst
@@ -285,7 +285,7 @@ makes use of MD5 hashes for verifying the integrity of data (and not for
cryptography) and will not run without the ability to use MD5 hashes.
The message below indicates that the operating system is running in "FIPS mode,"
-which among other restrictions does not allow the use of OpenSSL's MD5 funtions:
+which among other restrictions does not allow the use of OpenSSL's MD5 functions:
.. code-block:: text
[couchdb-documentation] 14/18: respond to additional feedback
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit 869e69296c521b6d393d850e5f6e9cacc02f8cd2
Author: Diana Thayer <ga...@gmail.com>
AuthorDate: Fri Jul 20 08:20:51 2018 -0700
respond to additional feedback
---
src/cluster/sharding.rst | 110 ++++++++++++++++++++++++++++++-----------------
1 file changed, 71 insertions(+), 39 deletions(-)
diff --git a/src/cluster/sharding.rst b/src/cluster/sharding.rst
index 7f92fef..51b1630 100644
--- a/src/cluster/sharding.rst
+++ b/src/cluster/sharding.rst
@@ -52,8 +52,8 @@ receive 6 shards. We recommend in the general case that the number of
nodes in your cluster should be a multiple of ``n``, so that shards are
distributed evenly.
-CouchDB nodes have a ``etc/default.ini`` file with a section named
-``[cluster]`` which looks like this:
+CouchDB nodes have a ``etc/local.ini`` file with a section named
+`cluster <../config/cluster.html>`__ which looks like this:
::
@@ -68,7 +68,7 @@ example:
.. code:: bash
- $ curl -X PUT "$COUCH_URL/database-name?q=4&n=2"
+ $ curl -X PUT "$COUCH_URL:5984/database-name?q=4&n=2"
That creates a database that is split into 4 shards and 2 replicas,
yielding 8 shard replicas distributed throughout the cluster.
@@ -89,6 +89,10 @@ once a `quorum
<https://en.wikipedia.org/wiki/Quorum_(distributed_computing)>`__ of
database nodes have responded; 2, by default.
+.. note::
+ Each node in a cluster can be a coordinating node for any one
+ request. There are no special roles for nodes inside the cluster.
+
The size of the required quorum can be configured at request time by
setting the ``r`` parameter for document and view reads, and the ``w``
parameter for document writes. For example, here is a request that
@@ -125,23 +129,32 @@ from server metrics that database/shard layout is non-optimal and you
have some "hot spots" that need resolving.
Consider a three-node cluster with q=8 and n=3. Each database has 24
-shards, distributed across the three nodes. If you add a fourth node to
-the cluster, CouchDB will not redistribute existing database shards to
-it. This leads to unbalanced load, as the new node will only host shards
-for databases created after it joined the cluster. To balance the
-distribution of shards from existing databases, they must be moved
-manually.
+shards, distributed across the three nodes. If you :ref:`add a fourth
+node <cluster/nodes/add>` to the cluster, CouchDB will not redistribute
+existing database shards to it. This leads to unbalanced load, as the
+new node will only host shards for databases created after it joined the
+cluster. To balance the distribution of shards from existing databases,
+they must be moved manually.
Moving shards between nodes in a cluster involves the following steps:
-0. Ensure the target node has joined the cluster.
-1. Copy the shard(s) and any secondary index shard(s) onto the target node.
-2. Set the target node to maintenance mode.
-3. Update cluster metadata to reflect the new target shard(s).
-4. Monitor internal replication to ensure up-to-date shard(s).
-5. Clear the target node's maintenance mode.
-6. Update cluster metadata again to remove the source shard(s)
-7. Remove the shard file(s) and secondary index file(s) from the source node.
+0. :ref:`Ensure the target node has joined the cluster
+ <cluster/nodes/add>`.
+1. :ref:`Copy the shard(s) and any secondary index shard(s) onto the
+ target node <cluster/sharding/copying>`.
+2. :ref:`Set the target node to maintenance mode <cluster/sharding/mm>`.
+3. :ref:`Update cluster metadata to reflect the new target shard(s)
+ <cluster/sharding/add-shard>`.
+4. :ref:`Monitor internal replication to ensure up-to-date shard(s)
+ <cluster/sharding/verify>`.
+5. :ref:`Clear the target node's maintenance mode
+ <cluster/sharding/mm-2>`.
+6. :ref:`Update cluster metadata again to remove the source shard(s)
+ <cluster/sharding/remove-shard>`
+7. :ref:`Remove the shard file(s) and secondary index file(s) from the
+ source node <cluster/sharding/remove-shard-files>`.
+
+.. _cluster/sharding/copying:
Copying shard files
~~~~~~~~~~~~~~~~~~~
@@ -149,12 +162,10 @@ Copying shard files
.. note::
Technically, copying database and secondary index
shards is optional. If you proceed to the next step without
- performing
- this data copy, CouchDB will use internal replication to populate
- the
- newly added shard replicas. However, this process can be very slow,
- especially on a busy cluster—which is why we recommend performing this
- manual data copy first.
+ performing this data copy, CouchDB will use internal replication
+ to populate the newly added shard replicas. However, copying files
+ is faster than internal replication, especially on a busy cluster,
+ which is why we recommend performing this manual data copy first.
Shard files live in the ``data/shards`` directory of your CouchDB
install. Within those subdirectories are the shard files themselves. For
@@ -172,17 +183,6 @@ files:
data/shards/c0000000-dfffffff/abc.1529362187.couch
data/shards/e0000000-ffffffff/abc.1529362187.couch
-Since they are just files, you can use ``cp``, ``rsync``,
-``scp`` or any other command to copy them from one node to another. For
-example:
-
-.. code:: bash
-
- # one one machine
- $ mkdir -p data/shards/<range>
- # on the other
- $ scp <couch-dir>/data/shards/<range>/<database>.<datecode>.couch <node>:<couch-dir>/data/shards/<range>/
-
Secondary indexes (including JavaScript views, Erlang views and Mango
indexes) are also sharded, and their shards should be moved to save the
new node the effort of rebuilding the view. View shards live in
@@ -200,10 +200,32 @@ new node the effort of rebuilding the view. View shards live in
data/.shards/c0000000-dfffffff/_replicator.1518451591_design/mrview/3e823c2a4383ac0c18d4e574135a5b08.view
...
+Since they are files, you can use ``cp``, ``rsync``,
+``scp`` or other file-copying command to copy them from one node to
+another. For example:
+
+.. code:: bash
+
+ # one one machine
+ $ mkdir -p data/.shards/<range>
+ $ mkdir -p data/shards/<range>
+ # on the other
+ $ scp <couch-dir>/data/.shards/<range>/<database>.<datecode>* \
+ <node>:<couch-dir>/data/.shards/<range>/
+ $ scp <couch-dir>/data/shards/<range>/<database>.<datecode>.couch \
+ <node>:<couch-dir>/data/shards/<range>/
+
+.. note::
+ Remember to move view files before database files! If a view index
+ is ahead of its database, the database will rebuild it from
+ scratch.
+
+.. _cluster/sharding/mm:
+
Set the target node to ``true`` maintenance mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Before telling CouchDB about the new shards on the node in question, it
+Before telling CouchDB about these new shards on the node, the node
must be put into maintenance mode. Maintenance mode instructs CouchDB to
return a ``404 Not Found`` response on the ``/_up`` endpoint, and
ensures it does not participate in normal interactive clustered requests
@@ -245,12 +267,14 @@ Then, verify that the node is in maintenance mode by performing a ``GET
Finally, check that your load balancer has removed the node from the
pool of available backend nodes.
-Updating cluster metadata to reflect the move
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. _cluster/sharding/add-shard:
+
+Updating cluster metadata to reflect the new target shard(s)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Now we need to tell CouchDB that the target node (which must already be
-joined to the cluster) should be hosting shard replicas for a given
-database.
+:ref:`joined to the cluster <cluster/nodes/add>`) should be hosting
+shard replicas for a given database.
To update the cluster metadata, use the special ``/_dbs`` database,
which is an internal CouchDB database that maps databases to shards and
@@ -375,6 +399,8 @@ Now you can ``PUT`` this new metadata:
$ curl -X PUT http://localhost:5986/_dbs/{name} -d '{...}'
+.. _cluster/sharding/verify:
+
Monitor internal replication to ensure up-to-date shard(s)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -391,6 +417,8 @@ Once this metric has returned to the baseline from before you wrote the
document, or is ``0``, the shard replica is ready to serve data and we
can bring the node out of maintenance mode.
+.. _cluster/sharding/mm-2:
+
Clear the target node's maintenance mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -404,6 +432,8 @@ Verify that the node is not in maintenance mode by performing a ``GET
Finally, check that your load balancer has returned the node to the pool
of available backend nodes.
+.. _cluster/sharding/remove-shard:
+
Update cluster metadata again to remove the source shard
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -413,6 +443,8 @@ the ``["remove", <range>, <source-shard>]`` entry to the end of the
changelog as well as modifying both the ``by_node`` and ``by_range`` sections of
the database metadata document.
+.. _cluster/sharding/remove-shard-files:
+
Remove the shard and secondary index files from the source node
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[couchdb-documentation] 12/18: put 2.x-relevant documentation
before 1.x-relevant documentation
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit 2e551915fe80d9ccaf10ed53ca42ccf1906fa8c0
Author: Jan Lehnardt <ja...@apache.org>
AuthorDate: Thu Jul 19 09:12:34 2018 +0200
put 2.x-relevant documentation before 1.x-relevant documentation
---
src/config/http.rst | 138 ++++++++++++++++++++++++++--------------------------
1 file changed, 69 insertions(+), 69 deletions(-)
diff --git a/src/config/http.rst b/src/config/http.rst
index 2d82d1c..80c0cfb 100644
--- a/src/config/http.rst
+++ b/src/config/http.rst
@@ -22,6 +22,75 @@ CouchDB HTTP Server
HTTP Server Options
===================
+.. config:section:: chttpd :: Clustered HTTP Server Options
+
+.. note::
+ In CouchDB 2.x, the `chttpd` section refers to the standard, clustered
+ port. All use of CouchDB, aside from a few specific maintenance tasks as
+ described in this documentation, should be performed over this port.
+
+ Defines the IP address by which the clustered port is available::
+
+ [chttpd]
+ bind_address = 127.0.0.1
+
+ To let CouchDB listen any available IP address, use `0.0.0.0`::
+
+ [chttpd]
+ bind_address = 0.0.0.0
+
+ For IPv6 support you need to set `::1` if you want to let CouchDB
+ listen correctly::
+
+ [chttpd]
+ bind_address = ::1
+
+ or `::` for any available::
+
+ [chttpd]
+ bind_address = ::
+
+ .. config:option:: port :: Listen port
+
+ Defines the port number to listen::
+
+ [chttpd]
+ port = 5984
+
+ To let CouchDB use any free port, set this option to `0`::
+
+ [chttpd]
+ port = 0
+
+ .. config:option:: prefer_minimal :: Sends minimal set of headers
+
+ If a request has the header `"Prefer": "return=minimal"`, CouchDB
+ will only send the headers that are listed for the `prefer_minimal`
+ configuration.::
+
+ [chttpd]
+ prefer_minimal = Cache-Control, Content-Length, Content-Range, Content-Type, ETag, Server, Transfer-Encoding, Vary
+
+ .. warning::
+ Removing the Server header from the settings will mean that
+ the CouchDB server header is replaced with the
+ MochiWeb server header.
+
+ .. config:option:: authentication_handlers :: Authentication handlers
+
+ List of authentication handlers used by CouchDB. You may
+ extend them via third-party plugins or remove some of them if you won't
+ let users to use one of provided methods::
+
+ [chttpd]
+ authentication_handlers = {chttpd_auth, cookie_authentication_handler}, {chttpd_auth, default_authentication_handler}
+
+ - ``{chttpd_auth, cookie_authentication_handler}``: used for Cookie auth;
+ - ``{couch_httpd_auth, proxy_authentication_handler}``: used for Proxy auth;
+ - ``{chttpd_auth, default_authentication_handler}``: used for Basic auth;
+ - ``{couch_httpd_auth, null_authentication_handler}``: disables auth.
+ Everlasting `Admin Party`!
+
.. config:section:: httpd :: HTTP Server Options
.. warning::
@@ -222,75 +291,6 @@ HTTP Server Options
upgrade, it is advisable to review the usage of these configuration
settings.
-.. config:section:: chttpd :: Clustered HTTP Server Options
-
-.. note::
- In CouchDB 2.x, the `chttpd` section refers to the standard, clustered
- port. All use of CouchDB, aside from a few specific maintenance tasks as
- described in this documentation, should be performed over this port.
-
- Defines the IP address by which the clustered port is available::
-
- [chttpd]
- bind_address = 127.0.0.1
-
- To let CouchDB listen any available IP address, use `0.0.0.0`::
-
- [chttpd]
- bind_address = 0.0.0.0
-
- For IPv6 support you need to set `::1` if you want to let CouchDB
- listen correctly::
-
- [chttpd]
- bind_address = ::1
-
- or `::` for any available::
-
- [chttpd]
- bind_address = ::
-
- .. config:option:: port :: Listen port
-
- Defines the port number to listen::
-
- [chttpd]
- port = 5984
-
- To let CouchDB use any free port, set this option to `0`::
-
- [chttpd]
- port = 0
-
- .. config:option:: prefer_minimal :: Sends minimal set of headers
-
- If a request has the header `"Prefer": "return=minimal"`, CouchDB
- will only send the headers that are listed for the `prefer_minimal`
- configuration.::
-
- [chttpd]
- prefer_minimal = Cache-Control, Content-Length, Content-Range, Content-Type, ETag, Server, Transfer-Encoding, Vary
-
- .. warning::
- Removing the Server header from the settings will mean that
- the CouchDB server header is replaced with the
- MochiWeb server header.
-
- .. config:option:: authentication_handlers :: Authentication handlers
-
- List of authentication handlers used by CouchDB. You may
- extend them via third-party plugins or remove some of them if you won't
- let users to use one of provided methods::
-
- [chttpd]
- authentication_handlers = {chttpd_auth, cookie_authentication_handler}, {chttpd_auth, default_authentication_handler}
-
- - ``{chttpd_auth, cookie_authentication_handler}``: used for Cookie auth;
- - ``{couch_httpd_auth, proxy_authentication_handler}``: used for Proxy auth;
- - ``{chttpd_auth, default_authentication_handler}``: used for Basic auth;
- - ``{couch_httpd_auth, null_authentication_handler}``: disables auth.
- Everlasting `Admin Party`!
-
.. _config/ssl:
Secure Socket Level Options
[couchdb-documentation] 03/18: respond to feedback
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit 3f89590d77ab536fe1d2f1195abbf7defc35a992
Author: Diana Thayer <ga...@gmail.com>
AuthorDate: Fri Apr 13 09:47:51 2018 -0700
respond to feedback
---
src/cluster/sharding.rst | 115 ++++++++++++++++++-----------------------------
1 file changed, 44 insertions(+), 71 deletions(-)
diff --git a/src/cluster/sharding.rst b/src/cluster/sharding.rst
index d86b0f6..71d6c35 100644
--- a/src/cluster/sharding.rst
+++ b/src/cluster/sharding.rst
@@ -12,9 +12,9 @@
.. _cluster/sharding:
-========
-Sharding
-========
+================
+Shard Management
+================
.. _cluster/sharding/scaling-out:
@@ -38,11 +38,14 @@ level, or on a per-database basis. The relevant parameters are *q* and
*n*.
*q* is the number of database shards to maintain. *n* is the number of
-copies of each document to distribute. With q=8, the database is split
-into 8 shards. With n=3, the cluster distributes three replicas of each
-shard. Altogether, that's 24 shards for a single database. In a default
-3-node cluster, each node would receive 8 shards. In a 4-node cluster,
-each node would receive 6 shards.
+copies of each document to distribute. The default value for n is 3,
+and for q is 8. With q=8, the database is split into 8 shards. With
+n=3, the cluster distributes three replicas of each shard. Altogether,
+that's 24 shards for a single database. In a default 3-node cluster,
+each node would receive 8 shards. In a 4-node cluster, each node would
+receive 6 shards. We recommend in the general case that the number of
+nodes in your cluster should be a multiple of n, so that shards are
+distributed evenly.
CouchDB nodes have a ``etc/default.ini`` file with a section named
``[cluster]`` which looks like this:
@@ -70,13 +73,14 @@ Quorum
When a CouchDB cluster serves reads and writes, it proxies the request
to nodes with relevant shards and responds once enough nodes have
-responded to establish
-`quorum <https://en.wikipedia.org/wiki/Quorum_(distributed_computing)>`__.
-The size of the required quorum can be configured at request time by
-setting the ``r`` parameter for document and view reads, and the ``w``
-parameter for document writes. For example, here is a request that
-specifies that at least two nodes must respond in order to establish
-quorum:
+responded to obtain a `quorum
+<https://en.wikipedia.org/wiki/Quorum_(distributed_computing)>`__. By
+default, CouchDB requires a quorum of ``(n+1)/2`` nodes, or 2 in a
+default cluster. The size of the required quorum can be configured at
+request time by setting the ``r`` parameter for document and view
+reads, and the ``w`` parameter for document writes. For example, here
+is a request that specifies that at least two nodes must respond in
+order to establish a quorum:
.. code:: bash
@@ -86,59 +90,15 @@ Here is a similar example for writing a document:
.. code:: bash
- $ curl -X PUT "$COUCH_URL:5984/{docId}?w=2" -d '{}'
+ $ curl -X PUT "http://xxx.xxx.xxx.xxx:5984/{docId}?w=2" -d '{...}'
Setting ``r`` or ``w`` to be equal to ``n`` (the number of replicas)
means you will only receive a response once all nodes with relevant
-shards have responded, however even this does not guarantee `ACIDic
-consistency <https://en.wikipedia.org/wiki/ACID#Consistency>`__. Setting
-``r`` or ``w`` to 1 means you will receive a response after only one
-relevant node has responded.
-
-Adding a node
--------------
-
-To add a node to a cluster, first you must have the additional node
-running somewhere. Make note of the address it binds to, like
-``127.0.0.1``, then ``PUT`` an empty document to the ``/_node``
-endpoint:
-
-.. code:: bash
-
- $ curl -X PUT "$COUCH_URL:5984/_node/{name}@{address}" -d '{}'
-
-This will add the node to the cluster. Existing shards will not be moved
-or re-balanced in response to the addition, but future operations will
-distribute shards to the new node.
-
-Now when you GET the ``/_membership`` endpoint, you will see the new
-node.
-
-Removing a node
----------------
-
-To remove a node from the cluster, you must first acquire the ``_rev``
-value for the document that signifies its existence:
-
-.. code:: bash
-
- $ curl "$COUCH_URL:5984/_node/{name}@{address}"
- {"_id":"{name}@{address}","_rev":"{rev}"}
-
-Using that ``_rev``, you can delete the node using the ``/_node``
-endpoint:
-
-.. code:: bash
-
- $ curl -X DELETE "$COUCH_URL:5984/_node/{name}@{address}?rev={rev}"
-
-.. raw:: html
-
- <div class="alert alert-warning">
-
-**Note**: Before you remove a node, make sure to
-`move its shards <#moving-a-shard>`__
-or else they will be lost.
+shards have responded or timed out, however even this does not
+guarantee `ACIDic consistency
+<https://en.wikipedia.org/wiki/ACID#Consistency>`__. Setting ``r`` or
+``w`` to 1 means you will receive a response after only one relevant
+node has responded.
Moving a shard
--------------
@@ -167,14 +127,27 @@ example:
Views are also sharded, and their shards should be moved to save the new
node the effort of rebuilding the view. View shards live in
-``data/.shards``.
+``data/.shards``. For example:
+
+::
+
+ data/.shards
+ data/.shards/e0000000-ffffffff/_replicator.1518451591_design
+ data/.shards/e0000000-ffffffff/_replicator.1518451591_design/mrview
+ data/.shards/e0000000-ffffffff/_replicator.1518451591_design/mrview/3e823c2a4383ac0c18d4e574135a5b08.view
+ data/.shards/c0000000-dfffffff
+ data/.shards/c0000000-dfffffff/_replicator.1518451591_design
+ data/.shards/c0000000-dfffffff/_replicator.1518451591_design/mrview
+ data/.shards/c0000000-dfffffff/_replicator.1518451591_design/mrview/3e823c2a4383ac0c18d4e574135a5b08.view
+ ...
Updating cluster metadata
~~~~~~~~~~~~~~~~~~~~~~~~~
-To update the cluster metadata, use the special node-specific ``/_dbs``
-database, accessible via a node's private port, usually at port 5986.
-First, retrieve the database's current metadata:
+To update the cluster metadata, use the special node-specific
+``/_dbs`` database, accessible via a node's private port, usually at
+port 5986. This port is only available on the localhost interface for
+security purposes. So first, retrieve the database's current metadata:
.. code:: bash
@@ -256,7 +229,7 @@ metadata's ``changelog`` attribute:
"{name}@{address}"
]
-*Note*: You can remove a node by specifying 'remove' instead of 'add'.
+*Note*: You can remove a shard from a node by specifying 'remove' instead of 'add'.
Once you have figured out the new changelog entries, you will need to
update the ``by_node`` and ``by_range`` to reflect who is storing what
@@ -269,7 +242,7 @@ adds shards to a second node called ``node2``:
.. code:: json
{
- "_id": "small",
+ "_id": "{name}",
"_rev": "1-5e2d10c29c70d3869fb7a1fd3a827a64",
"shard_suffix": [
46,
[couchdb-documentation] 09/18: update sharding documentation for
clarity, detail
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit 350b16c4d6ae8c8589bd87e5d4af4af1b96ecaab
Author: Diana Thayer <ga...@gmail.com>
AuthorDate: Tue Jul 17 17:34:32 2018 -0700
update sharding documentation for clarity, detail
---
src/cluster/sharding.rst | 467 ++++++++++++++++++++++++++++-------------------
1 file changed, 277 insertions(+), 190 deletions(-)
diff --git a/src/cluster/sharding.rst b/src/cluster/sharding.rst
index 71d6c35..d4786c0 100644
--- a/src/cluster/sharding.rst
+++ b/src/cluster/sharding.rst
@@ -21,14 +21,15 @@ Shard Management
Introduction
------------
-A
-`shard <https://en.wikipedia.org/wiki/Shard_(database_architecture)>`__
-is a horizontal partition of data in a database. Partitioning data into
-shards and distributing copies of each shard (called "replicas") to
-different nodes in a cluster gives the data greater durability against
-node loss. CouchDB clusters automatically shard and distribute data
-among nodes, but modifying cluster membership and customizing shard
-behavior must be done manually.
+A `shard
+<https://en.wikipedia.org/wiki/Shard_(database_architecture)>`__ is a
+horizontal partition of data in a database. Partitioning data into
+shards and distributing copies of each shard (called "shard replicas" or
+just "replicas") to different nodes in a cluster gives the data greater
+durability against node loss. CouchDB clusters automatically shard
+databases and distribute the subsections of documents that compose each
+shard among nodes. Modifying cluster membership and sharding behavior
+must be done manually.
Shards and Replicas
~~~~~~~~~~~~~~~~~~~
@@ -69,14 +70,22 @@ That creates a database that is split into 4 shards and 2 replicas,
yielding 8 shards distributed throughout the cluster.
Quorum
-------
-
-When a CouchDB cluster serves reads and writes, it proxies the request
-to nodes with relevant shards and responds once enough nodes have
-responded to obtain a `quorum
-<https://en.wikipedia.org/wiki/Quorum_(distributed_computing)>`__. By
-default, CouchDB requires a quorum of ``(n+1)/2`` nodes, or 2 in a
-default cluster. The size of the required quorum can be configured at
+~~~~~~
+
+Depending on the size of the cluster, the number of shards per database,
+and the number of shard replicas, not every node may have access to
+every shard, but every node knows where all the replicas of each shard
+can be found through CouchDB's internal shard map.
+
+Each request that comes in to a CouchDB cluster is handled by any one
+random coordinating node. This coordinating node proxies the request to
+the other nodes that have the relevant data, which may or may not
+include itself. The coordinating node sends a response to the client
+once a `quorum
+<https://en.wikipedia.org/wiki/Quorum_(distributed_computing)>`__ of
+database nodes have responded; 2, by default.
+
+The size of the required quorum can be configured at
request time by setting the ``r`` parameter for document and view
reads, and the ``w`` parameter for document writes. For example, here
is a request that specifies that at least two nodes must respond in
@@ -84,49 +93,85 @@ order to establish a quorum:
.. code:: bash
- $ curl "$COUCH_URL:5984/{docId}?r=2"
+ $ curl "$COUCH_URL:5984/<doc>?r=2"
Here is a similar example for writing a document:
.. code:: bash
- $ curl -X PUT "http://xxx.xxx.xxx.xxx:5984/{docId}?w=2" -d '{...}'
+ $ curl -X PUT "$COUCH_URL:5984/<doc>?w=2" -d '{...}'
Setting ``r`` or ``w`` to be equal to ``n`` (the number of replicas)
means you will only receive a response once all nodes with relevant
-shards have responded or timed out, however even this does not
+shards have responded or timed out, and as such this approach does not
guarantee `ACIDic consistency
<https://en.wikipedia.org/wiki/ACID#Consistency>`__. Setting ``r`` or
``w`` to 1 means you will receive a response after only one relevant
node has responded.
+.. _cluster/sharding/move:
+
Moving a shard
--------------
-Moving shards between nodes involves the following steps:
-
-1. Copy the shard file onto the new node.
-2. Update cluster metadata to reflect the move.
-3. Replicate from the old to the new to catch any changes.
-4. Delete the old shard file.
+This section describes how to manually place and replace shards, and how
+to set up placement rules to assign shards to specific nodes. These
+activities are critical steps when you determine your cluster is too big
+or too small, and want to resize it successfully, or you have noticed
+from server metrics that database/shard layout is non-optimal and you
+have some "hot spots" that need resolving.
+
+Consider a three node cluster with q=8 and n=3. Each database has 24
+shards, distributed across the three nodes. If you add a fourth node to
+the cluster, CouchDB will not redistribute existing database shards to
+it. This leads to unbalanced load, as the new node will only host shards
+for databases created after it joined the cluster. To balance the
+distribution of shards from existing databases, they must be moved
+manually.
+
+Moving shards between nodes in a cluster involves the following steps:
+
+1. Copy the shard(s) and any secondary index shard(s) onto the target node.
+2. Set the target node to maintenance mode.
+3. Update cluster metadata to reflect the new target shard(s).
+4. Monitor internal replication to ensure up-to-date shard(s).
+5. Clear the target node's maintenance mode.
+6. Update cluster metadata again to remove the source shard(s)
+7. Remove the shard file(s) and secondary index file(s) from the source node.
Copying shard files
~~~~~~~~~~~~~~~~~~~
Shard files live in the ``data/shards`` directory of your CouchDB
-install. Since they are just files, you can use ``cp``, ``rsync``,
+install. Within those subdirectories are the shard files themselves. For
+instance, for a q=8 database called abc, here is its database shard
+files:
+
+::
+
+ data/shards/00000000-1fffffff/abc.1529362187.couch
+ data/shards/20000000-3fffffff/abc.1529362187.couch
+ data/shards/40000000-5fffffff/abc.1529362187.couch
+ data/shards/60000000-7fffffff/abc.1529362187.couch
+ data/shards/80000000-9fffffff/abc.1529362187.couch
+ data/shards/a0000000-bfffffff/abc.1529362187.couch
+ data/shards/c0000000-dfffffff/abc.1529362187.couch
+ data/shards/e0000000-ffffffff/abc.1529362187.couch
+
+Since they are just files, you can use ``cp``, ``rsync``,
``scp`` or other command to copy them from one node to another. For
example:
.. code:: bash
# one one machine
- mkdir -p data/shards/{range}
+ $ mkdir -p data/shards/<range>
# on the other
- scp $COUCH_PATH/data/shards/{range}/{database}.{timestamp}.couch $OTHER:$COUCH_PATH/data/shards/{range}/
+ $ scp <couch-dir>/data/shards/<range>/<database>.<datecode>.couch <node>:<couch-dir>/data/shards/<range>/
-Views are also sharded, and their shards should be moved to save the new
-node the effort of rebuilding the view. View shards live in
+Secondary indexes (including JavaScript views, Erlang views and Mango
+indexes) are also sharded, and their shards should be moved to save the
+new node the effort of rebuilding the view. View shards live in
``data/.shards``. For example:
::
@@ -141,60 +186,105 @@ node the effort of rebuilding the view. View shards live in
data/.shards/c0000000-dfffffff/_replicator.1518451591_design/mrview/3e823c2a4383ac0c18d4e574135a5b08.view
...
-Updating cluster metadata
-~~~~~~~~~~~~~~~~~~~~~~~~~
+.. warning::
+ Technically, copying database and secondary index
+ shards is optional. If you proceed to the next step without
+ performing
+ this data copy, CouchDB will use internal replication to populate
+ the
+ newly added shard replicas. However, this process can be very slow,
+ especially on a busy cluster—which is why we recommend performing this
+ manual data copy first.
+
+Set the target node to ``true`` maintenance mode
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Before we tell CouchDB about the new shards on the node in question, we
+need to set the node to ``true`` maintenance mode. This special mode
+instructs CouchDB to return a ``404 Not Found`` response on the ``/_up``
+endpoint, and ensures it not participate in normal interactive clustered
+requests for its shards. A properly configured load balancer that uses
+``GET /_up`` to check the health of nodes will detect this 404 and
+remove that node from the backend target, preventing any HTTP requests
+from being sent to that node. An example HAProxy configuration to use
+the ``/_up`` endpoint is as follows:
-To update the cluster metadata, use the special node-specific
-``/_dbs`` database, accessible via a node's private port, usually at
-port 5986. This port is only available on the localhost interface for
-security purposes. So first, retrieve the database's current metadata:
+::
-.. code:: bash
+ http-check disable-on-404
+ option httpchk GET /_up
+
+If you do not set maintenance mode, or the load balancer ignores this
+maintenance mode status, after the next step is performed the cluster
+may return incorrect responses when consulting the node in question. You
+don't want this! In the next steps, we will ensure that this shard is
+up-to-date before allowing it to participate in end-user requests.
+
+To set true maintenance mode:
+
+.. code::bash
- $ curl $COUCH_URL:5986/_dbs/{name}
+ $ curl -X PUT -H "Content-type: application/json" \
+ http://localhost:5984/_node/<nodename>/_config/couchdb/maintenance_mode \
+ -d "\"true\""
+
+Then, verify that the node is in maintenance mode by performing a ``GET
+/_up`` on that node's individual endpoint:
+
+.. code::bash
+
+ $ curl -v http://localhost:5984/_up
+ …
+ < HTTP/1.1 404 Object Not Found
+ …
+ {"status":"maintenance_mode"}
+
+Finally, check that your load balancer has removed the node from the
+pool of available backend nodes.
+
+Updating cluster metadata to reflect the move
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now we need to tell CouchDB that the target node (which must already be
+joined to the cluster) should be hosting shard replicas for a given
+database.
+
+To update the cluster metadata, use the special ``/_dbs`` database,
+which is an internal CouchDB database that maps databases to shards and
+nodes. It is accessible only via a node's private port, usually at port
+5986. By default, this port is only available on the localhost interface
+for security purposes.
+
+First, retrieve the database's current metadata:
+
+.. code:: bash
+ $ curl localhost:5986/_dbs/{name}
{
- "_id": "{name}",
- "_rev": "1-5e2d10c29c70d3869fb7a1fd3a827a64",
- "shard_suffix": [
- 46,
- 49,
- 52,
- 50,
- 53,
- 50,
- 48,
- 50,
- 53,
- 55,
- 55
+ "_id": "{name}",
+ "_rev": "1-e13fb7e79af3b3107ed62925058bfa3a",
+ "shard_suffix": [46, 49, 53, 51, 48, 50, 51, 50, 53, 50, 54],
+ "changelog": [
+ ["add", "00000000-1fffffff", "node1@xxx.xxx.xxx.xxx"],
+ ["add", "00000000-1fffffff", "node2@xxx.xxx.xxx.xxx"],
+ ["add", "00000000-1fffffff", "node3@xxx.xxx.xxx.xxx"],
+ …
+ ],
+ "by_node": {
+ "node1@xxx.xxx.xxx.xxx": [
+ "00000000-1fffffff",
+ …
],
- "changelog": [
- [
- "add",
- "00000000-7fffffff",
- "node1@xxx.xxx.xxx.xxx"
+ …
+ },
+ "by_range": {
+ "00000000-1fffffff": [
+ "node1@xxx.xxx.xxx.xxx",
+ "node2@xxx.xxx.xxx.xxx",
+ "node3@xxx.xxx.xxx.xxx"
],
- [
- "add",
- "80000000-ffffffff",
- "node1@xxx.xxx.xxx.xxx"
- ]
- ],
- "by_node": {
- "node1@xxx.xxx.xxx.xxx": [
- "00000000-7fffffff",
- "80000000-ffffffff"
- ]
- },
- "by_range": {
- "00000000-7fffffff": [
- "node1@xxx.xxx.xxx.xxx"
- ],
- "80000000-ffffffff": [
- "node1@xxx.xxx.xxx.xxx"
- ]
- }
+ …
+ }
}
Here is a brief anatomy of that document:
@@ -223,82 +313,55 @@ metadata's ``changelog`` attribute:
.. code:: json
- [
- "add",
- "{range}",
- "{name}@{address}"
- ]
+ ["add", "<range>", "<node-name>"]
+
+The ``<range>`` is the specific shard range for the shard. The ``<node-
+name>`` should match the name and address of the node as displayed in
+``GET /_membership`` on the cluster.
-*Note*: You can remove a shard from a node by specifying 'remove' instead of 'add'.
+.. warning::
+ When removing a shard from a node, specifying ``remove`` instead of ``add``.
Once you have figured out the new changelog entries, you will need to
update the ``by_node`` and ``by_range`` to reflect who is storing what
shards. The data in the changelog entries and these attributes must
match. If they do not, the database may become corrupted.
-As an example, here is an updated version of the metadata above that
-adds shards to a second node called ``node2``:
+Continuing our example, here is an updated version of the metadata above
+that adds shards to an additional node called ``node4``:
.. code:: json
{
- "_id": "{name}",
- "_rev": "1-5e2d10c29c70d3869fb7a1fd3a827a64",
- "shard_suffix": [
- 46,
- 49,
- 52,
- 50,
- 53,
- 50,
- 48,
- 50,
- 53,
- 55,
- 55
- ],
- "changelog": [
- [
- "add",
- "00000000-7fffffff",
- "node1@xxx.xxx.xxx.xxx"
- ],
- [
- "add",
- "80000000-ffffffff",
- "node1@xxx.xxx.xxx.xxx"
+ "_id": "{name}",
+ "_rev": "1-e13fb7e79af3b3107ed62925058bfa3a",
+ "shard_suffix": [46, 49, 53, 51, 48, 50, 51, 50, 53, 50, 54],
+ "changelog": [
+ ["add", "00000000-1fffffff", "node1@xxx.xxx.xxx.xxx"],
+ ["add", "00000000-1fffffff", "node2@xxx.xxx.xxx.xxx"],
+ ["add", "00000000-1fffffff", "node3@xxx.xxx.xxx.xxx"],
+ …
+ ["add", "00000000-1fffffff", "node4@xxx.xxx.xxx.xxx"]
+ ],
+ "by_node": {
+ "node1@xxx.xxx.xxx.xxx": [
+ "00000000-1fffffff",
+ …
],
- [
- "add",
- "00000000-7fffffff",
- "node2@yyy.yyy.yyy.yyy"
- ],
- [
- "add",
- "80000000-ffffffff",
- "node2@yyy.yyy.yyy.yyy"
+ …
+ "node4@xxx.xxx.xxx.xxx": [
+ "00000000-1fffffff"
]
+ },
+ "by_range": {
+ "00000000-1fffffff": [
+ "node1@xxx.xxx.xxx.xxx",
+ "node2@xxx.xxx.xxx.xxx",
+ "node3@xxx.xxx.xxx.xxx",
+ "node4@xxx.xxx.xxx.xxx"
],
- "by_node": {
- "node1@xxx.xxx.xxx.xxx": [
- "00000000-7fffffff",
- "80000000-ffffffff"
- ],
- "node2@yyy.yyy.yyy.yyy": [
- "00000000-7fffffff",
- "80000000-ffffffff"
- ]
- },
- "by_range": {
- "00000000-7fffffff": [
- "node1@xxx.xxx.xxx.xxx",
- "node2@yyy.yyy.yyy.yyy"
- ],
- "80000000-ffffffff": [
- "node1@xxx.xxx.xxx.xxx",
- "node2@yyy.yyy.yyy.yyy"
- ]
- }
+ …
+ }
}
Now you can ``PUT`` this new metadata:
@@ -307,40 +370,57 @@ Now you can ``PUT`` this new metadata:
$ curl -X PUT $COUCH_URL:5986/_dbs/{name} -d '{...}'
-Replicating from old to new
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Monitor internal replication to ensure up-to-date shard(s)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Because shards are just CouchDB databases, you can replicate them
-around. In order to make sure the new shard receives any updates the old
-one processed while you were updating its metadata, you should replicate
-the old shard to the new one:
+After you complete the previous step, as soon as CouchDB receives a
+write request for a shard on the target node, CouchDB will check if the
+target node's shard(s) are up to date. If it finds they are not up to
+date, it will trigger an internal replication job to complete this task.
+You can observe this happening by triggering a write to the database
+(update a document, or create a new one), while monitoring the
+``/_node/<nodename>/_system`` endpoint, which includes the
+``internal_replication_jobs`` metric.
-::
+Once this metric has returned to the baseline from before you wrote the
+document, or is zero (0., it is safe to proceed.
- $ curl -X POST $COUCH_URL:5986/_replicate -d '{ \
- "source": $OLD_SHARD_URL,
- "target": $NEW_SHARD_URL
- }'
+Clear the target node's maintenance mode
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-This will bring the new shard up to date so that we can safely delete
-the old one.
+You can now let the node back into the rotation for load balancing by
+putting ``"false"`` to the maintenance mode configuration endpoint, just
+as in step 2.
-Delete old shard
-~~~~~~~~~~~~~~~~
+Verify that the node is in not maintenance mode by performing a ``GET
+/_up`` on that node's individual endpoint.
-You can remove the old shard either by deleting its file or by deleting
-it through the private 5986 port:
+Finally, check that your load balancer has returned the node to the pool
+of available backend nodes.
-.. code:: bash
+Update cluster metadata again to remove the source shard
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now, remove the source shard from the shard map the same way that you
+added the new target shard to the shard map in step 2. Be sure to add
+the ``["remove", <range>, <source-shard>]`` entry to the end of the
+changelog as well as modifying both the by_node and by_range sections of
+the database metadata document.
+
+Remove the shard and secondary index files from the source node
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- # delete the file
- rm $COUCH_DIR/data/shards/$OLD_SHARD
+Finally, you can remove the source shard by deleting its file from the
+command line on the source host, along with any view shards:
- # OR delete the database
- curl -X DELETE $COUCH_URL:5986/$OLD_SHARD
+.. code::bash
-Congratulations! You have manually added a new shard. By adding and
-removing database shards in this way, they can be moved between nodes.
+ $ rm <couch-dir>/data/shards/<range>/<dbname>.<datecode>.couch
+ $ rm -r <couch-dir>/data/.shards/<range>/<dbname>.<datecode>*
+
+Congratulations! You have moved a database shard. By adding and removing
+database shards in this way, you can change the cluster's shard layout,
+also known as a shard map.
Specifying database placement
-----------------------------
@@ -351,7 +431,7 @@ placement rules.
First, each node must be labeled with a zone attribute. This defines
which zone each node is in. You do this by editing the node’s document
in the ``/nodes`` database, which is accessed through the “back-door”
-(5986) port. Add a key value pair of the form:
+(5986. port. Add a key value pair of the form:
::
@@ -361,49 +441,52 @@ Do this for all of the nodes in your cluster. For example:
.. code:: bash
- $ curl -X PUT $COUCH_URL:5986/_nodes/{name}@{address} \
+ $ curl -X PUT $COUCH_URL:5986/_nodes/<node-name> \
-d '{ \
- "_id": "{name}@{address}",
- "_rev": "{rev}",
- "zone": "{zone-name}"
+ "_id": "<node-name>",
+ "_rev": "<rev>",
+ "zone": "<zone-name>"
}'
-In the config file (local.ini or default.ini) of each node, define a
+In the local config file (``local.ini``) of each node, define a
consistent cluster-wide setting like:
::
[cluster]
- placement = {zone-name-1}:2,{zone-name-2}:1
+ placement = <zone-name-1>:2,<zone-name-2>:1
+
+In this example, CouchDB will ensure that two replicas for a shard will
+be hosted on nodes with the zone attribute set to ``<zone-name-1>`` and
+one replica will be hosted on a new with the zone attribute set to
+``<zone-name-2>``.
+
+This approach is flexible, since you can also specify zones on a per-
+database basis by specifying the placement setting as a query parameter
+when the database is created, using the same syntax as the ini file:
+
+.. code:: bash
-In this example, it will ensure that two replicas for a shard will be
-hosted on nodes with the zone attribute set to ``{zone-name-1}`` and one
-replica will be hosted on a new with the zone attribute set to
-``{zone-name-2}``.
+ curl -X PUT $COUCH_URL:5984/<dbname>?zone=<zone>
Note that you can also use this system to ensure certain nodes in the
cluster do not host any replicas for newly created databases, by giving
them a zone attribute that does not appear in the ``[cluster]``
placement string.
-You can also specify zones on a per-database basis by specifying the
-zone as a query parameter when the database is created:
+Resharding a database to a new q value
+--------------------------------------
-.. code:: bash
-
- curl -X PUT $COUCH_URL:5984/{dbName}?zone={zone}
-
-Resharding
-----------
+The q value for a database can only be set when the database is created,
+precluding live resharding. Instead, to reshard a database, it must be
+regenerated. Here are the steps:
-Shard settings for databases can only be set when the database is
-created, precluding live resharding. Instead, to reshard a database, it
-must be regenerated. Here are the steps:
-
-1. Create a temporary database with the desired shard settings.
-2. Replicate the primary database to the temporary. Multiple
- replications may be required if the primary database is under active
- use.
+1. Create a temporary database with the desired shard settings, by
+ specifying the q value as a query parameter during the PUT
+ operation.
+2. Replicate the primary database to the temporary one. Multiple
+ replications may be required if the primary database is under
+ active use.
3. Delete the primary database. **Make sure nobody is using it!**
4. Recreate the primary database with the desired shard settings.
5. Replicate the temporary back to the primary.
@@ -412,3 +495,7 @@ must be regenerated. Here are the steps:
Once all steps have completed, the database can be used again. The
cluster will create and distribute its shards according to placement
rules automatically.
+
+Downtime can be avoided in production if the client application(s) can
+be instructed to use the new database instead of the old one, and a cut-
+over is performed during a very brief outage window.
[couchdb-documentation] 10/18: respond to feedback
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit 976c0fe62f70612883b3b042f36a9568c44fd440
Author: Diana Thayer <ga...@gmail.com>
AuthorDate: Wed Jul 18 10:42:23 2018 -0700
respond to feedback
---
src/cluster/sharding.rst | 152 +++++++++++++++++++++++++----------------------
1 file changed, 80 insertions(+), 72 deletions(-)
diff --git a/src/cluster/sharding.rst b/src/cluster/sharding.rst
index d4786c0..7f92fef 100644
--- a/src/cluster/sharding.rst
+++ b/src/cluster/sharding.rst
@@ -21,13 +21,17 @@ Shard Management
Introduction
------------
+This document discusses how sharding works in CouchDB along with how to
+safely add, move, remove, and create placement rules for shards and
+shard replicas.
+
A `shard
<https://en.wikipedia.org/wiki/Shard_(database_architecture)>`__ is a
horizontal partition of data in a database. Partitioning data into
shards and distributing copies of each shard (called "shard replicas" or
just "replicas") to different nodes in a cluster gives the data greater
durability against node loss. CouchDB clusters automatically shard
-databases and distribute the subsections of documents that compose each
+databases and distribute the subsets of documents that compose each
shard among nodes. Modifying cluster membership and sharding behavior
must be done manually.
@@ -35,17 +39,17 @@ Shards and Replicas
~~~~~~~~~~~~~~~~~~~
How many shards and replicas each database has can be set at the global
-level, or on a per-database basis. The relevant parameters are *q* and
-*n*.
+level, or on a per-database basis. The relevant parameters are ``q`` and
+``n``.
*q* is the number of database shards to maintain. *n* is the number of
-copies of each document to distribute. The default value for n is 3,
-and for q is 8. With q=8, the database is split into 8 shards. With
-n=3, the cluster distributes three replicas of each shard. Altogether,
-that's 24 shards for a single database. In a default 3-node cluster,
+copies of each document to distribute. The default value for ``n`` is ``3``,
+and for ``q`` is ``8``. With ``q=8``, the database is split into 8 shards. With
+``n=3``, the cluster distributes three replicas of each shard. Altogether,
+that's 24 shard replicas for a single database. In a default 3-node cluster,
each node would receive 8 shards. In a 4-node cluster, each node would
receive 6 shards. We recommend in the general case that the number of
-nodes in your cluster should be a multiple of n, so that shards are
+nodes in your cluster should be a multiple of ``n``, so that shards are
distributed evenly.
CouchDB nodes have a ``etc/default.ini`` file with a section named
@@ -67,7 +71,7 @@ example:
$ curl -X PUT "$COUCH_URL/database-name?q=4&n=2"
That creates a database that is split into 4 shards and 2 replicas,
-yielding 8 shards distributed throughout the cluster.
+yielding 8 shard replicas distributed throughout the cluster.
Quorum
~~~~~~
@@ -85,11 +89,11 @@ once a `quorum
<https://en.wikipedia.org/wiki/Quorum_(distributed_computing)>`__ of
database nodes have responded; 2, by default.
-The size of the required quorum can be configured at
-request time by setting the ``r`` parameter for document and view
-reads, and the ``w`` parameter for document writes. For example, here
-is a request that specifies that at least two nodes must respond in
-order to establish a quorum:
+The size of the required quorum can be configured at request time by
+setting the ``r`` parameter for document and view reads, and the ``w``
+parameter for document writes. For example, here is a request that
+directs the coordinating node to send a response once at least two nodes
+have responded:
.. code:: bash
@@ -114,14 +118,13 @@ node has responded.
Moving a shard
--------------
-This section describes how to manually place and replace shards, and how
-to set up placement rules to assign shards to specific nodes. These
+This section describes how to manually place and replace shards. These
activities are critical steps when you determine your cluster is too big
or too small, and want to resize it successfully, or you have noticed
from server metrics that database/shard layout is non-optimal and you
have some "hot spots" that need resolving.
-Consider a three node cluster with q=8 and n=3. Each database has 24
+Consider a three-node cluster with q=8 and n=3. Each database has 24
shards, distributed across the three nodes. If you add a fourth node to
the cluster, CouchDB will not redistribute existing database shards to
it. This leads to unbalanced load, as the new node will only host shards
@@ -131,6 +134,7 @@ manually.
Moving shards between nodes in a cluster involves the following steps:
+0. Ensure the target node has joined the cluster.
1. Copy the shard(s) and any secondary index shard(s) onto the target node.
2. Set the target node to maintenance mode.
3. Update cluster metadata to reflect the new target shard(s).
@@ -142,9 +146,19 @@ Moving shards between nodes in a cluster involves the following steps:
Copying shard files
~~~~~~~~~~~~~~~~~~~
+.. note::
+ Technically, copying database and secondary index
+ shards is optional. If you proceed to the next step without
+ performing
+ this data copy, CouchDB will use internal replication to populate
+ the
+ newly added shard replicas. However, this process can be very slow,
+ especially on a busy cluster—which is why we recommend performing this
+ manual data copy first.
+
Shard files live in the ``data/shards`` directory of your CouchDB
install. Within those subdirectories are the shard files themselves. For
-instance, for a q=8 database called abc, here is its database shard
+instance, for a ``q=8`` database called ``abc``, here is its database shard
files:
::
@@ -159,7 +173,7 @@ files:
data/shards/e0000000-ffffffff/abc.1529362187.couch
Since they are just files, you can use ``cp``, ``rsync``,
-``scp`` or other command to copy them from one node to another. For
+``scp`` or any other command to copy them from one node to another. For
example:
.. code:: bash
@@ -186,28 +200,17 @@ new node the effort of rebuilding the view. View shards live in
data/.shards/c0000000-dfffffff/_replicator.1518451591_design/mrview/3e823c2a4383ac0c18d4e574135a5b08.view
...
-.. warning::
- Technically, copying database and secondary index
- shards is optional. If you proceed to the next step without
- performing
- this data copy, CouchDB will use internal replication to populate
- the
- newly added shard replicas. However, this process can be very slow,
- especially on a busy cluster—which is why we recommend performing this
- manual data copy first.
-
Set the target node to ``true`` maintenance mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Before we tell CouchDB about the new shards on the node in question, we
-need to set the node to ``true`` maintenance mode. This special mode
-instructs CouchDB to return a ``404 Not Found`` response on the ``/_up``
-endpoint, and ensures it not participate in normal interactive clustered
-requests for its shards. A properly configured load balancer that uses
-``GET /_up`` to check the health of nodes will detect this 404 and
-remove that node from the backend target, preventing any HTTP requests
-from being sent to that node. An example HAProxy configuration to use
-the ``/_up`` endpoint is as follows:
+Before telling CouchDB about the new shards on the node in question, it
+must be put into maintenance mode. Maintenance mode instructs CouchDB to
+return a ``404 Not Found`` response on the ``/_up`` endpoint, and
+ensures it does not participate in normal interactive clustered requests
+for its shards. A properly configured load balancer that uses ``GET
+/_up`` to check the health of nodes will detect this 404 and remove the
+node from circulation, preventing requests from being sent to that node.
+For example, to configure HAProxy to use the ``/_up`` endpoint, use:
::
@@ -220,12 +223,12 @@ may return incorrect responses when consulting the node in question. You
don't want this! In the next steps, we will ensure that this shard is
up-to-date before allowing it to participate in end-user requests.
-To set true maintenance mode:
+To enable maintenance mode:
.. code::bash
$ curl -X PUT -H "Content-type: application/json" \
- http://localhost:5984/_node/<nodename>/_config/couchdb/maintenance_mode \
+ $COUCH_URL:5984/_node/<nodename>/_config/couchdb/maintenance_mode \
-d "\"true\""
Then, verify that the node is in maintenance mode by performing a ``GET
@@ -233,7 +236,7 @@ Then, verify that the node is in maintenance mode by performing a ``GET
.. code::bash
- $ curl -v http://localhost:5984/_up
+ $ curl -v $COUCH_URL/_up
…
< HTTP/1.1 404 Object Not Found
…
@@ -251,15 +254,15 @@ database.
To update the cluster metadata, use the special ``/_dbs`` database,
which is an internal CouchDB database that maps databases to shards and
-nodes. It is accessible only via a node's private port, usually at port
-5986. By default, this port is only available on the localhost interface
-for security purposes.
+nodes. This database is replicated between nodes. It is accessible only
+via a node-local port, usually at port 5986. By default, this port is
+only available on the localhost interface for security purposes.
First, retrieve the database's current metadata:
.. code:: bash
- $ curl localhost:5986/_dbs/{name}
+ $ curl http://localhost:5986/_dbs/{name}
{
"_id": "{name}",
"_rev": "1-e13fb7e79af3b3107ed62925058bfa3a",
@@ -304,9 +307,11 @@ To reflect the shard move in the metadata, there are three steps:
2. Update the ``by_node`` entries.
3. Update the ``by_range`` entries.
-As of this writing, this process must be done manually. **WARNING: Be
-very careful! Mistakes during this process can irreperably corrupt the
-cluster!**
+.. warning::
+ Be very careful! Mistakes during this process can
+ irreparably corrupt the cluster!
+
+As of this writing, this process must be done manually.
To add a shard to a node, add entries like this to the database
metadata's ``changelog`` attribute:
@@ -319,8 +324,8 @@ The ``<range>`` is the specific shard range for the shard. The ``<node-
name>`` should match the name and address of the node as displayed in
``GET /_membership`` on the cluster.
-.. warning::
- When removing a shard from a node, specifying ``remove`` instead of ``add``.
+.. note::
+ When removing a shard from a node, specify ``remove`` instead of ``add``.
Once you have figured out the new changelog entries, you will need to
update the ``by_node`` and ``by_range`` to reflect who is storing what
@@ -368,7 +373,7 @@ Now you can ``PUT`` this new metadata:
.. code:: bash
- $ curl -X PUT $COUCH_URL:5986/_dbs/{name} -d '{...}'
+ $ curl -X PUT http://localhost:5986/_dbs/{name} -d '{...}'
Monitor internal replication to ensure up-to-date shard(s)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -383,16 +388,17 @@ You can observe this happening by triggering a write to the database
``internal_replication_jobs`` metric.
Once this metric has returned to the baseline from before you wrote the
-document, or is zero (0., it is safe to proceed.
+document, or is ``0``, the shard replica is ready to serve data and we
+can bring the node out of maintenance mode.
Clear the target node's maintenance mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-You can now let the node back into the rotation for load balancing by
+You can now let the node start servicing data requests by
putting ``"false"`` to the maintenance mode configuration endpoint, just
as in step 2.
-Verify that the node is in not maintenance mode by performing a ``GET
+Verify that the node is not in maintenance mode by performing a ``GET
/_up`` on that node's individual endpoint.
Finally, check that your load balancer has returned the node to the pool
@@ -404,34 +410,34 @@ Update cluster metadata again to remove the source shard
Now, remove the source shard from the shard map the same way that you
added the new target shard to the shard map in step 2. Be sure to add
the ``["remove", <range>, <source-shard>]`` entry to the end of the
-changelog as well as modifying both the by_node and by_range sections of
+changelog as well as modifying both the ``by_node`` and ``by_range`` sections of
the database metadata document.
Remove the shard and secondary index files from the source node
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Finally, you can remove the source shard by deleting its file from the
-command line on the source host, along with any view shards:
+Finally, you can remove the source shard replica by deleting its file from the
+command line on the source host, along with any view shard replicas:
.. code::bash
$ rm <couch-dir>/data/shards/<range>/<dbname>.<datecode>.couch
$ rm -r <couch-dir>/data/.shards/<range>/<dbname>.<datecode>*
-Congratulations! You have moved a database shard. By adding and removing
-database shards in this way, you can change the cluster's shard layout,
+Congratulations! You have moved a database shard replica. By adding and removing
+database shard replicas in this way, you can change the cluster's shard layout,
also known as a shard map.
Specifying database placement
-----------------------------
-Database shards can be configured to live solely on specific nodes using
-placement rules.
+You can configure CouchDB to put shard replicas on certain nodes at
+database creation time using placement rules.
First, each node must be labeled with a zone attribute. This defines
which zone each node is in. You do this by editing the node’s document
-in the ``/nodes`` database, which is accessed through the “back-door”
-(5986. port. Add a key value pair of the form:
+in the ``/_nodes`` database, which is accessed through the node-local
+port. Add a key value pair of the form:
::
@@ -441,7 +447,7 @@ Do this for all of the nodes in your cluster. For example:
.. code:: bash
- $ curl -X PUT $COUCH_URL:5986/_nodes/<node-name> \
+ $ curl -X PUT http://localhost:5986/_nodes/<node-name> \
-d '{ \
"_id": "<node-name>",
"_rev": "<rev>",
@@ -477,20 +483,22 @@ placement string.
Resharding a database to a new q value
--------------------------------------
-The q value for a database can only be set when the database is created,
-precluding live resharding. Instead, to reshard a database, it must be
-regenerated. Here are the steps:
+The ``q`` value for a database can only be set when the database is
+created, precluding live resharding. Instead, to reshard a database, it
+must be regenerated. Here are the steps:
1. Create a temporary database with the desired shard settings, by
specifying the q value as a query parameter during the PUT
operation.
-2. Replicate the primary database to the temporary one. Multiple
+2. Stop clients accessing the database.
+3. Replicate the primary database to the temporary one. Multiple
replications may be required if the primary database is under
active use.
-3. Delete the primary database. **Make sure nobody is using it!**
-4. Recreate the primary database with the desired shard settings.
-5. Replicate the temporary back to the primary.
-6. Delete the temporary database.
+4. Delete the primary database. **Make sure nobody is using it!**
+5. Recreate the primary database with the desired shard settings.
+6. Clients can now access the database again.
+7. Replicate the temporary back to the primary.
+8. Delete the temporary database.
Once all steps have completed, the database can be used again. The
cluster will create and distribute its shards according to placement
[couchdb-documentation] 13/18: feat: remove proxy docs, as they don’t work in 2.x
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit 3d0dd3a69d59ce08c6c6733b11e6370eb4dc99f8
Author: Jan Lehnardt <ja...@apache.org>
AuthorDate: Thu Jul 19 09:31:09 2018 +0200
feat: remove proxy docs, as they don’t work in 2.x
---
src/config/index.rst | 1 -
src/config/proxying.rst | 94 -------------------------------------------------
2 files changed, 95 deletions(-)
diff --git a/src/config/index.rst b/src/config/index.rst
index 439a980..1fec7a7 100644
--- a/src/config/index.rst
+++ b/src/config/index.rst
@@ -32,4 +32,3 @@ Configuring CouchDB
http-handlers
services
misc
- proxying
diff --git a/src/config/proxying.rst b/src/config/proxying.rst
deleted file mode 100644
index 1e19bf9..0000000
--- a/src/config/proxying.rst
+++ /dev/null
@@ -1,94 +0,0 @@
-.. Licensed under the Apache License, Version 2.0 (the "License"); you may not
-.. use this file except in compliance with the License. You may obtain a copy of
-.. the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing, software
-.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
-.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
-.. License for the specific language governing permissions and limitations under
-.. the License.
-
-.. highlight:: ini
-.. _config/proxy:
-
-======================
-Proxying Configuration
-======================
-
-.. _http-proxying:
-.. _config/proxy/couchdb:
-
-CouchDB As Proxy
-================
-
-The HTTP proxy feature makes it easy to map and redirect different content
-through your CouchDB URL. The proxy works by mapping a pathname and passing all
-content after that prefix through to the configured proxy address.
-
-Configuration of the proxy redirect is handled through the
-``[httpd_global_handlers]`` section of the CouchDB configuration file
-(typically ``local.ini``). The format is::
-
- [httpd_global_handlers]
- PREFIX = {couch_httpd_proxy, handle_proxy_req, <<"DESTINATION">>}
-
-Where:
-
-- ``PREFIX``
-
- Is the string that will be matched. The string can be any valid qualifier,
- although to ensure that existing database names are not overridden by a proxy
- configuration, you can use an underscore prefix.
-
-- ``DESTINATION``
-
- The fully-qualified URL to which the request should be sent. The destination
- must include the ``http`` prefix. The content is used verbatim in the
- original request, so you can also forward to servers on different ports and
- to specific paths on the target host.
-
-The proxy process then translates requests of the form:
-
-.. code-block:: text
-
- http://couchdb:5984/PREFIX/path
-
-To:
-
-.. code-block:: text
-
- DESTINATION/path
-
-.. note::
- Everything after ``PREFIX`` including the required forward slash will be
- appended to the ``DESTINATION``.
-
-The response is then communicated back to the original client.
-
-For example, the following configuration::
-
- [httpd_global_handlers]
- _google = {couch_httpd_proxy, handle_proxy_req, <<"http://www.google.com">>}
-
-Would forward all requests for ``http://couchdb:5984/_google`` to the
-Google website.
-
-The service can also be used to forward to related CouchDB services,
-such as `Lucene`::
-
- [httpd_global_handlers]
- _fti = {couch_httpd_proxy, handle_proxy_req, <<"http://127.0.0.1:5985">>}
-
-.. note::
- The proxy service is basic. If the request is not identified by the
- ``DESTINATION``, or the remainder of the ``PATH`` specification is
- incomplete, the original request URL is interpreted as if the
- ``PREFIX`` component of that URL does not exist.
-
- For example, requesting ``http://couchdb:5984/_intranet/media`` when
- ``/media`` on the proxy destination does not exist, will cause the request
- URL to be interpreted as ``http://couchdb:5984/media``. Care should be
- taken to ensure that both requested URLs and destination URLs are able to
- cope.
[couchdb-documentation] 18/18: [WIP] Partial 2.2 release notes
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit 2ab4aa9cf0bbf1d6f5fea4317079bc2a661316c1
Author: Joan Touzet <wo...@atypical.net>
AuthorDate: Mon Jul 23 19:56:57 2018 -0400
[WIP] Partial 2.2 release notes
---
src/whatsnew/2.2.rst | 148 +++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 143 insertions(+), 5 deletions(-)
diff --git a/src/whatsnew/2.2.rst b/src/whatsnew/2.2.rst
index 8b19c67..8c41004 100644
--- a/src/whatsnew/2.2.rst
+++ b/src/whatsnew/2.2.rst
@@ -32,28 +32,166 @@ Upgrade Notes
version of CouchDB is replicating TO this instance/cluster. Replicating FROM older
versions is fine. A future version will make this a default.
-* Item 2
+* :ghissue:`820`, :ghissue:`1032`: Multiple queries can now be made at the
+ ``POST /{db}/_all_docs/queries``, ``POST /{db}/_design_docs/queries`` and
+ ``POST /{db}/_local_docs/queries`` endpoints. Also, a new endpoint
+ ``POST /{db}/_design/{ddoc}/_view/{view}/queries`` has been introduced to replace
+ the ``?queries`` parameter formerly provided for making multiple queries to a view.
+ The old ``?queries`` parameter *is now deprecated and will be removed in a future release
+ of CouchDB.*
+
+* The maximum http request limit, which had been lowered in 2.1.0, has been re-raised
+ to a 4GB limit for now. (:ghissue:`1446`). Ongoing discussion about the path forward
+ for future releases is available in :ghissue:`1200` and :ghissue:`1253`.
+
+* :ghissue:`1153`: The CouchDB replicator can now make use of the ``/_session`` endpoint
+ rather than relying entirely on HTTP basic authentication headers. This can greatly
+ improve replication performance. We encourage you to upgrade any nodes or clusters that
+ regularly act as replication clients to use this new feature, which is enabled by
+ default (:ghissue:`1462`).
-* Item 3
Version 2.2.0
=============
-Security
+Features
+--------
+
+.. rst-class:: open
+
+* Much improved documentation. Highlights include:
+ * TODO: LINK Database sharding/clustering: complete rewrite
+ * Developer installation notes (``INSTALL.*.rst``)
+ *
+* Much improved Fauxton functionality. Highlights include:
+ *
+* :ghissue:`496`, :issue:`3287`: New pluggable storage engine framework has landed in
+ CouchDB. This internal refactor makes it possible for CouchDB to use different backends
+ for storing the base database file itself. The refactor included a full migration of
+ the existing "legacy" storage engine into the new framework.
+* :ghissue:`603`: When creating a new database on a cluster without quorum, CouchDB will
+ now return a ``202 Accepted`` code if possible, indicating that at least one node
+ has written the database record to disk, and that other nodes will be updated as they
+ return to an online state. This replaces the former ``500`` internal error.
+* :ghissue:`745`: CouchDB no longer fails to complete replicating databases with
+ large attachments. The fix for this issue included several related changes:
+ * The maximum http request limit, which had been lowered in 2.1.0, has been re-raised
+ to a 4GB limit for now. (:ghissue:`1446`). Ongoing discussion about the path forward
+ for future releases is available in :ghissue:`1200` and :ghissue:`1253`.
+ * An update to the replicator http client that improves active socket accounting,
+ without which CouchDB can cease to be responsive over the main http interface
+ (:ghissue:`1117`)
+ * The replicator's http client no longer performs unconditional retries on failure
+ (:ghissue:`1177`)
+ * A path by which CouchDB could lose track of their RPC workers during multipart
+ attachment processing was removed. (:ghissue:`1178`)
+ * When CouchDB transmits a ``413 Payload Too Large`` response on attachment upload,
+ it now correctly flushes the receive socket before closing the connection to avoid
+ a TCP reset, and to give the client a better chance of parsing the 413 response. In
+ tandem, the replicator http client correctly closes its own socket after processing
+ any 413 response. (:ghissue:`1234`)
+TODO: LINK below
+* :ghissue:`822`: A new ``POST /_dbs_info`` endpoint has been added to return information
+ about a list of specified databases. This endpoint can take the place of multiple
+ queries to ``/{db}``.
+* :ghissue:`875`, :ghissue:`1030`: ``couch_peruser`` installations can now specify a
+ default ``q`` value for each peruser-created database that is different from the
+ cluster's ``q`` value. Set this in your local ini file, under ``[couch_peruser] q``.
+* :ghissue:`876`, :ghissue:`1068`: The ``couch_peruser`` database prefix is now
+ configurable through your local ini file, under ``[couch_peruser] database_prefix``.
+* :ghissue:`887`: Replicator documents can now include parameters for target database
+ creation, such as ``"create_target_params": {"q": "1"}``. This can assist in
+ database resharding or placement.
+* :ghissue:`977`: When using ``COPY`` to copy a document, CouchDB no longer fails if
+ the new ID includes Unicode characters.
+* :ghissue:`1095`: Recognize the environment variables ``ARGS_FILE``, ``SYSCONFIG_FILE``,
+ ``COUCHDB_ARGS_FILE`` and ``COUCHDB_SYSCONFIG_FILE`` to overrride where CouchDB looks
+ for the ``vm.args`` and ``sys.config`` files at startup.
+* :ghissue:`1126`: When queried back after saving, replication documents no longer
+ contain sensitive credential information (such as basic authenticataion headers).
+* :ghissue:`1203`: The compaction daemon now has a snooze period, during which it waits
+ to start the next compaction after finishing the previous one. This value is useful
+ in ``couch_peruser`` setups, which can cause a CPU spike every ``check_interval``
+ seconds. The setting can be adjusted in your local ini file via
+ ``[compaction_daemon] snooze_period``. The current default is a 3 second pause.
+
+Bugfixes
--------
-General
--------
+.. rst-class:: open
+
+* :ghissue:`832`, :ghissue:`1064`: Tracking of Couch logging stats has been added back
+ into the per-node ``/_node/<node-name>/_stats`` endpoint.
+* :ghissue:`955`: The ``/{db}/_bulk_docs`` endpoint now correctly responds with a
+ ``400 Bad Request`` error if the ``new_edits`` parameter is not a boolean.
+* :ghissue:`953`, :ghissue:`973`: Return ``404 Not Found`` on ``GET /_scheduler``,
+ not ``405 Method Not Allowed``.
+* :ghissue:`969`: CouchDB now returns ``offset`` and ``update_seq`` values when ``keys``
+ are provided to the ``GET`` or ``POST`` ``/{db}/_all_docs?update_seq=true`` endpoints.
+ This was affecting PouchDB compatibility.
+* :ghissue:`1012`: Address a theoretical race condition the replication scheduler could
+ encounter when trying to determine if the cluster is "stable" enough to resume
+ handling replication-introduced document updates.
+* :ghissue:`1051`: Return a user-friendly error message when attempting to create a
+ CouchDB user with an invalid password field (non-string).
Performance
-----------
+.. rst-class:: open
+
Mango
-----
+.. rst-class:: open
+
+* :ghissue:`816`, :ghissue:`962`, :ghissue:`1038`: If a user specifies a value for
+ ``use_index`` that is not valid for the selector (does not meet coverage requirements
+ or proper sort fields), attempt to fall back to a valid index or full DB scan rather
+ than returning a ``400``. If we fall back, populate a ``warning`` field in the
+ response. Mango also tries to use indexes where ``$or`` may select a field only when
+ certain values are present.
+* :ghissue:`849`: When ``{"seq_indexed": true}`` is specified, a badmatch error was
+ returned. This is now fixed.
+* :ghissue:`951`: When using ``GET /{db}/_index``, only use a partial filter selector for
+ an index if it is set to something other than the default.
+* :ghissue:`961`: Do not prefix ``_design/`` to a Mango index name whose user-specified
+ name already starts with ``_design/``.
+* :ghissue:`988`, :ghissue:`989`: When specifying a ``use_index`` value with an invalid
+ index, correctly return a ``400 Bad Request`` showing that the requested index is
+ invalid for the request specified.
+* :ghissue:`998`: The fix for :ref:`CVE 2017-12635 <cve/2017-12635>` presented a breaking
+ change to Mango's ``/{db}/_find``, which would evaluate all instances of all JSON
+ fields in a selector. Mango is now tested to ensure it only considers the last instance
+ of a field, silently ignoring those that appear before it.
+* :ghissue:`1014`: Correctly deduce list of indexed fields in a selector when nested
+ ``$and`` operators are specified.
+* :ghissue:`1023`: Fix an unexpected ``500`` error if ``startkey`` and ``endkey`` in a
+ Mango selector were reversed.
+* :ghissue:`1067`: Prevent an ``invalid_cast`` crash when the ``couch_proc_manager`` soft
+ limit for processes is reached and mango idle processes are stopped.
+* Multiple fixes related to using Mango as a front-end for full text indexing (a feature
+ not shipped with couch, but for which support is in place as a compile-time addon).
+
+
Other
-----
The 2.2.0 release also includes the following minor improvements:
+.. rst-class:: open
+
+* Developers can, at build time, enable curl libraries & disable Fauxton and documentation
+ builds by specifying the new ``--dev`` option to the ``configure`` script.
+* The ``mochiweb`` dependency was bumped to version 2.17.0, in part to address the
+ difficult :ghissue:`745` issue.
+* Improved compatibility with newer versions of Erlang (20.x)
+* Multiple test suite improvements, focused on increased coverage, speed, and
+ reliability.
+* Improvements to the Travis CI and Jenkins CI setups, focused on improved long-term
+ project maintenance and automatability.
+* Related improvements to the CouchDB deb/rpm packaging and Docker repositories to
+ make deployment even easier.
+* :ghissue:`1007`: Move ``etc/default.ini`` entries back into ``[replicator]`` section
+ (incorrectly moved to ``[couch_peruser]`` section)
* A pony! OK, no, not really. If you got this far, thank you for reading.
[couchdb-documentation] 01/18: revisit sharding documentation
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit 764ee3f5ff132fc8035c2062d1513e3eb0116ab7
Author: Diana Thayer <ga...@gmail.com>
AuthorDate: Fri Apr 6 11:52:38 2018 -0700
revisit sharding documentation
---
src/cluster/sharding.rst | 400 ++++++++++++++++++++++++++++++++---------------
1 file changed, 270 insertions(+), 130 deletions(-)
diff --git a/src/cluster/sharding.rst b/src/cluster/sharding.rst
index dec89d4..ee536bb 100644
--- a/src/cluster/sharding.rst
+++ b/src/cluster/sharding.rst
@@ -18,43 +18,170 @@ Sharding
.. _cluster/sharding/scaling-out:
-Scaling out
-===========
+Introduction
+------------
-Normally you start small and grow over time. In the beginning you might do just
-fine with one node, but as your data and number of clients grows, you need to
-scale out.
+A
+`shard <https://en.wikipedia.org/wiki/Shard_(database_architecture)>`__
+is a horizontal partition of data in a database. Partitioning data into
+shards and distributing copies of each shard (called "replicas") to
+different nodes in a cluster gives the data greater durability against
+node loss. CouchDB clusters automatically shard and distribute data
+among nodes, but modifying cluster membership and customizing shard
+behavior must be done manually.
-For simplicity we will start fresh and small.
+Shards and Replicas
+~~~~~~~~~~~~~~~~~~~
-Start ``node1`` and add a database to it. To keep it simple we will have 2
-shards and no replicas.
+How many shards and replicas each database has can be set at the global
+level, or on a per-database basis. The relevant parameters are *q* and
+*n*.
-.. code-block:: bash
+*q* is the number of database shards to maintain. *n* is the number of
+copies of each document to distribute. With q=8, the database is split
+into 8 shards. With n=3, the cluster distributes three replicas of each
+shard. Altogether, that's 24 shards for a single database. In a default
+3-node cluster, each node would receive 8 shards. In a 4-node cluster,
+each node would receive 6 shards.
- curl -X PUT "http://xxx.xxx.xxx.xxx:5984/small?n=1&q=2" --user daboss
+CouchDB nodes have a ``etc/default.ini`` file with a section named
+``[cluster]`` which looks like this:
-If you look in the directory ``data/shards`` you will find the 2 shards.
+::
-.. code-block:: text
+ [cluster]
+ q=8
+ n=3
- data/
- +-- shards/
- | +-- 00000000-7fffffff/
- | | -- small.1425202577.couch
- | +-- 80000000-ffffffff/
- | -- small.1425202577.couch
+These settings can be modified to set sharding defaults for all
+databases, or they can be set on a per-database basis by specifying the
+``q`` and ``n`` query parameters when the database is created. For
+example:
-Now, check the node-local ``_dbs_`` database. Here, the metadata for each
-database is stored. As the database is called ``small``, there is a document
-called ``small`` there. Let us look in it. Yes, you can get it with curl too:
+.. code:: bash
-.. code-block:: javascript
+ $ curl -X PUT "$COUCH_URL/database-name?q=4&n=2"
- curl -X GET "http://xxx.xxx.xxx.xxx:5986/_dbs/small"
+That creates a database that is split into 4 shards and 2 replicas,
+yielding 8 shards distributed throughout the cluster.
+
+Quorum
+------
+
+When a CouchDB cluster serves reads and writes, it proxies the request
+to nodes with relevant shards and responds once enough nodes have
+responded to establish
+`quorum <https://en.wikipedia.org/wiki/Quorum_(distributed_computing)>`__.
+The size of the required quorum can be configured at request time by
+setting the ``r`` parameter for document and view reads, and the ``w``
+parameter for document writes. For example, here is a request that
+specifies that at least two nodes must respond in order to establish
+quorum:
+
+.. code:: bash
+
+ $ curl "$COUCH_URL:5984/{docId}?r=2"
+
+Here is a similar example for writing a document:
+
+.. code:: bash
+
+ $ curl -X PUT "$COUCH_URL:5984/{docId}?w=2" -d '{}'
+
+Setting ``r`` or ``w`` to be equal to ``n`` (the number of replicas)
+means you will only receive a response once all nodes with relevant
+shards have responded, however even this does not guarantee `ACIDic
+consistency <https://en.wikipedia.org/wiki/ACID#Consistency>`__. Setting
+``r`` or ``w`` to 1 means you will receive a response after only one
+relevant node has responded.
+
+Adding a node
+-------------
+
+To add a node to a cluster, first you must have the additional node
+running somewhere. Make note of the address it binds to, like
+``127.0.0.1``, then ``PUT`` an empty document to the ``/_node``
+endpoint:
+
+.. code:: bash
+
+ $ curl -X PUT "$COUCH_URL:5984/_node/{name}@{address}" -d '{}'
+
+This will add the node to the cluster. Existing shards will not be moved
+or re-balanced in response to the addition, but future operations will
+distribute shards to the new node.
+
+Now when you GET the ``/_membership`` endpoint, you will see the new
+node.
+
+Removing a node
+---------------
+
+To remove a node from the cluster, you must first acquire the ``_rev``
+value for the document that signifies its existence:
+
+.. code:: bash
+
+ $ curl "$COUCH_URL:5984/_node/{name}@{address}"
+ {"_id":"{name}@{address}","_rev":"{rev}"}
+
+Using that ``_rev``, you can delete the node using the ``/_node``
+endpoint:
+
+.. code:: bash
+
+ $ curl -X DELETE "$COUCH_URL:5984/_node/{name}@{address}?rev={rev}"
+
+.. raw:: html
+
+ <div class="alert alert-warning">
+
+**Note**: Before you remove a node, make sure to
+`move its shards <#moving-a-shard>`__
+or else they will be lost.
+
+Moving a shard
+--------------
+
+Moving shards between nodes involves the following steps:
+
+1. Copy the shard file onto the new node.
+2. Update cluster metadata to reflect the move.
+3. Replicate from the old to the new to catch any changes.
+4. Delete the old shard file.
+
+Copying shard files
+~~~~~~~~~~~~~~~~~~~
+
+Shard files live in the ``data/shards`` directory of your CouchDB
+install. Since they are just files, you can use ``cp``, ``rsync``,
+``scp`` or other command to copy them from one node to another. For
+example:
+
+.. code:: bash
+
+ # one one machine
+ mkdir -p data/shards/{range}
+ # on the other
+ scp $COUCH_PATH/data/shards/{range}/{database}.{timestamp}.couch $OTHER:$COUCH_PATH/data/shards/{range}/
+
+Views are also sharded, and their shards should be moved to save the new
+node the effort of rebuilding the view. View shards live in
+``data/.shards``.
+
+Updating cluster metadata
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To update the cluster metadata, use the special node-specific ``/_dbs``
+database, accessible via a node's private port, usually at port 5986.
+First, retrieve the database's current metadata:
+
+.. code:: bash
+
+ $ curl $COUCH_URL:5986/_dbs/{name}
{
- "_id": "small",
+ "_id": "{name}",
"_rev": "1-5e2d10c29c70d3869fb7a1fd3a827a64",
"shard_suffix": [
46,
@@ -97,27 +224,49 @@ called ``small`` there. Let us look in it. Yes, you can get it with curl too:
}
}
-* ``_id`` The name of the database.
-* ``_rev`` The current revision of the metadata.
-* ``shard_suffix`` The numbers after small and before .couch. This is seconds
- after UNIX epoch when the database was created. Stored as ASCII characters.
-* ``changelog`` Self explaining. Mostly used for debugging.
-* ``by_node`` List of shards on each node.
-* ``by_range`` On which nodes each shard is.
+Here is a brief anatomy of that document:
+
+- ``_id``: The name of the database.
+- ``_rev``: The current revision of the metadata.
+- ``shard_suffix``: A timestamp of the database's creation, marked as
+ seconds after the Unix epoch mapped to the codepoints for ASCII
+ numerals.
+- ``changelog``: History of the database's shards.
+- ``by_node``: List of shards on each node.
+- ``by_range``: On which nodes each shard is.
+
+To reflect the shard move in the metadata, there are three steps:
+
+1. Add appropriate changelog entries.
+2. Update the ``by_node`` entries.
+3. Update the ``by_range`` entries.
+
+As of this writing, this process must be done manually. **WARNING: Be
+very careful! Mistakes during this process can irreperably corrupt the
+cluster!**
-Nothing here, nothing there, a shard in my sleeve
--------------------------------------------------
+To add a shard to a node, add entries like this to the database
+metadata's ``changelog`` attribute:
-Start node2 and add it to the cluster. Check in ``/_membership`` that the
-nodes are talking with each other.
+.. code:: json1
-If you look in the directory ``data`` on node2, you will see that there is no
-directory called shards.
+ [
+ "add",
+ "{range}",
+ "{name}@{address}"
+ ]
-Use curl to change the ``_dbs/small`` node-local document for small, so it
-looks like this:
+*Note*: You can remove a node by specifying 'remove' instead of 'add'.
-.. code-block:: javascript
+Once you have figured out the new changelog entries, you will need to
+update the ``by_node`` and ``by_range`` to reflect who is storing what
+shards. The data in the changelog entries and these attributes must
+match. If they do not, the database may become corrupted.
+
+As an example, here is an updated version of the metadata above that
+adds shards to a second node called ``node2``:
+
+.. code:: json
{
"_id": "small",
@@ -179,123 +328,114 @@ looks like this:
}
}
-After PUTting this document, it's like magic: the shards are now on node2 too!
-We now have ``n=2``!
+Now you can ``PUT`` this new metadata:
-If the shards are large, then you can copy them over manually and only have
-CouchDB sync the changes from the last minutes instead.
+.. code:: bash
-.. _cluster/sharding/move:
+ $ curl -X PUT $COUCH_URL:5986/_dbs/{name} -d '{...}'
-Moving Shards
-=============
+Replicating from old to new
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Add, then delete
-----------------
+Because shards are just CouchDB databases, you can replicate them
+around. In order to make sure the new shard receives any updates the old
+one processed while you were updating its metadata, you should replicate
+the old shard to the new one:
-In the world of CouchDB there is no such thing as "moving" shards, only adding
-and removing shard replicas.
-You can add a new replica of a shard and then remove the old replica,
-thereby creating the illusion of moving.
-If you do this for a database that has ``n=1``,
-you might be caught by the following mistake:
+::
-#. Copy the shard onto a new node.
-#. Update the metadata to use the new node.
-#. Delete the shard on the old node.
-#. Oh, no!: You have lost all writes made between 1 and 2.
+ $ curl -X POST $COUCH_URL:5986/_replicate -d '{ \
+ "source": $OLD_SHARD_URL,
+ "target": $NEW_SHARD_URL
+ }'
-To avoid this mistake, you always want to make sure
-that both shards have been live for some time
-and that the shard on your new node is fully caught up
-before removing a shard on an old node.
-Since "moving" is a more conceptually (if not technically)
-accurate description of what you want to do,
-we'll use that word in this documentation as well.
+This will bring the new shard up to date so that we can safely delete
+the old one.
-Moving
-------
+Delete old shard
+~~~~~~~~~~~~~~~~
+
+You can remove the old shard either by deleting its file or by deleting
+it through the private 5986 port:
+
+.. code:: bash
+
+ # delete the file
+ rm $COUCH_DIR/data/shards/$OLD_SHARD
-When you get to ``n=3`` you should start moving the shards instead of adding
-more replicas.
+ # OR delete the database
+ curl -X DELETE $COUCH_URL:5986/$OLD_SHARD
-We will stop on ``n=2`` to keep things simple. Start node number 3 and add it to
-the cluster. Then create the directories for the shard on node3:
+Congratulations! You have manually added a new shard. By adding and
+removing database shards in this way, they can be moved between nodes.
-.. code-block:: bash
+Specifying database placement
+-----------------------------
- mkdir -p data/shards/00000000-7fffffff
+Database shards can be configured to live solely on specific nodes using
+placement rules.
-And copy over ``data/shards/00000000-7fffffff/small.1425202577.couch`` from
-node1 to node3. Do not move files between the shard directories as that will
-confuse CouchDB!
+First, each node must be labeled with a zone attribute. This defines
+which zone each node is in. You do this by editing the node’s document
+in the ``/nodes`` database, which is accessed through the “back-door”
+(5986) port. Add a key value pair of the form:
-Edit the database document in ``_dbs`` again. Make it so that node3 have a
-replica of the shard ``00000000-7fffffff``. Save the document and let CouchDB
-sync. If we do not do this, then writes made during the copy of the shard and
-the updating of the metadata will only have ``n=1`` until CouchDB has synced.
+::
-Then update the metadata document so that node2 no longer have the shard
-``00000000-7fffffff``. You can now safely delete
-``data/shards/00000000-7fffffff/small.1425202577.couch`` on node 2.
+ "zone": "{zone-name}"
-The changelog is nothing that CouchDB cares about, it is only for the admins.
-But for the sake of completeness, we will update it again. Use ``delete`` for
-recording the removal of the shard ``00000000-7fffffff`` from node2.
+Do this for all of the nodes in your cluster. For example:
-Start node4, add it to the cluster and do the same as above with shard
-``80000000-ffffffff``.
+.. code:: bash
-All documents added during this operation was saved and all reads responded to
-without the users noticing anything.
+ $ curl -X PUT $COUCH_URL:5986/_nodes/{name}@{address} \
+ -d '{ \
+ "_id": "{name}@{address}",
+ "_rev": "{rev}",
+ "zone": "{zone-name}"
+ }'
-.. _cluster/sharding/views:
+In the config file (local.ini or default.ini) of each node, define a
+consistent cluster-wide setting like:
-Views
-=====
+::
-The views need to be moved together with the shards. If you do not, then
-CouchDB will rebuild them and this will take time if you have a lot of
-documents.
+ [cluster]
+ placement = {zone-name-1}:2,{zone-name-2}:1
-The views are stored in ``data/.shards``.
+In this example, it will ensure that two replicas for a shard will be
+hosted on nodes with the zone attribute set to ``{zone-name-1}`` and one
+replica will be hosted on a new with the zone attribute set to
+``{zone-name-2}``.
-It is possible to not move the views and let CouchDB rebuild the view every
-time you move a shard. As this can take quite some time, it is not recommended.
+Note that you can also use this system to ensure certain nodes in the
+cluster do not host any replicas for newly created databases, by giving
+them a zone attribute that does not appear in the ``[cluster]``
+placement string.
-.. _cluster/sharding/preshard:
+You can also specify zones on a per-database basis by specifying the
+zone as a query parameter when the database is created:
-Reshard? No, Preshard!
-======================
+.. code:: bash
-Reshard? Nope. It cannot be done. So do not create databases with too few
-shards.
+ curl -X PUT $COUCH_URL:5984/{dbName}?zone={zone}
-If you can not scale out more because you set the number of shards too low, then
-you need to create a new cluster and migrate over.
+Resharding
+----------
-#. Build a cluster with enough nodes to handle one copy of your data.
-#. Create a database with the same name, n=1 and with enough shards so you do
- not have to do this again.
-#. Set up 2 way replication between the 2 clusters.
-#. Let it sync.
-#. Tell clients to use both the clusters.
-#. Add some nodes to the new cluster and add them as replicas.
-#. Remove some nodes from the old cluster.
-#. Repeat 6 and 7 until you have enough nodes in the new cluster to have 3
- replicas of every shard.
-#. Redirect all clients to the new cluster
-#. Turn off the 2 way replication between the clusters.
-#. Shut down the old cluster and add the servers as new nodes to the new
- cluster.
-#. Relax!
+Shard settings for databases can only be set when the database is
+created, precluding live resharding. Instead, to reshard a database, it
+must be regenerated. Here are the steps:
-Creating more shards than you need and then move the shards around is called
-presharding. The number of shards you need depends on how much data you are
-going to store. But, creating too many shards increases the complexity without
-any real gain. You might even get lower performance. As an example of this, we
-can take the author's (15 year) old lab server. It gets noticeably slower with
-more than one shard and high load, as the hard drive must seek more.
+1. Create a temporary database with the desired shard settings.
+2. Replicate the primary database to the temporary. Multiple
+ replications may be required if the primary database is under active
+ use.
+3. Delete the primary database. **Make sure nobody is using it!**
+4. Recreate the primary database with the desired shard settings.
+5. Replicate the temporary back to the primary.
+6. Delete the temporary database.
-How many shards you should have depends, as always, on your use case and your
-hardware. If you do not know what to do, use the default of 8 shards.
+Once all steps have completed, the database can be used again. The
+cluster will create and distribute its shards according to placement
+rules automatically.
[couchdb-documentation] 16/18: fix linter
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit f8bc9005ffd434d42e40f50bc2d2faf5b5c87228
Author: Jan Lehnardt <ja...@apache.org>
AuthorDate: Sat Jul 21 19:29:42 2018 +0200
fix linter
---
src/cluster/sharding.rst | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/cluster/sharding.rst b/src/cluster/sharding.rst
index 5aa81c5..2a72ef2 100644
--- a/src/cluster/sharding.rst
+++ b/src/cluster/sharding.rst
@@ -89,9 +89,9 @@ once a `quorum
<https://en.wikipedia.org/wiki/Quorum_(distributed_computing)>`__ of
database nodes have responded; 2, by default. The default required size
of a quorum is equal to ``r=w=((n+1)/2)`` where ``r`` refers to the size
-of a read quorum, ``w`` refers to the size of a write quorum, and ``n`` refers to
-the number of replicas of each shard. In a default cluster where ``n``
-is 3, ``((n+1)/2)`` would be 2.
+of a read quorum, ``w`` refers to the size of a write quorum, and ``n``
+refers to the number of replicas of each shard. In a default cluster where
+``n`` is 3, ``((n+1)/2)`` would be 2.
.. note::
Each node in a cluster can be a coordinating node for any one
[couchdb-documentation] 17/18: lint and ref and multiline labels
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit 3efe7648090f4dde0d73bf6fd0d4e91cc4b50325
Author: Jan Lehnardt <ja...@apache.org>
AuthorDate: Sat Jul 21 19:57:16 2018 +0200
lint and ref and multiline labels
---
src/cluster/sharding.rst | 26 ++++++++++++--------------
1 file changed, 12 insertions(+), 14 deletions(-)
diff --git a/src/cluster/sharding.rst b/src/cluster/sharding.rst
index 2a72ef2..862e326 100644
--- a/src/cluster/sharding.rst
+++ b/src/cluster/sharding.rst
@@ -142,21 +142,19 @@ they must be moved manually.
Moving shards between nodes in a cluster involves the following steps:
-0. :ref:`Ensure the target node has joined the cluster
- <cluster/nodes/add>`.
-1. :ref:`Copy the shard(s) and any secondary index shard(s) onto the
- target node <cluster/sharding/copying>`.
+0. :ref:`Ensure the target node has joined the cluster <cluster/nodes/add>`.
+1. Copy the shard(s) and any secondary
+ :ref:`index shard(s) onto the target node <cluster/sharding/copying>`.
2. :ref:`Set the target node to maintenance mode <cluster/sharding/mm>`.
-3. :ref:`Update cluster metadata to reflect the new target shard(s)
- <cluster/sharding/add-shard>`.
-4. :ref:`Monitor internal replication to ensure up-to-date shard(s)
- <cluster/sharding/verify>`.
-5. :ref:`Clear the target node's maintenance mode
- <cluster/sharding/mm-2>`.
-6. :ref:`Update cluster metadata again to remove the source shard(s)
- <cluster/sharding/remove-shard>`
-7. :ref:`Remove the shard file(s) and secondary index file(s) from the
- source node <cluster/sharding/remove-shard-files>`.
+3. Update cluster metadata
+ :ref:`to reflect the new target shard(s) <cluster/sharding/add-shard>`.
+4. Monitor internal replication
+ :ref:`to ensure up-to-date shard(s) <cluster/sharding/verify>`.
+5. :ref:`Clear the target node's maintenance mode <cluster/sharding/mm-2>`.
+6. Update cluster metadata again
+ :ref:`to remove the source shard(s)<cluster/sharding/remove-shard>`
+7. Remove the shard file(s) and secondary index file(s)
+ :ref:`from the source node <cluster/sharding/remove-shard-files>`.
.. _cluster/sharding/copying:
[couchdb-documentation] 11/18: Configure cluster Erlang port range
using vm.args
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit d35007288ea6844095d86d99488761fca6d34401
Author: Aleksandar Kodzhabashev <d3...@users.noreply.github.com>
AuthorDate: Wed Jul 18 17:32:33 2018 +0300
Configure cluster Erlang port range using vm.args
Ports specified in sys.config as per previous documentation seem to be ignored by CouchDB
---
src/cluster/setup.rst | 29 +++++++----------------------
1 file changed, 7 insertions(+), 22 deletions(-)
diff --git a/src/cluster/setup.rst b/src/cluster/setup.rst
index d7fb40f..683062f 100644
--- a/src/cluster/setup.rst
+++ b/src/cluster/setup.rst
@@ -119,31 +119,16 @@ To close the shells, run in both:
Make CouchDB use the open ports.
--------------------------------
-Open ``sys.config``, on all nodes, and add ``inet_dist_listen_min, 9100`` and
-``inet_dist_listen_max, 9200`` like below:
+Open ``vm.args``, on all nodes, and add ``-kernel inet_dist_listen_min 9100``
+and ``-kernel inet_dist_listen_max 9200`` like below:
.. code-block:: erlang
- [
- {lager, [
- {error_logger_hwm, 1000},
- {error_logger_redirect, true},
- {handlers, [
- {lager_console_backend, [debug, {
- lager_default_formatter,
- [
- date, " ", time,
- " [", severity, "] ",
- node, " ", pid, " ",
- message,
- "\n"
- ]
- }]}
- ]},
- {inet_dist_listen_min, 9100},
- {inet_dist_listen_max, 9200}
- ]}
- ].
+ -name ...
+ -setcookie ...
+ ...
+ -kernel inet_dist_listen_max 9100
+ -kernel inet_dist_listen_max 9200
.. _cluster/setup/wizard:
[couchdb-documentation] 15/18: add description of how quorum sizes
are calculated
Posted by wo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wohali pushed a commit to branch 2.2.0-release-notes
in repository https://gitbox.apache.org/repos/asf/couchdb-documentation.git
commit d005d14a0dde86119d15ebe15fa6dc0423cef5ed
Author: Diana Thayer <ga...@gmail.com>
AuthorDate: Fri Jul 20 08:38:20 2018 -0700
add description of how quorum sizes are calculated
---
src/cluster/sharding.rst | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/src/cluster/sharding.rst b/src/cluster/sharding.rst
index 51b1630..5aa81c5 100644
--- a/src/cluster/sharding.rst
+++ b/src/cluster/sharding.rst
@@ -87,7 +87,11 @@ the other nodes that have the relevant data, which may or may not
include itself. The coordinating node sends a response to the client
once a `quorum
<https://en.wikipedia.org/wiki/Quorum_(distributed_computing)>`__ of
-database nodes have responded; 2, by default.
+database nodes have responded; 2, by default. The default required size
+of a quorum is equal to ``r=w=((n+1)/2)`` where ``r`` refers to the size
+of a read quorum, ``w`` refers to the size of a write quorum, and ``n`` refers to
+the number of replicas of each shard. In a default cluster where ``n``
+is 3, ``((n+1)/2)`` would be 2.
.. note::
Each node in a cluster can be a coordinating node for any one