You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2019/01/25 21:56:03 UTC
[GitHub] flimzy commented on a change in pull request #385: Documentation
for partitioned dbs
flimzy commented on a change in pull request #385: Documentation for partitioned dbs
URL: https://github.com/apache/couchdb-documentation/pull/385#discussion_r251147106
##########
File path: src/partitioned-dbs/index.rst
##########
@@ -0,0 +1,378 @@
+.. Licensed under the Apache License, Version 2.0 (the "License"); you may not
+.. use this file except in compliance with the License. You may obtain a copy of
+.. the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+.. WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+.. License for the specific language governing permissions and limitations under
+.. the License.
+
+.. _partitioned-dbs:
+
+=====================
+Partitioned Databases
+=====================
+
+As a means to introducing partitioned databases we'll consider a motivating
+use case to describe the benefits of this feature. For this example we'll
+consider a database that stores readings from a large network of soil
+moisture sensors.
+
+.. note::
+ Before reading this document you should be familiar with the
+ :ref:`theory <cluster/theory>` of :ref:`sharding <cluster/sharding>`
+ in CouchDB.
+
+
+Traditionally, a document in this database may have something like the
+following structure:
+
+.. code-block:: javascript
+
+ {
+ "_id": "sensor-reading-ca33c748-2d2c-4ed1-8abf-1bca4d9d03cf",
+ "_rev":"1-14e8f3262b42498dbd5c672c9d461ff0",
+ "sensor_id": "sensor-260",
+ "location": [41.6171031, -93.7705674],
+ "field_name": "Bob's Corn Field #5",
+ "readings": [
+ ["2019-01-21T00:00:00", 0.15],
+ ["2019-01-21T06:00:00", 0.14],
+ ["2019-01-21T12:00:00", 0.16],
+ ["2019-01-21T18:00:00", 0.11]
+ ]
+ }
+
+
+.. note::
+ While this example uses IoT sensors, the main thing to consider is that
+ there is a logical grouping of documents. Similar use cases might be
+ documents grouped by user or scientific data grouped by experiment.
+
+
+So we've got a bunch of sensors, all grouped by the field they monitor
+along with their readouts for a given day (or other appropriate time period).
+
+Along with our documents we might expect to have two secondary indexes
+for querying our database that might look something like:
+
+.. code-block:: javascript
+
+ function(doc) {
+ if(doc._id.indexOf("sensor-reading-") != 0) {
+ return;
+ }
+ for(var r in doc.readings) {
+ emit([doc.sensor_id, r[0]], r[1])
+ }
+ }
+
+and:
+
+.. code-block:: javascript
+
+ function(doc) {
+ if(doc._id.indexOf("sensor-reading-") != 0) {
+ return;
+ }
+ emit(doc.field_name, doc.sensor_id)
+ }
+
+With these two indexes defined we can easily find all requests for a given
+sensor or list all sensors in a given field.
+
+Unfortunately, in CouchDB when we read from either of these indexes it
+requires finding a copy of every shard and asking for any documents related
+to the particular sensor or field. This means that as our database scales
+up the number of shards, the more work every index request must perform.
+Fortunately for you dear reader, partitioned databases were created to solve
+this precise problem.
+
+
+What is a partition?
+====================
+
+In the previous section we introduced a hypothetical database that contains
+sensor readings from an IoT field monitoring service. In this particular
+use case it's quite logical to group all documents by their ``sensor_id``
+field. In this case we would call the sensor_id the partition.
+
+A good partition has two basic properties. First, it should have a high
+cardinality. That is, there is a large number of values for the partition.
+A database that has a single partition would be an anti-pattern for this
+feature. Secondly, the amount of data per partition should be "small". The
+general recommendation is to limit individual partitions to less than ten
+gigabytes of data. Which for the example sensor documents equates to roughly
+60,000 years of data.
+
+
+Why use partitions?
+===================
+
+The primary benefit of using partitioned databases is for the performance
+of partitioned queries. Large databases with lots of documents often
+have a similar pattern where there are groups of related documents that
+are queried together often.
+
+By using partitions we can execute queries against these individual groups
+of documents more efficiently by placing the entire group within a specific
+shard on disk. Thus the view engine only has to consult one copy of the
+given shard range when executing a query instead of the normal requirement
+having to execute the query across all ``Q`` shards in the database.
+
+
+Partitions By Example
+=====================
+
+To create a partitioned database we simply need to pass a query string
+parameter.
+
+.. code-block:: bash
+
+ shell> curl -X PUT http://127.0.0.1:5984/my_new_db?partitioned=true
+ {"ok":true}
+
+To see that our database is partitioned we can look at the database
+information:
+
+.. code-block:: bash
+
+ shell> curl http://127.0.0.1:5984/my_new_db
+ {
+ "cluster": {
+ "n": 3,
+ "q": 8,
+ "r": 2,
+ "w": 2
+ },
+ "compact_running": false,
+ "data_size": 0,
+ "db_name": "my_new_db",
+ "disk_format_version": 7,
+ "disk_size": 66784,
+ "doc_count": 0,
+ "doc_del_count": 0,
+ "instance_start_time": "0",
+ "other": {
+ "data_size": 0
+ },
+ "props": {
+ "partitioned": true
+ },
+ "purge_seq": "0-g1AAAAFDeJzLYWBg4M...",
+ "sizes": {
+ "active": 0,
+ "external": 0,
+ "file": 66784
+ },
+ "update_seq": "0-g1AAAAFDeJzLYWBg4M..."
+ }
+
+
+You'll now see that the ``"props"`` member contains ``"partitioned": true``.
+
+.. note::
+
+ The format for document ids in a partitioned database is
+ ``partition:docid``. Every regular document (i.e., everything
+ except design and local documents) in a partitioned database
+ must follow this format.
+
+Now that we've created a partitioned database its time to add some documents.
+Using our earlier example we could do this as such:
+
+.. code-block:: bash
+
+ shell> cat doc.json
+ {
+ "_id": "sensor-260:sensor-reading-ca33c748-2d2c-4ed1-8abf-1bca4d9d03cf",
+ "sensor_id": "sensor-260",
+ "location": [41.6171031, -93.7705674],
+ "field_name": "Bob's Corn Field #5",
+ "readings": [
+ ["2019-01-21T00:00:00", 0.15],
+ ["2019-01-21T06:00:00", 0.14],
+ ["2019-01-21T12:00:00", 0.16],
+ ["2019-01-21T18:00:00", 0.11]
+ ]
+ }
+ shell> $ curl -X POST -H "Content-Type: application/json" \
+ http://127.0.0.1:5984/my_new_db -d @doc.json
+ {
+ "ok": true,
+ "id": "sensor-260:sensor-reading-ca33c748-2d2c-4ed1-8abf-1bca4d9d03cf",
+ "rev": "1-05ed6f7abf84250e213fcb847387f6f5"
+ }
+
+The only change required to the first example document is that we are now
+including the partition name in the document id by prepending the
+old id separated by a colon.
+
+.. note::
+
+ The partition name in the document id is not magical. Internally
+ the database is simply using only the partition for hashing
+ the document to a given shard instead of the entire document id.
+
+Working with documents in a partitioned database is no different than
+a non-partitioned database. All APIs are available and existing client
+code will all work seamlessly.
+
+Now that we have created a document we can get some info about the partition
+containing the document:
+
+.. code-block:: bash
+
+ shell> curl http://127.0.0.1:5984/my_new_db/_partition/sensor-260
+ {
+ "db_name": "my_new_db",
+ "doc_count": 1,
+ "doc_del_count": 0,
+ "partition": "sensor-260",
+ "sizes": {
+ "active": 244,
+ "external": 347
+ }
+ }
+
+And we can also list all documents in a partition:
+
+.. code-block:: bash
+
+ shell> curl http://127.0.0.1:5984/my_new_db/_partition/sensor-260/_all_docs
+ {"total_rows": 1, "offset": 0, "rows":[
+ {
+ "id":"sensor-260:sensor-reading-ca33c748-2d2c-4ed1-8abf-1bca4d9d03cf",
+ "key":"sensor-260:sensor-reading-ca33c748-2d2c-4ed1-8abf-1bca4d9d03cf",
+ "value": {"rev": "1-05ed6f7abf84250e213fcb847387f6f5"}
+ }
+ ]}
+
+Note that we can use all of the normal bells and whistles available to
+``_all_docs`` requests. Accessing ``_all_docs`` through the
+``/dbname/_partition/name/_all_docs`` endpoint is mostly a convenience
+so that requests are guaranteed to be scoped to a given partition. Users
+are free to use the normal ``/dbname/_all_docs`` to read documents from
+multiple partitions.
+
+Next, we'll create a design document containing our index for
+getting all readings from a given sensor. The map function is similar to
+our earlier example except we've accounted for the change in the document
+id.
+
+.. code-block:: javascript
+
+ function(doc) {
+ if(doc._id.indexOf(":sensor-reading-") < 0) {
+ return;
+ }
+ for(var r in doc.readings) {
+ emit([doc.sensor_id, r[0]], r[1])
+ }
+ }
+
+We can go ahead and upload our design document and try out a partitioned
+query:
+
+.. code-block:: bash
+
+ shell> cat ddoc.json
+ {
+ "_id": "_design/sensor-readings",
+ "views": {
+ "by_sensor": {
+ "map": "function(doc) { ... }"
+ }
+ }
+ }
+ shell> curl http://127.0.0.1:5984/my_new_db/_partition/sensor-260/_design/sensor-readings/_view/by_sensor
+ {"total_rows":4,"offset":0,"rows":[
+ {"id":"sensor-260:sensor-reading-ca33c748-2d2c-4ed1-8abf-1bca4d9d03cf","key":["sensor-260","0"],"value":null},
+ {"id":"sensor-260:sensor-reading-ca33c748-2d2c-4ed1-8abf-1bca4d9d03cf","key":["sensor-260","1"],"value":null},
+ {"id":"sensor-260:sensor-reading-ca33c748-2d2c-4ed1-8abf-1bca4d9d03cf","key":["sensor-260","2"],"value":null},
+ {"id":"sensor-260:sensor-reading-ca33c748-2d2c-4ed1-8abf-1bca4d9d03cf","key":["sensor-260","3"],"value":null}
+ ]}
+
+Hooray! Our first partitioned query. For experienced users that may not
Review comment:
```suggestion
Hooray! Our first partitioned query. For experienced users, that may not
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services