[GitHub] [couchdb] nickva commented on a diff in pull request #4410: Mango covering JSON indexes RFC

["nickva (via GitHub)" <gi...@apache.org>]

nickva commented on code in PR #4410:
URL: https://github.com/apache/couchdb/pull/4410#discussion_r1091101499


##########
src/docs/rfcs/018-mango-covering-json-index.md:
##########
@@ -0,0 +1,360 @@
+---
+name: Formal RFC
+about: Submit a formal Request For Comments for consideration by the team.
+title: 'Support covering indexes when using Mango JSON (view) indexes'
+labels: rfc, discussion
+assignees: ''
+
+---
+
+[NOTE]: # ( ^^ Provide a general summary of the RFC in the title above. ^^ )
+
+# Introduction
+
+## Abstract
+
+[NOTE]: # ( Provide a 1-to-3 paragraph overview of the requested change. )
+[NOTE]: # ( Describe what problem you are solving, and the general approach. )
+
+Covering indexes are used to reduce the time the database takes to respond to
+queries. An index "covers" a query when the query only requires fields that are
+in the index (in this way, "covering" is a property of index and query
+combined). When this is the case, the database doesn't need to consult primary
+data and can return results for the query from only the index. In more familiar
+CouchDB terminology, this is equivalent to querying a view with
+`include_docs=false`.
+
+When evaluating a query, Mango currently doesn't use the concept of covering
+indexes; even if a query could be answered without reading each result's full
+JSON document, Mango will still read it. This makes it impossible for Mango to
+return data as quickly as the underlying view.
+
+My benchmarking shows that Mango can answer at the same rate as the underlying
+view index. It currently runs at the same pace as calling the view with
+`include_docs=true`. Preliminary modifications to Mango showed that, with
+covering index support and a query that can use it, Mango can stream results
+as quickly as the underlying view. Adding covering indexes therefore increases
+the production use-cases Mango can support substantially.
+
+There are likely two phases to this:
+
+- Enable covering indexing processing for current indexes (ie, over view keys).
+- Allow Mango view indexes to include extra data from documents, storing it in
+  the `value` of the view. Support use of this extra data within the covering
+  indexes feature.
+
+### Out of scope
+
+This proposal only covers adding covering indexes to JSON indexes and not text
+indexes. The aim is to reduce the need for CouchDB users to run separate
+processes, such as Lucene, to get improved querying performance and capability.
+
+We do not aim to replicate `reduce` functionality from views, only to bring
+parity to non-reduced view execution speed (ie, when views are used to search
+the document space) to Mango.
+
+## Requirements Language
+
+[NOTE]: # ( Do not alter the section below. Follow its instructions. )
+
+The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+"SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in this
+document are to be interpreted as described in
+[RFC 2119](https://www.rfc-editor.org/rfc/rfc2119.txt).
+
+## Terminology
+
+[TIP]:  # ( Provide a list of any unique terms or acronyms, and their definitions here.)
+
+- Mango: CouchDB's Mongo inspired querying system.
+- View / JSON index: Mango index that uses the same index as Cloudant views.
+- Coordinator: the erlang process that handles doing a distributed query across
+    a CouchDB cluster.
+
+---
+
+# Detailed Description
+
+[NOTE]: # ( Describe the solution being proposed in greater detail. )
+[NOTE]: # ( Assume your audience has knowledge of, but not necessarily familiarity )
+[NOTE]: # ( with, the CouchDB internals. Provide enough context so that the reader )
+[NOTE]: # ( can make an informed decision about the proposal. )
+
+[TIP]:  # ( Artwork may be attached to the submission and linked as necessary. )
+[TIP]:  # ( ASCII artwork can also be included in code blocks, if desired. )
+
+This would take place within `mango_view_cursor.erl`. The key functions
+involved are the shard-level `view_cb/2`, the streaming result handler at the
+coordinator end (`handle_message/2`) and the `execute/3` function.
+
+## Mango JSON index selection
+
+A Mango JSON index is implemented as a view with a complex key. The first field
+in the index is the first entry in the complex key, the second field is the
+second key and so on. Even indexes with one field use a complex key with length
+`1`.
+
+When choosing a JSON index to use for a query, there are a couple of things that
+are important to covering indexes.
+
+Firstly, note there are certain predicate operators that can be used with an
+index, currently: `$lt`, $lte`, `$eq`, $gte` and `$gt`. These can easily be

Review Comment:
   Missing '`` for `$lte`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org