[GitHub] [couchdb] jcoglan opened a new pull request #3912: Feature: build Mango indexes from dynamic expressions

[GitBox <gi...@apache.org>]

jcoglan opened a new pull request #3912:
URL: https://github.com/apache/couchdb/pull/3912


   ## Overview
   
   This PR represents a prototype of a feature @janl and I have been working on. It lets us build indexes on dynamically computed values inside Mango, which would normally require writing a JS view. It does this by extending the syntax for indexes so that as well as the `asc` and `desc` sort directions, we allow an _expression_ to define a _virtual field_, for example:
   
   ```json
   "fields": [
     { "foo_words": { "$jq": ".foo | split(\" \") | .[]" } }
   ]
   ```
   
   This definition means that the virtual field `foo_words` is generated by splitting the `foo` property on spaces and emitting each result. So if we have a document like:
   
   ```json
   {
     "foo": "a b c"
   }
   ```
   
   then this index lets us find that doc using the `_find` query `"foo_words": "b"`.
   
   This is a prototype we're presenting to see if the functionality is of interest, before we commit any more work to making it more production-ready. Our reasoning for using `jq` for this is:
   
   - It's a ready-built expression language, we don't need to build a lot of the same functionality ourselves
   - It addresses design issues we faced trying to come up with our own function definition syntax, e.g.:
     - How do we indicate that a function input is taken from doc property vs a literal value
     - How do we indicate that we want to use an array result as an index key vs using each member of the array as a key
     - How do we support composition of different functions to produce a result
     - jq has nice answers to these questions already
   - CouchDB users are likely to be familiar with jq so it's one less thing to learn, and they can experiment with it in their shell while designing their indexes
   - It's very concise, compare `.foo | split(" ") | .[]` to our `{ "$explode": { "$field": "foo", "$separator": " " } }`, which doesn't address the array vs elements problem
   
   That said, there is risk with adopting a native dependency and we fully understand if that's not a path others think we should go down. We're opening this to gauge interest in the idea of indexing on dynamic functions inside Mango, rather than whether we use jq specifically.
   
   ## Testing recommendations
   
   The Python test script included in the PR indicates how to use the functionality. You may need to augment the rebar script to add build flags for your environment; this was developed on macOS with `jq` installed via Homebrew.
   
   If we developed this further for production, we would want to add comprehensive unit tests for the `couch_jq` module to make sure it round-trips all JSON values correctly (I have verified this by hand but not written automated tests as such).
   
   If we decide to stick with jq then we should also fuzz-test the native code, and we should decide whether to vendor the `jq` codebase or compile against the system copy.
   
   There are also some warts in the implementation such as the addition of the virtual field into results based on the selector, which we'd need to come up with a cleaner solution for.
   
   ## Checklist
   
   - [ ] Code is written and works correctly
   - [ ] Changes are covered by tests
   - [ ] Any new configurable parameters are documented in `rel/overlay/etc/default.ini`
   - [ ] A PR for documentation changes has been made in https://github.com/apache/couchdb-documentation
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [couchdb] tonysun83 edited a comment on pull request #3912: Feature: build Mango indexes from dynamic expressions

[GitBox <gi...@apache.org>]

tonysun83 edited a comment on pull request #3912:
URL: https://github.com/apache/couchdb/pull/3912#issuecomment-1021665360


   Pretty neat feature! IIUC, these new virtual fields will usually lead to a full key range scan because the answer derived by from the jq query will be what the user wants.
   
   Ex: "Give me all docs where foo does not contain 1,2, or 3.":
   ```
   {"foo" : {"$nin": [1, 2, 3]}
   ```
   
   Currently, we don't use an index for this query, so we'll scan `_all_docs` and then run the filter.
   
   Now with this new feature, you can have a precompiled jq query that filters all `foo` fields that don't have 1,2,3:
   ```
     { "foo_words": { "$jq": ".foo | <some expression to get all all values not in [1,2,3]> | .[]" } }
   ```
   ```
   Doc1
   {"foo": "1"}
   Doc2
   {"foo": "4"}
   Doc3
   {"foo": "5"}
   ```
   The table with the new feature would look like:
   
   ```
   foo_words, 4
   foo_words, 5
   ```
   
   From the original query, the user wants 4,5 back. So the query would `{"foo_words" : {"$gt" : null}}`
   
   I'm thinking this would be more common than the user having a precompiled jq, filtering the results and then running another query on top of that. Perhaps we can think of just doing the full key range scan (from the emitted keys) for them to make things simpler?  Basically have something like `{"foo_words" : "$results"}`. 
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [couchdb] tonysun83 commented on pull request #3912: Feature: build Mango indexes from dynamic expressions

[GitBox <gi...@apache.org>]

tonysun83 commented on pull request #3912:
URL: https://github.com/apache/couchdb/pull/3912#issuecomment-1021665360


   Pretty neat feature! IIUC, these new virtual fields will usually lead to a full key range scan because the answer derived by from the jq query will be what the user wants.
   
   Ex: "Give me all docs where foo does not contain 1,2, or 3.":
   ```
   {"foo" : {"$nin": [1, 2, 3]}
   ```
   
   Currently, we don't use an index for this query, so we'll scan `_all_docs` and then run the filter.
   
   Now with this new feature, you can have a precompiled jq query that filters all `foo` fields that don't have 1,2,3:
   ```
     { "foo_words": { "$jq": ".foo | <some expression to get all all values not in [1,2,3]> | .[]" } }
   ```
   ```
   Doc1
   {"foo": "1"}
   Doc2
   {"foo": "4"}
   Doc3
   {"foo": "5"}
   ```
   The table with the new feature would look like:
   
   ```
   foo_words, 4
   foo_words, 5
   ```
   
   From the original query, the user wants 4,5 back. So the query would `{"foo_words" : {"$gt" : null}}
   
   I'm thinking this would be more common than the user having a precompiled jq, filtering the results and then running another query with one of the normal operators. Perhaps we can think of just doing the full key range scan for them to make things simpler? 
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [couchdb] tonysun83 edited a comment on pull request #3912: Feature: build Mango indexes from dynamic expressions

[GitBox <gi...@apache.org>]

tonysun83 edited a comment on pull request #3912:
URL: https://github.com/apache/couchdb/pull/3912#issuecomment-1021665360


   Pretty neat feature! IIUC, these new virtual fields will usually lead to a full key range scan because the answer derived by from the jq query will be what the user wants.
   
   Ex: "Give me all docs where foo does not contain 1,2, or 3.":
   ```
   {"foo" : {"$nin": [1, 2, 3]}
   ```
   
   Currently, we don't use an index for this query, so we'll scan `_all_docs` and then run the filter.
   
   Now with this new feature, you can have a precompiled jq query that filters all `foo` fields that don't have 1,2,3:
   ```
     { "foo_words": { "$jq": ".foo | <some expression to get all all values not in [1,2,3]> | .[]" } }
   ```
   ```
   Doc1
   {"foo": "1"}
   Doc2
   {"foo": "4"}
   Doc3
   {"foo": "5"}
   ```
   The table with the new feature would look like:
   
   ```
   foo_words, 4
   foo_words, 5
   ```
   
   From the original query, the user wants 4,5 back. So the query would `{"foo_words" : {"$gt" : null}}
   
   I'm thinking this would be more common than the user having a precompiled jq, filtering the results and then running another query on top of that. Perhaps we can think of just doing the full key range scan (from the emitted keys) for them to make things simpler? 
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org