Proposal: Computed Indices in Mango Query Server

[Norbert Nemec <No...@gmx.de>]

Hi there,

I am quite new to the world of CouchDB and I know that this is quite a 
fundamental proposal, but I very much hope that it will still be 
received and discussed openly:

The Mango Query Server aims at offering a simpler alternative to 
MapReduce queries. As it stands now, it is also fundamentally limited in 
terms of expressiveness an performance. Indices can be defined only over 
a plain set of fields with none of the possibilities that a map function 
offers. Selectors allow powerful combinations, but require to perform 
much of the computational effort at query time. The $regex operator, for 
example even contains a warning about this in the documentation.

My proposal would be to add an alternative way to define indices by 
explicitly providing a design document map function. The 'fields' 
argument of such a "computed index" would not have to exist as an actual 
field in the database, but would be made available as a "computed field" 
for regular use in subsequent find requests.

Ultimately, this approach would bridge the gap between the 
simple-but-limited Mango queries and the powerful-but-unwieldy MapReduce 
queries. Rather than having to decide between both approaches, 
developers could start with the simple Mango approach and then just 
learn one more concept if they need the full power.

(It is my understanding that the currently recommended approach is to 
add computed fields to the documents directly at creation time. Though 
this is a workaround for the limitations of selectors, it does not offer 
any guarantees for internal consistency of a database and mixes the 
concerns of content creation with those of retrieval.)

Thank you for your careful consideration. I am looking forward to 
constructive discussion!

Greetings,
Norbert Nemec

Re: Proposal: Computed Indices in Mango Query Server

[Norbert Nemec <No...@gmx.de>]

Thanks for the encouragement!

-> https://issues.apache.org/jira/browse/COUCHDB-3280


Am 24/01/2017 um 6:52 schrieb Adam Kocoloski:
> Hi Norbert, I quite like this proposal. I\u2019d suggested something similar a few years back when we were first designing Mango but we never got around to implementing it. I\u2019d defer to Paul, Tony or anyone else deeply familiar with Mango on the level of effort required to establish this linkage, but it doesn\u2019t seem crazy. If you like, please create an issue in JIRA:
>
> https://issues.apache.org/jira/browse/COUCHDB
>
> You\u2019ll need an account first if you don\u2019t already have one:
>
> https://issues.apache.org/jira/secure/Signup!default.jspa
>
> Cheers, Adam
>
>> On Jan 20, 2017, at 1:10 PM, Norbert Nemec <No...@gmx.de> wrote:
>>
>> Hi there,
>>
>> I am quite new to the world of CouchDB and I know that this is quite a fundamental proposal, but I very much hope that it will still be received and discussed openly:
>>
>> The Mango Query Server aims at offering a simpler alternative to MapReduce queries. As it stands now, it is also fundamentally limited in terms of expressiveness an performance. Indices can be defined only over a plain set of fields with none of the possibilities that a map function offers. Selectors allow powerful combinations, but require to perform much of the computational effort at query time. The $regex operator, for example even contains a warning about this in the documentation.
>>
>> My proposal would be to add an alternative way to define indices by explicitly providing a design document map function. The 'fields' argument of such a "computed index" would not have to exist as an actual field in the database, but would be made available as a "computed field" for regular use in subsequent find requests.
>>
>> Ultimately, this approach would bridge the gap between the simple-but-limited Mango queries and the powerful-but-unwieldy MapReduce queries. Rather than having to decide between both approaches, developers could start with the simple Mango approach and then just learn one more concept if they need the full power.
>>
>> (It is my understanding that the currently recommended approach is to add computed fields to the documents directly at creation time. Though this is a workaround for the limitations of selectors, it does not offer any guarantees for internal consistency of a database and mixes the concerns of content creation with those of retrieval.)
>>
>> Thank you for your careful consideration. I am looking forward to constructive discussion!
>>
>> Greetings,
>> Norbert Nemec

Re: Proposal: Computed Indices in Mango Query Server

[Adam Kocoloski <ko...@apache.org>]

Hi Norbert, I quite like this proposal. I’d suggested something similar a few years back when we were first designing Mango but we never got around to implementing it. I’d defer to Paul, Tony or anyone else deeply familiar with Mango on the level of effort required to establish this linkage, but it doesn’t seem crazy. If you like, please create an issue in JIRA:

https://issues.apache.org/jira/browse/COUCHDB

You’ll need an account first if you don’t already have one:

https://issues.apache.org/jira/secure/Signup!default.jspa

Cheers, Adam

> On Jan 20, 2017, at 1:10 PM, Norbert Nemec <No...@gmx.de> wrote:
> 
> Hi there,
> 
> I am quite new to the world of CouchDB and I know that this is quite a fundamental proposal, but I very much hope that it will still be received and discussed openly:
> 
> The Mango Query Server aims at offering a simpler alternative to MapReduce queries. As it stands now, it is also fundamentally limited in terms of expressiveness an performance. Indices can be defined only over a plain set of fields with none of the possibilities that a map function offers. Selectors allow powerful combinations, but require to perform much of the computational effort at query time. The $regex operator, for example even contains a warning about this in the documentation.
> 
> My proposal would be to add an alternative way to define indices by explicitly providing a design document map function. The 'fields' argument of such a "computed index" would not have to exist as an actual field in the database, but would be made available as a "computed field" for regular use in subsequent find requests.
> 
> Ultimately, this approach would bridge the gap between the simple-but-limited Mango queries and the powerful-but-unwieldy MapReduce queries. Rather than having to decide between both approaches, developers could start with the simple Mango approach and then just learn one more concept if they need the full power.
> 
> (It is my understanding that the currently recommended approach is to add computed fields to the documents directly at creation time. Though this is a workaround for the limitations of selectors, it does not offer any guarantees for internal consistency of a database and mixes the concerns of content creation with those of retrieval.)
> 
> Thank you for your careful consideration. I am looking forward to constructive discussion!
> 
> Greetings,
> Norbert Nemec