You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Jan Lehnardt <ja...@apache.org> on 2021/11/23 17:27:08 UTC
Mango Functions & Values
Hi all,
I had some time playing with an idea that I wanted to see happen for a long time: value manipulation functions for Mango.
The goal here is to allow more flexible indexing with Mango. It’ll not be a super flexible solution like JS views, but I think it is a neat addition.
I’ve drafted things up with a small example patch and I’m looking for feedback whether y’all are interested in me turning this into a proper RFC:
https://gist.github.com/janl/e5469f6f08c9be0405f31451889d5030
h/t Adam for scope-creeping the “& Values” part into this ;) We can do that separately, if folks prefer.
Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
24/7 Observation for your CouchDB Instances:
https://opservatory.app
Re: Mango Functions & Values
Posted by Jan Lehnardt <ja...@apache.org>.
Hey all,
my colleague James took this a bit further and made a jq Nif and extended Mango so we can build indexes from jq expressions.
There is a demo PR here: https://github.com/apache/couchdb/pull/3912 <https://github.com/apache/couchdb/pull/3912>
We’d love your feedback,
Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
24/7 Observation for your CouchDB Instances:
https://opservatory.app
> On 23. Nov 2021, at 18:27, Jan Lehnardt <ja...@apache.org> wrote:
>
> Hi all,
>
> I had some time playing with an idea that I wanted to see happen for a long time: value manipulation functions for Mango.
>
> The goal here is to allow more flexible indexing with Mango. It’ll not be a super flexible solution like JS views, but I think it is a neat addition.
>
> I’ve drafted things up with a small example patch and I’m looking for feedback whether y’all are interested in me turning this into a proper RFC:
>
> https://gist.github.com/janl/e5469f6f08c9be0405f31451889d5030
>
> h/t Adam for scope-creeping the “& Values” part into this ;) We can do that separately, if folks prefer.
>
> Best
> Jan
> —
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
> 24/7 Observation for your CouchDB Instances:
> https://opservatory.app
>
Re: Mango Functions & Values
Posted by Jan Lehnardt <ja...@apache.org>.
Heya Glynn,
I’ve been playing with this at work a bit and a coworker suggested something
that looks quite a bit what you’re proposing. I’m happy to take the lead on
producing an RFC in the next couple of weeks, so we can hammer this out.
Ideally, we want to make it so that our declarative VDU can re-use the same
syntax and functions later, so it’s definitely worth spending a good amount
of time on the API: https://github.com/apache/couchdb/issues/1554
Best
Jan
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
24/7 Observation for your CouchDB Instances:
https://opservatory.app
> On 3. Dec 2021, at 17:49, Glynn Bird <gl...@apache.org> wrote:
>
> I think Mango needs _something_ to allow folks to process data between the
> document and the index - thanks for posting Jan!.
>
> I had a think about this and came up with another simpler syntax which is
> in this gist:
>
> https://gist.github.com/glynnbird/ad932de13eb0e276e6ffab5fcaa350f8
>
> Cheers
>
> Glynn
>
> On Fri, 26 Nov 2021 at 15:05, Adam Kocoloski <ko...@apache.org> wrote:
>
>> Hi Jan, good stuff. I think it’s worth expanding into an RFC if this is a
>> limitation that you see users bumping into frequently. Some thoughts:
>>
>> - We’ll want to address the case where a virtual field clashes with a
>> “physical" field in a doc. Should the value of the virtual field override
>> the one in the document? I think so.
>>
>> - I can imagine users wanting to apply these functions conditionally, e.g.
>> to do schema evolution on read where they’ve got older documents that
>> serialized a field in a different format than they’re currently using for
>> new inserts. I’m OK with “use a view” being our response to that case, but
>> figured I’d bring it up.
>>
>> - We’ve had occasional conversations about how to integrate views into
>> Mango so that _find could be a common endpoint for interrogating the DB.
>> Last time I thought about it a combination of “use_index” and a
>> registration of the keys and values from the view as virtual fields seemed
>> like a sane way to do it. It’s a separate topic but I’m wondering if the
>> query interface ends up looking similar for two features.
>>
>> Cheers, Adam
>>
>>> On Nov 23, 2021, at 12:27 PM, Jan Lehnardt <ja...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I had some time playing with an idea that I wanted to see happen for a
>> long time: value manipulation functions for Mango.
>>>
>>> The goal here is to allow more flexible indexing with Mango. It’ll not
>> be a super flexible solution like JS views, but I think it is a neat
>> addition.
>>>
>>> I’ve drafted things up with a small example patch and I’m looking for
>> feedback whether y’all are interested in me turning this into a proper RFC:
>>>
>>> https://gist.github.com/janl/e5469f6f08c9be0405f31451889d5030
>>>
>>> h/t Adam for scope-creeping the “& Values” part into this ;) We can do
>> that separately, if folks prefer.
>>>
>>> Best
>>> Jan
>>> —
>>> Professional Support for Apache CouchDB:
>>> https://neighbourhood.ie/couchdb-support/
>>>
>>> 24/7 Observation for your CouchDB Instances:
>>> https://opservatory.app
>>>
>>
>>
Re: Mango Functions & Values
Posted by Glynn Bird <gl...@apache.org>.
I think Mango needs _something_ to allow folks to process data between the
document and the index - thanks for posting Jan!.
I had a think about this and came up with another simpler syntax which is
in this gist:
https://gist.github.com/glynnbird/ad932de13eb0e276e6ffab5fcaa350f8
Cheers
Glynn
On Fri, 26 Nov 2021 at 15:05, Adam Kocoloski <ko...@apache.org> wrote:
> Hi Jan, good stuff. I think it’s worth expanding into an RFC if this is a
> limitation that you see users bumping into frequently. Some thoughts:
>
> - We’ll want to address the case where a virtual field clashes with a
> “physical" field in a doc. Should the value of the virtual field override
> the one in the document? I think so.
>
> - I can imagine users wanting to apply these functions conditionally, e.g.
> to do schema evolution on read where they’ve got older documents that
> serialized a field in a different format than they’re currently using for
> new inserts. I’m OK with “use a view” being our response to that case, but
> figured I’d bring it up.
>
> - We’ve had occasional conversations about how to integrate views into
> Mango so that _find could be a common endpoint for interrogating the DB.
> Last time I thought about it a combination of “use_index” and a
> registration of the keys and values from the view as virtual fields seemed
> like a sane way to do it. It’s a separate topic but I’m wondering if the
> query interface ends up looking similar for two features.
>
> Cheers, Adam
>
> > On Nov 23, 2021, at 12:27 PM, Jan Lehnardt <ja...@apache.org> wrote:
> >
> > Hi all,
> >
> > I had some time playing with an idea that I wanted to see happen for a
> long time: value manipulation functions for Mango.
> >
> > The goal here is to allow more flexible indexing with Mango. It’ll not
> be a super flexible solution like JS views, but I think it is a neat
> addition.
> >
> > I’ve drafted things up with a small example patch and I’m looking for
> feedback whether y’all are interested in me turning this into a proper RFC:
> >
> > https://gist.github.com/janl/e5469f6f08c9be0405f31451889d5030
> >
> > h/t Adam for scope-creeping the “& Values” part into this ;) We can do
> that separately, if folks prefer.
> >
> > Best
> > Jan
> > —
> > Professional Support for Apache CouchDB:
> > https://neighbourhood.ie/couchdb-support/
> >
> > 24/7 Observation for your CouchDB Instances:
> > https://opservatory.app
> >
>
>
Re: Mango Functions & Values
Posted by Adam Kocoloski <ko...@apache.org>.
Hi Jan, good stuff. I think it’s worth expanding into an RFC if this is a limitation that you see users bumping into frequently. Some thoughts:
- We’ll want to address the case where a virtual field clashes with a “physical" field in a doc. Should the value of the virtual field override the one in the document? I think so.
- I can imagine users wanting to apply these functions conditionally, e.g. to do schema evolution on read where they’ve got older documents that serialized a field in a different format than they’re currently using for new inserts. I’m OK with “use a view” being our response to that case, but figured I’d bring it up.
- We’ve had occasional conversations about how to integrate views into Mango so that _find could be a common endpoint for interrogating the DB. Last time I thought about it a combination of “use_index” and a registration of the keys and values from the view as virtual fields seemed like a sane way to do it. It’s a separate topic but I’m wondering if the query interface ends up looking similar for two features.
Cheers, Adam
> On Nov 23, 2021, at 12:27 PM, Jan Lehnardt <ja...@apache.org> wrote:
>
> Hi all,
>
> I had some time playing with an idea that I wanted to see happen for a long time: value manipulation functions for Mango.
>
> The goal here is to allow more flexible indexing with Mango. It’ll not be a super flexible solution like JS views, but I think it is a neat addition.
>
> I’ve drafted things up with a small example patch and I’m looking for feedback whether y’all are interested in me turning this into a proper RFC:
>
> https://gist.github.com/janl/e5469f6f08c9be0405f31451889d5030
>
> h/t Adam for scope-creeping the “& Values” part into this ;) We can do that separately, if folks prefer.
>
> Best
> Jan
> —
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
> 24/7 Observation for your CouchDB Instances:
> https://opservatory.app
>