You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Suraj Kumar <su...@inmobi.com> on 2014/05/30 12:57:14 UTC
Proposal for new feature: Auto Update Functions
[x-posted]
Hi,
This email is to propose a new feature in CouchDB, specifically, a new type
of design document, called "Auto Update Functions". The intention is to
describe this and see whether the community would like to see this feature
in CouchDB and if yes, then get into the act of preparing a patch for
CouchDB.
*What are "Auto Update Functions"?*
We often have use-cases where we want to introduce or mutate attribute in a
document based on a code computable state of the document. *Auto Update
Functions* will be invoked for each document being updated after VDU (if
any) passes. The return value of the AUF is persisted on disk instead of
the actual doc sent in the request.
This way, features such as appending the last modified timestamp to a
document being modified, or adding which user modified a document or
computing the 'state' of an object can all be done via Auto Update
Functions.
Currently, we solve these use-cases through an external agent which
subscribes to Changes API and writes back to the very documents. But this
brings in all sorts of operational headaches (how do we ensure this
external agent will be highly available? would it use a 'queue' to keep
track of messages to be processed? ultimately, Changes API does not give
all we need either (ex: time of modification, user who modified the doc,
old doc (too late?), etc.,).
*Auto Update Function structure*
At the design doc level, it would look much similar to a Validate Doc
Update function:
function (NewDoc, OldDoc, UserCtx) {
// Mutate NewDoc here
return NewDoc;
}
What are your thoughts, both from the use-case as well as
internals/performance of CouchDB about this?
Regards,
-Suraj
--
An Onion is the Onion skin and the Onion under the skin until the Onion
Skin without any Onion underneath.
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: Proposal for new feature: Auto Update Functions
Posted by Robert Samuel Newson <rn...@apache.org>.
I can’t think of a reason to object to enhancing document update handlers to return an array of documents, might enable some neat tricks beyond just supporting bulk updates too.
On 1 Jun 2014, at 08:48, Simon Metson <si...@cloudant.com> wrote:
> Wouldn't the alternative be making update functions work in bulk be safe?
>
>
> On Saturday, 31 May 2014 at 22:30, Robert Samuel Newson wrote:
>
>> The main trouble is loops. If couchdb edits the doc after the user edits it, this edit is also replicated out to wherever you replicate to, which will make this edit again, and then back again, and then back again.
>>
>> In couchdb, by design, one request causes one action, not multiple, not loops. We should think very carefully about changing that. This proposal (and its been proposed before in other guises) seems to suffer a fatal flaw.
>>
>> B.
>>
>> On 31 May 2014, at 20:12, Alexander Shorin <kx...@gmail.com> wrote:
>>> The problem with update functions that they cannot be applied for bulk
>>> updates. API reroute via reverse proxy cannot solve that too. I
>>> believe, proposed feature assumed to handle this case.
>>> --
>>> ,,,^..^,,,
>>>
>>>
>>> On Sat, May 31, 2014 at 11:06 PM, Joan Touzet <jo...@lrtw.org> wrote:
>>>> The typical solution to this is to use a reverse proxy or API layer to ensure that all client updates go through the required document update handler. It's unclear to me that this functionality native in CouchDB is necessary.
>>>>
>>>> If you decide to move ahead with implementation , keep in mind that the 1843-feature-bigcouch branch as that will be landing very soon. Any proposed patch should be compatible the fabric/chttpd-based approach and support single- and multi-node (BigCouch cluster) approaches.
>>>>
>>>> -Joan
>>>>
>>>> ----- Original Message -----
>>>> From: "Franck Eyraud" <fr...@nospam.yrnm.net>
>>>> To: dev@couchdb.apache.org
>>>> Cc: user@couchdb.apache.org
>>>> Sent: Saturday, May 31, 2014 8:34:57 AM
>>>> Subject: Re: Proposal for new feature: Auto Update Functions
>>>>
>>>> Le 30/05/2014 13:04, Dirkjan Ochtman a écrit :
>>>>> On Fri, May 30, 2014 at 12:57 PM, Suraj Kumar <su...@inmobi.com> wrote:
>>>>>> What are your thoughts, both from the use-case as well as
>>>>>> internals/performance of CouchDB about this?
>>>>>>
>>>>>
>>>>> How is this different from document update handlers?
>>>>>
>>>>
>>>> Update handlers must be called by the client to be used. Auto update
>>>> functions would be called even if the client directly POST/PUT a doc to
>>>> the DB (so they would be mandatory).
>>>>
>>>> At first sight a good idea, it seems to me that auto update functions
>>>> would cause problem during replication : the replicated doc might be
>>>> different from the original one.
>>>>
>>>> Franck
>
Re: Proposal for new feature: Auto Update Functions
Posted by Simon Metson <si...@cloudant.com>.
Wouldn't the alternative be making update functions work in bulk be safe?
On Saturday, 31 May 2014 at 22:30, Robert Samuel Newson wrote:
> The main trouble is loops. If couchdb edits the doc after the user edits it, this edit is also replicated out to wherever you replicate to, which will make this edit again, and then back again, and then back again.
>
> In couchdb, by design, one request causes one action, not multiple, not loops. We should think very carefully about changing that. This proposal (and its been proposed before in other guises) seems to suffer a fatal flaw.
>
> B.
>
> On 31 May 2014, at 20:12, Alexander Shorin <kx...@gmail.com> wrote:
> > The problem with update functions that they cannot be applied for bulk
> > updates. API reroute via reverse proxy cannot solve that too. I
> > believe, proposed feature assumed to handle this case.
> > --
> > ,,,^..^,,,
> >
> >
> > On Sat, May 31, 2014 at 11:06 PM, Joan Touzet <jo...@lrtw.org> wrote:
> > > The typical solution to this is to use a reverse proxy or API layer to ensure that all client updates go through the required document update handler. It's unclear to me that this functionality native in CouchDB is necessary.
> > >
> > > If you decide to move ahead with implementation , keep in mind that the 1843-feature-bigcouch branch as that will be landing very soon. Any proposed patch should be compatible the fabric/chttpd-based approach and support single- and multi-node (BigCouch cluster) approaches.
> > >
> > > -Joan
> > >
> > > ----- Original Message -----
> > > From: "Franck Eyraud" <fr...@nospam.yrnm.net>
> > > To: dev@couchdb.apache.org
> > > Cc: user@couchdb.apache.org
> > > Sent: Saturday, May 31, 2014 8:34:57 AM
> > > Subject: Re: Proposal for new feature: Auto Update Functions
> > >
> > > Le 30/05/2014 13:04, Dirkjan Ochtman a écrit :
> > > > On Fri, May 30, 2014 at 12:57 PM, Suraj Kumar <su...@inmobi.com> wrote:
> > > > > What are your thoughts, both from the use-case as well as
> > > > > internals/performance of CouchDB about this?
> > > > >
> > > >
> > > > How is this different from document update handlers?
> > > >
> > >
> > > Update handlers must be called by the client to be used. Auto update
> > > functions would be called even if the client directly POST/PUT a doc to
> > > the DB (so they would be mandatory).
> > >
> > > At first sight a good idea, it seems to me that auto update functions
> > > would cause problem during replication : the replicated doc might be
> > > different from the original one.
> > >
> > > Franck
Re: Proposal for new feature: Auto Update Functions
Posted by Suraj Kumar <su...@inmobi.com>.
On Sun, Jun 1, 2014 at 3:00 AM, Robert Samuel Newson <rn...@apache.org>
wrote:
> The main trouble is loops. If couchdb edits the doc after the user edits
> it, this edit is also replicated out to wherever you replicate to, which
> will make this edit again, and then back again, and then back again.
>
> In couchdb, by design, one request causes one action, not multiple, not
> loops. We should think very carefully about changing that. This proposal
> (and its been proposed before in other guises) seems to suffer a fatal flaw.
>
What if this is written to not interfere with replicator? ie., this is a
sequential update done just before doc is written to disk and this does not
happen in the replicator pathway at all (ie., when another
auto-update-handler of a remote machine does it, the nodes pulling this
change will store it as-is). Also, this means this edit is in the
synchronous path, just like how VDU calls are in the synchronous path.
does that make sense?
Regards,
-Suraj
--
An Onion is the Onion skin and the Onion under the skin until the Onion
Skin without any Onion underneath.
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: Proposal for new feature: Auto Update Functions
Posted by Jens Alfke <je...@couchbase.com>.
On May 31, 2014, at 2:30 PM, Robert Samuel Newson <rn...@apache.org> wrote:
> The main trouble is loops. If couchdb edits the doc after the user edits it, this edit is also replicated out to wherever you replicate to, which will make this edit again, and then back again, and then back again.
I agree — this has some problematic interactions with the replicator.
If a document is replicated to two databases, and one of them uses this feature to modify the document, the destination databases now contain two documents with the same doc and revision ID but different contents. That explicitly breaks one of the invariants the replicator relies on.
(The same issue applies if both databases use this feature, but don’t make identical changes. This could happen even if they’re both using identical auto-update functions, e.g. if the auto-update adds a timestamp or a randomly-generated identifier.)
This can be averted if the auto-update is only used in the case where new_edits=true, i.e. a normal PUT. Both because the replicator doesn’t use this mode, and because this mode assigns a new revID, so it can be based on the post-processed doc’s contents.
—Jens
Re: Proposal for new feature: Auto Update Functions
Posted by Suraj Kumar <su...@inmobi.com>.
On Sun, Jun 1, 2014 at 3:00 AM, Robert Samuel Newson <rn...@apache.org>
wrote:
> The main trouble is loops. If couchdb edits the doc after the user edits
> it, this edit is also replicated out to wherever you replicate to, which
> will make this edit again, and then back again, and then back again.
>
> In couchdb, by design, one request causes one action, not multiple, not
> loops. We should think very carefully about changing that. This proposal
> (and its been proposed before in other guises) seems to suffer a fatal flaw.
>
What if this is written to not interfere with replicator? ie., this is a
sequential update done just before doc is written to disk and this does not
happen in the replicator pathway at all (ie., when another
auto-update-handler of a remote machine does it, the nodes pulling this
change will store it as-is). Also, this means this edit is in the
synchronous path, just like how VDU calls are in the synchronous path.
does that make sense?
Regards,
-Suraj
--
An Onion is the Onion skin and the Onion under the skin until the Onion
Skin without any Onion underneath.
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: Proposal for new feature: Auto Update Functions
Posted by Robert Samuel Newson <rn...@apache.org>.
The main trouble is loops. If couchdb edits the doc after the user edits it, this edit is also replicated out to wherever you replicate to, which will make this edit again, and then back again, and then back again.
In couchdb, by design, one request causes one action, not multiple, not loops. We should think very carefully about changing that. This proposal (and its been proposed before in other guises) seems to suffer a fatal flaw.
B.
On 31 May 2014, at 20:12, Alexander Shorin <kx...@gmail.com> wrote:
> The problem with update functions that they cannot be applied for bulk
> updates. API reroute via reverse proxy cannot solve that too. I
> believe, proposed feature assumed to handle this case.
> --
> ,,,^..^,,,
>
>
> On Sat, May 31, 2014 at 11:06 PM, Joan Touzet <jo...@lrtw.org> wrote:
>> The typical solution to this is to use a reverse proxy or API layer to ensure that all client updates go through the required document update handler. It's unclear to me that this functionality native in CouchDB is necessary.
>>
>> If you decide to move ahead with implementation , keep in mind that the 1843-feature-bigcouch branch as that will be landing very soon. Any proposed patch should be compatible the fabric/chttpd-based approach and support single- and multi-node (BigCouch cluster) approaches.
>>
>> -Joan
>>
>> ----- Original Message -----
>> From: "Franck Eyraud" <fr...@nospam.yrnm.net>
>> To: dev@couchdb.apache.org
>> Cc: user@couchdb.apache.org
>> Sent: Saturday, May 31, 2014 8:34:57 AM
>> Subject: Re: Proposal for new feature: Auto Update Functions
>>
>> Le 30/05/2014 13:04, Dirkjan Ochtman a écrit :
>>> On Fri, May 30, 2014 at 12:57 PM, Suraj Kumar <su...@inmobi.com> wrote:
>>>> What are your thoughts, both from the use-case as well as
>>>> internals/performance of CouchDB about this?
>>> How is this different from document update handlers?
>> Update handlers must be called by the client to be used. Auto update
>> functions would be called even if the client directly POST/PUT a doc to
>> the DB (so they would be mandatory).
>>
>> At first sight a good idea, it seems to me that auto update functions
>> would cause problem during replication : the replicated doc might be
>> different from the original one.
>>
>> Franck
Re: Proposal for new feature: Auto Update Functions
Posted by Alexander Shorin <kx...@gmail.com>.
The problem with update functions that they cannot be applied for bulk
updates. API reroute via reverse proxy cannot solve that too. I
believe, proposed feature assumed to handle this case.
--
,,,^..^,,,
On Sat, May 31, 2014 at 11:06 PM, Joan Touzet <jo...@lrtw.org> wrote:
> The typical solution to this is to use a reverse proxy or API layer to ensure that all client updates go through the required document update handler. It's unclear to me that this functionality native in CouchDB is necessary.
>
> If you decide to move ahead with implementation , keep in mind that the 1843-feature-bigcouch branch as that will be landing very soon. Any proposed patch should be compatible the fabric/chttpd-based approach and support single- and multi-node (BigCouch cluster) approaches.
>
> -Joan
>
> ----- Original Message -----
> From: "Franck Eyraud" <fr...@nospam.yrnm.net>
> To: dev@couchdb.apache.org
> Cc: user@couchdb.apache.org
> Sent: Saturday, May 31, 2014 8:34:57 AM
> Subject: Re: Proposal for new feature: Auto Update Functions
>
> Le 30/05/2014 13:04, Dirkjan Ochtman a écrit :
>> On Fri, May 30, 2014 at 12:57 PM, Suraj Kumar <su...@inmobi.com> wrote:
>>> What are your thoughts, both from the use-case as well as
>>> internals/performance of CouchDB about this?
>> How is this different from document update handlers?
> Update handlers must be called by the client to be used. Auto update
> functions would be called even if the client directly POST/PUT a doc to
> the DB (so they would be mandatory).
>
> At first sight a good idea, it seems to me that auto update functions
> would cause problem during replication : the replicated doc might be
> different from the original one.
>
> Franck
Re: Proposal for new feature: Auto Update Functions
Posted by Alexander Shorin <kx...@gmail.com>.
The problem with update functions that they cannot be applied for bulk
updates. API reroute via reverse proxy cannot solve that too. I
believe, proposed feature assumed to handle this case.
--
,,,^..^,,,
On Sat, May 31, 2014 at 11:06 PM, Joan Touzet <jo...@lrtw.org> wrote:
> The typical solution to this is to use a reverse proxy or API layer to ensure that all client updates go through the required document update handler. It's unclear to me that this functionality native in CouchDB is necessary.
>
> If you decide to move ahead with implementation , keep in mind that the 1843-feature-bigcouch branch as that will be landing very soon. Any proposed patch should be compatible the fabric/chttpd-based approach and support single- and multi-node (BigCouch cluster) approaches.
>
> -Joan
>
> ----- Original Message -----
> From: "Franck Eyraud" <fr...@nospam.yrnm.net>
> To: dev@couchdb.apache.org
> Cc: user@couchdb.apache.org
> Sent: Saturday, May 31, 2014 8:34:57 AM
> Subject: Re: Proposal for new feature: Auto Update Functions
>
> Le 30/05/2014 13:04, Dirkjan Ochtman a écrit :
>> On Fri, May 30, 2014 at 12:57 PM, Suraj Kumar <su...@inmobi.com> wrote:
>>> What are your thoughts, both from the use-case as well as
>>> internals/performance of CouchDB about this?
>> How is this different from document update handlers?
> Update handlers must be called by the client to be used. Auto update
> functions would be called even if the client directly POST/PUT a doc to
> the DB (so they would be mandatory).
>
> At first sight a good idea, it seems to me that auto update functions
> would cause problem during replication : the replicated doc might be
> different from the original one.
>
> Franck
Re: Proposal for new feature: Auto Update Functions
Posted by Joan Touzet <jo...@lrtw.org>.
The typical solution to this is to use a reverse proxy or API layer to ensure that all client updates go through the required document update handler. It's unclear to me that this functionality native in CouchDB is necessary.
If you decide to move ahead with implementation , keep in mind that the 1843-feature-bigcouch branch as that will be landing very soon. Any proposed patch should be compatible the fabric/chttpd-based approach and support single- and multi-node (BigCouch cluster) approaches.
-Joan
----- Original Message -----
From: "Franck Eyraud" <fr...@nospam.yrnm.net>
To: dev@couchdb.apache.org
Cc: user@couchdb.apache.org
Sent: Saturday, May 31, 2014 8:34:57 AM
Subject: Re: Proposal for new feature: Auto Update Functions
Le 30/05/2014 13:04, Dirkjan Ochtman a écrit :
> On Fri, May 30, 2014 at 12:57 PM, Suraj Kumar <su...@inmobi.com> wrote:
>> What are your thoughts, both from the use-case as well as
>> internals/performance of CouchDB about this?
> How is this different from document update handlers?
Update handlers must be called by the client to be used. Auto update
functions would be called even if the client directly POST/PUT a doc to
the DB (so they would be mandatory).
At first sight a good idea, it seems to me that auto update functions
would cause problem during replication : the replicated doc might be
different from the original one.
Franck
Re: Proposal for new feature: Auto Update Functions
Posted by Joan Touzet <jo...@lrtw.org>.
The typical solution to this is to use a reverse proxy or API layer to ensure that all client updates go through the required document update handler. It's unclear to me that this functionality native in CouchDB is necessary.
If you decide to move ahead with implementation , keep in mind that the 1843-feature-bigcouch branch as that will be landing very soon. Any proposed patch should be compatible the fabric/chttpd-based approach and support single- and multi-node (BigCouch cluster) approaches.
-Joan
----- Original Message -----
From: "Franck Eyraud" <fr...@nospam.yrnm.net>
To: dev@couchdb.apache.org
Cc: user@couchdb.apache.org
Sent: Saturday, May 31, 2014 8:34:57 AM
Subject: Re: Proposal for new feature: Auto Update Functions
Le 30/05/2014 13:04, Dirkjan Ochtman a écrit :
> On Fri, May 30, 2014 at 12:57 PM, Suraj Kumar <su...@inmobi.com> wrote:
>> What are your thoughts, both from the use-case as well as
>> internals/performance of CouchDB about this?
> How is this different from document update handlers?
Update handlers must be called by the client to be used. Auto update
functions would be called even if the client directly POST/PUT a doc to
the DB (so they would be mandatory).
At first sight a good idea, it seems to me that auto update functions
would cause problem during replication : the replicated doc might be
different from the original one.
Franck
Re: Proposal for new feature: Auto Update Functions
Posted by Franck Eyraud <fr...@nospam.yrnm.net>.
Le 30/05/2014 13:04, Dirkjan Ochtman a écrit :
> On Fri, May 30, 2014 at 12:57 PM, Suraj Kumar <su...@inmobi.com> wrote:
>> What are your thoughts, both from the use-case as well as
>> internals/performance of CouchDB about this?
> How is this different from document update handlers?
Update handlers must be called by the client to be used. Auto update
functions would be called even if the client directly POST/PUT a doc to
the DB (so they would be mandatory).
At first sight a good idea, it seems to me that auto update functions
would cause problem during replication : the replicated doc might be
different from the original one.
Franck
Re: Proposal for new feature: Auto Update Functions
Posted by Franck Eyraud <fr...@nospam.yrnm.net>.
Le 30/05/2014 13:04, Dirkjan Ochtman a écrit :
> On Fri, May 30, 2014 at 12:57 PM, Suraj Kumar <su...@inmobi.com> wrote:
>> What are your thoughts, both from the use-case as well as
>> internals/performance of CouchDB about this?
> How is this different from document update handlers?
Update handlers must be called by the client to be used. Auto update
functions would be called even if the client directly POST/PUT a doc to
the DB (so they would be mandatory).
At first sight a good idea, it seems to me that auto update functions
would cause problem during replication : the replicated doc might be
different from the original one.
Franck
Re: Proposal for new feature: Auto Update Functions
Posted by Dirkjan Ochtman <di...@ochtman.nl>.
On Fri, May 30, 2014 at 12:57 PM, Suraj Kumar <su...@inmobi.com> wrote:
> What are your thoughts, both from the use-case as well as
> internals/performance of CouchDB about this?
How is this different from document update handlers?
http://docs.couchdb.org/en/latest/couchapp/ddocs.html#update-functions
Cheers,
Dirkjan
Re: Proposal for new feature: Auto Update Functions
Posted by Dirkjan Ochtman <di...@ochtman.nl>.
On Fri, May 30, 2014 at 12:57 PM, Suraj Kumar <su...@inmobi.com> wrote:
> What are your thoughts, both from the use-case as well as
> internals/performance of CouchDB about this?
How is this different from document update handlers?
http://docs.couchdb.org/en/latest/couchapp/ddocs.html#update-functions
Cheers,
Dirkjan