You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Tim Parkin <in...@timparkin.co.uk> on 2009/03/13 10:54:10 UTC

Re: Bulk Docs - 'remove changes so far' Functionality?

Ignoring the atomicity of bulk docs. Would it be possible to retain some
sort of ability to simply remove a set of changes.

At the moment I have a 'changeset' as a result of some user or system
operation and using bulk_docs I can apply it and conditionally remove it
if unsuccessful.

The old bulk docs was one way of being able to cleanly apply and
conditionally remove a set of changes. With this feature removed, the
process of recovering from a changeset that breaks at the end is 'non
trivial' to say the least.

The current arguments against bulk docs are against it's atomicity. I
would be reasonably happy to have a bulk docs that wasn't atomic but
allowed some from of 'remove the changes applied so far' functionality.

I don't like to predict what problems other people might get but I think
that without something like this, people will have problems removing
changes. From the (very) little I understand, we can't rely on older revs.

Any ideas for a generic solution to this problem would be greatly
appreciated..

Tim

Re: Bulk Docs - 'remove changes so far' Functionality?

Posted by Chris Anderson <jc...@apache.org>.
On Fri, Mar 13, 2009 at 10:42 AM, Wout Mertens <wo...@gmail.com> wrote:
> On Mar 13, 2009, at 5:47 PM, Chris Anderson wrote:
>
>> On Fri, Mar 13, 2009 at 9:41 AM, Wout Mertens <wo...@gmail.com>
>> wrote:
>>>
>>> On Mar 13, 2009, at 4:12 PM, Tim Parkin wrote:
>>>
>>>> I think all bulk doc changes were written somewhere and then only
>>>> switched in once the last was successful.. so no locking.
>>>
>>> But then if a normal write comes in while the bulk doc is being
>>> processed,
>>> how did it know to fail that write/fail the bulk?
>>>
>>
>> CouchDB updates are serialized, so if a normal write comes in during a
>> bulk update, the normal write waits. Of course, all this is different
>> if the bulk update is spread across multiple nodes. Crossing that
>> bridge is part of the motivation for limiting the feature to skip
>> conflict checking.
>
> Again out of curiosity, how would the spreading work? The bulk write would
> go to ibrowse, which would then contact multiple couches and give each of
> them some work?
>

Something like that. Maybe leave out ibrowse / http all together and
communicate using erlang messages.

Alternatively, save the update to a local temporary database, ack the
client, and then replicate the updates to the proper home.

Chris


-- 
Chris Anderson
http://jchris.mfdz.com

Re: Bulk Docs - 'remove changes so far' Functionality?

Posted by Wout Mertens <wo...@gmail.com>.
On Mar 13, 2009, at 5:47 PM, Chris Anderson wrote:

> On Fri, Mar 13, 2009 at 9:41 AM, Wout Mertens  
> <wo...@gmail.com> wrote:
>> On Mar 13, 2009, at 4:12 PM, Tim Parkin wrote:
>>
>>> I think all bulk doc changes were written somewhere and then only
>>> switched in once the last was successful.. so no locking.
>>
>> But then if a normal write comes in while the bulk doc is being  
>> processed,
>> how did it know to fail that write/fail the bulk?
>>
>
> CouchDB updates are serialized, so if a normal write comes in during a
> bulk update, the normal write waits. Of course, all this is different
> if the bulk update is spread across multiple nodes. Crossing that
> bridge is part of the motivation for limiting the feature to skip
> conflict checking.

Again out of curiosity, how would the spreading work? The bulk write  
would go to ibrowse, which would then contact multiple couches and  
give each of them some work?

Wout.

Re: Bulk Docs - 'remove changes so far' Functionality?

Posted by Chris Anderson <jc...@apache.org>.
On Fri, Mar 13, 2009 at 9:41 AM, Wout Mertens <wo...@gmail.com> wrote:
> On Mar 13, 2009, at 4:12 PM, Tim Parkin wrote:
>
>> I think all bulk doc changes were written somewhere and then only
>> switched in once the last was successful.. so no locking.
>
> But then if a normal write comes in while the bulk doc is being processed,
> how did it know to fail that write/fail the bulk?
>

CouchDB updates are serialized, so if a normal write comes in during a
bulk update, the normal write waits. Of course, all this is different
if the bulk update is spread across multiple nodes. Crossing that
bridge is part of the motivation for limiting the feature to skip
conflict checking.

Chris

-- 
Chris Anderson
http://jchris.mfdz.com

Re: Bulk Docs - 'remove changes so far' Functionality?

Posted by Wout Mertens <wo...@gmail.com>.
On Mar 13, 2009, at 4:12 PM, Tim Parkin wrote:

> I think all bulk doc changes were written somewhere and then only
> switched in once the last was successful.. so no locking.

But then if a normal write comes in while the bulk doc is being  
processed, how did it know to fail that write/fail the bulk?

Wout.

Re: Bulk Docs - 'remove changes so far' Functionality?

Posted by Tim Parkin <in...@timparkin.co.uk>.
Damien Katz wrote:
> There is an all_or_nothing option for bulk updates, it does
> validation/security checks on the documents, and if all pass then it
> saves all the documents. If there is a failure or db crash during the
> commit, none of the documents are saved. However, it does not check for
> conflicts, documents are saved regardless if the conflict.
> 
> You can do all or nothing bulk updates with conflict checking by
> serializing updates at the app level (or use a programmable reverse
> proxy) and check bulk updates before committing to see if they'll
> conflict. With the all_or_nothing flag, you'll get exactly the behavior
> you desire.
> 

Hi Damian,

A couple of questions..

1) For the average developer who is making changes across a bunch of
documents and wants all of the changes to work or none of them, can you
recommend a workflow to help deal with conflicts?

With bulk_docs this is a trivial operation on a single node (which is
where most sharded changes will be applied) but with partial
successes/partial conflicts, I'm not sure how to make sure things get
cleaned up/reverted correctly?

2) If we wanted to include bulk_docs in a fork, do the new changes mean
the old bulk_docs is impossible or has it just been switched off and not
tested?

Tim



Re: Bulk Docs - 'remove changes so far' Functionality?

Posted by Damien Katz <da...@apache.org>.
On Mar 13, 2009, at 11:12 AM, Tim Parkin wrote:

> Wout Mertens wrote:
>> On Mar 13, 2009, at 10:54 AM, Tim Parkin wrote:
>>
>>> The current arguments against bulk docs are against it's  
>>> atomicity. I
>>> would be reasonably happy to have a bulk docs that wasn't atomic but
>>> allowed some from of 'remove the changes applied so far'  
>>> functionality.
>>
>> So out of interest, to get into the nitty gritty details, is this the
>> problem? The fact that CouchDB handles one document at a time and  
>> can't
>> remove a change?
> <snip>
>>
>> Wout.
>
> I think all bulk doc changes were written somewhere and then only
> switched in once the last was successful.. so no locking.
>
> With the new bulk docs, you could apply bulk doc change, find out the
> last one was wrong and then discover that all but the last one had
> already been replicated..

There is an all_or_nothing option for bulk updates, it does validation/ 
security checks on the documents, and if all pass then it saves all  
the documents. If there is a failure or db crash during the commit,  
none of the documents are saved. However, it does not check for  
conflicts, documents are saved regardless if the conflict.

You can do all or nothing bulk updates with conflict checking by  
serializing updates at the app level (or use a programmable reverse  
proxy) and check bulk updates before committing to see if they'll  
conflict. With the all_or_nothing flag, you'll get exactly the  
behavior you desire.

-Damien

Re: Bulk Docs - 'remove changes so far' Functionality?

Posted by Tim Parkin <in...@timparkin.co.uk>.
Wout Mertens wrote:
> On Mar 13, 2009, at 10:54 AM, Tim Parkin wrote:
> 
>> The current arguments against bulk docs are against it's atomicity. I
>> would be reasonably happy to have a bulk docs that wasn't atomic but
>> allowed some from of 'remove the changes applied so far' functionality.
> 
> So out of interest, to get into the nitty gritty details, is this the
> problem? The fact that CouchDB handles one document at a time and can't
> remove a change?
<snip>
> 
> Wout.

I think all bulk doc changes were written somewhere and then only
switched in once the last was successful.. so no locking.

With the new bulk docs, you could apply bulk doc change, find out the
last one was wrong and then discover that all but the last one had
already been replicated..

Tim

Re: Bulk Docs - 'remove changes so far' Functionality?

Posted by Wout Mertens <wo...@gmail.com>.
On Mar 13, 2009, at 10:54 AM, Tim Parkin wrote:

> The current arguments against bulk docs are against it's atomicity. I
> would be reasonably happy to have a bulk docs that wasn't atomic but
> allowed some from of 'remove the changes applied so far'  
> functionality.

So out of interest, to get into the nitty gritty details, is this the  
problem? The fact that CouchDB handles one document at a time and  
can't remove a change?

I think that I see why this is such a problem. If you provide rollback  
(or simple rewrite-prev-version) functionality, it impacts  
replication. You can only replicate changes up to the last seq before  
an unfinished bulk update started, otherwise a non-confirmed document  
might slip through. So if you have back-to-back bulk updates, there's  
no window for replicating.

Is this correct? If so I don't see how bulk_docs can be atomic within  
the spirit of the charter of CouchDB.

In fact, I wonder how it worked up to now. Suppose there is this  
sequence of updates: (Docs are ABC, revs 123)

1. DB has A1 B1 C1
2a. Update: B2
2b. Bulk update: A2 B2
3. GET A

If 2a is handled first, the bulk update 2b should fail. How can you be  
sure that in 3, A returned is then A1? I suppose you need to use a  
lock, which makes performance unpredictable.

Wout.