Re: Replication Filters: When changing restrictions data becomes out of sync

[Randall Leeds <ra...@gmail.com>]

If you're looking for replication to delete documents that don't fit the
filter from the target, you'll have to do that manually. It is never the
place of replication to remove documents on the target.

What you can do is set up an update handler on second db that deletes
documents exclusivly meant for seconddb from firstdb so in this way you can
make it automatic. This doesn't cover the case where a document that used to
replicate should be deleted everywhere except maindb.

On Apr 4, 2010 3:58 PM, "Ben Hall" <be...@googlemail.com> wrote:

Hi,

I have the following setup:

MainDB
FirstDB
SecondDB

First and Second will contain a subset of the data in MainDB. I
planned to use Replication Filters to populate the DB.  This is
working great, until I change a document in MainDB from being
restricted to FirstDB to being restricted to SecondDB.

When this happens, replication correctly applies it to SecondDB -
however it still exists in FirstDB. As such, my data is now
inconsistent.

Is this correct? The only thing I can think is that I'm going to have
to manually delete the document from FirstDB - which is a little bit
annoying.

Is there a better way?

Thanks

Ben
http://twitter.com/Ben_Hall

Re: Replication Filters: When changing restrictions data becomes out of sync

[Randall Leeds <ra...@gmail.com>]

I apologize. I was not clear.

There is a feature of CouchDB called update_notifications that allows
an external process, supervised by Erlang, to be sent information
about document updates.

Unfortunately, this feature is not well documented, but you can see a
couple examples[1][2].

It's not elegant, but it might work for you depending on how complex
your use case is. You can use whatever language and HTTP client
library you like to issue the appropriate commands.

You could also write a daemon that consumes the ?changes feed.

You might load all the _design documents at the beginning. When the
design document is updated, compare the filter functions to see if
they've changed. You could fetch all the documents one by one or in
chunks and run them through the old filter and the new filter and see
which ones should no longer be replicated and issue a DELETE.

In short, there's no particularly efficient way to handle "correcting"
a changed replication filter. But you can work around it if you're
really determined.

[1] http://wiki.apache.org/couchdb/Regenerating_views_on_update?highlight=(update\_notification)
[2] http://github.com/tilgovi/couchdb-lounge/raw/master/replicator/replication_notifier.py

Re: Replication Filters: When changing restrictions data becomes out of sync

[Ben Hall <be...@googlemail.com>]

Hi Randall,

Thanks for the comment.  I looked at the _update test cases but
couldn't see anything which would fit.

http://svn.apache.org/viewvc/couchdb/trunk/share/www/script/test/update_documents.js?view=markup

How would I issue a DELETE command from javascript against the FirstDB?

Thanks

Ben

On Mon, Apr 5, 2010 at 1:27 AM, Randall Leeds <ra...@gmail.com> wrote:
> If you're looking for replication to delete documents that don't fit the
> filter from the target, you'll have to do that manually. It is never the
> place of replication to remove documents on the target.
>
> What you can do is set up an update handler on second db that deletes
> documents exclusivly meant for seconddb from firstdb so in this way you can
> make it automatic. This doesn't cover the case where a document that used to
> replicate should be deleted everywhere except maindb.
>
> On Apr 4, 2010 3:58 PM, "Ben Hall" <be...@googlemail.com> wrote:
>
> Hi,
>
> I have the following setup:
>
> MainDB
> FirstDB
> SecondDB
>
> First and Second will contain a subset of the data in MainDB. I
> planned to use Replication Filters to populate the DB.  This is
> working great, until I change a document in MainDB from being
> restricted to FirstDB to being restricted to SecondDB.
>
> When this happens, replication correctly applies it to SecondDB -
> however it still exists in FirstDB. As such, my data is now
> inconsistent.
>
> Is this correct? The only thing I can think is that I'm going to have
> to manually delete the document from FirstDB - which is a little bit
> annoying.
>
> Is there a better way?
>
> Thanks
>
> Ben
> http://twitter.com/Ben_Hall
>