You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Mitja Kleider <mi...@kleider.name> on 2011/07/10 00:08:16 UTC
Replication filter ignoring deleted docs or changed values
Hi,
as an example I chose the replication filter function from
http://wiki.apache.org/couchdb/Replication#Filtered_Replication
> function(doc, req) {
> if (doc.type && doc.type == "foo") {
> return true;
> } else {
> return false;
> }
> }
After replicating to another db, change a document containing type="foo" to type="bar" and replicate again.
Because the filter function returns false, the document remains unchanged in the replicated db.
The same happens when deleting the document containing type="foo". The filter function returns false on the deleted document, the deletion is not replicated.
I would like to achieve a different behavior:
If the filter function returns false, but the document was replicated previously, it should be deleted.
Any ideas how to solve this?
Thanks,
Mitja
Re: Replication filter ignoring deleted docs or changed values
Posted by Zachary Zolton <za...@gmail.com>.
Mitja,
Perhaps you could record in some other field that the doc with type
"bar" used to be type "foo".
Then way you can filter the replication by docs where:
type == "foo" || type == "bar" && old_type == "foo"
This should then replicate docs that change their type from "foo" to "bar".
To give any better advice, I'd need to hear more concretely what
you're trying to do here.
-Zach
On Sunday, July 10, 2011, Mitja Kleider <mi...@kleider.name> wrote:
> On Sat, 2011-07-09 at 17:26 -0500, Zachary Zolton wrote:
>> Could you change your filter to something like this?
>>
>> function(doc) {
>> return doc._deleted || doc.type == 'foo';
>> }
>>
>> That way you replicate all deleted docs.
>
> That would work, thanks. I tried to avoid replicating millions of
> deleted documents together with a few hundred documents passing the
> filter.
>
> A solution might be this validate_doc_update function on the target
> database:
>
> function(newDoc, oldDoc, userCtx) {
> if (newDoc._deleted === true && !oldDoc) {
> throw({forbidden: 'do not create deleted docs'});
> }
> }
>
> This way all deleted documents will be transmitted, but at least not
> written if there is no previous revision.
>
>
> The problem for changed values remains: when the source database changes
> type='foo' to type='bar', the target database keeps the old revision
> with type='foo'.
>
>
> Mitja
>
>
Re: Replication filter ignoring deleted docs or changed values
Posted by Mitja Kleider <mi...@kleider.name>.
Zachary,
On Sun, 2011-07-10 at 13:35 -0500, Zachary Zolton wrote:
> Mitja,
>
> Perhaps you could record in some other field that the doc with type
> "bar" used to be type "foo".
>
> Then way you can filter the replication by docs where:
> type == "foo" || type == "bar" && old_type == "foo"
>
> This should then replicate docs that change their type from "foo" to "bar".
>
> To give any better advice, I'd need to hear more concretely what
> you're trying to do here.
I am working with geospatial data from the OpenStreetMap project.
Each document has infinitely many tags, similar to type="restaurant"
which might change in each revision.
I would also like to filter the geometry, to keep it simple something
like [x,y] and return (left <= x) && (x <= right) && (bottom <= y) &&
(y <= top).
The position might also change. I am afraid that I can not keep the
entire history because of the high amount of data.
The only possible other solution I have come up with is not using
filtered replication for updating the replicated target database, but
instead do full replication and filter in validate_doc_update(). Does
that make sense?
Mitja
> On Sunday, July 10, 2011, Mitja Kleider <mi...@kleider.name> wrote:
> > On Sat, 2011-07-09 at 17:26 -0500, Zachary Zolton wrote:
> >> Could you change your filter to something like this?
> >>
> >> function(doc) {
> >> return doc._deleted || doc.type == 'foo';
> >> }
> >>
> >> That way you replicate all deleted docs.
> >
> > That would work, thanks. I tried to avoid replicating millions of
> > deleted documents together with a few hundred documents passing the
> > filter.
> >
> > A solution might be this validate_doc_update function on the target
> > database:
> >
> > function(newDoc, oldDoc, userCtx) {
> > if (newDoc._deleted === true && !oldDoc) {
> > throw({forbidden: 'do not create deleted docs'});
> > }
> > }
> >
> > This way all deleted documents will be transmitted, but at least not
> > written if there is no previous revision.
> >
> >
> > The problem for changed values remains: when the source database changes
> > type='foo' to type='bar', the target database keeps the old revision
> > with type='foo'.
> >
> >
> > Mitja
> >
> >
Re: Replication filter ignoring deleted docs or changed values
Posted by Mark Hahn <ma...@boutiquing.com>.
This is similar to the problem that replication conflicts don't make
the old versions available so it is impossible to know what changed,
which makes it near impossible to resolve a conflict.
On Sun, Jul 10, 2011 at 11:35 AM, Zachary Zolton
<za...@gmail.com> wrote:
> Mitja,
>
> Perhaps you could record in some other field that the doc with type
> "bar" used to be type "foo".
>
> Then way you can filter the replication by docs where:
> type == "foo" || type == "bar" && old_type == "foo"
>
> This should then replicate docs that change their type from "foo" to "bar".
>
> To give any better advice, I'd need to hear more concretely what
> you're trying to do here.
>
> -Zach
>
> On Sunday, July 10, 2011, Mitja Kleider <mi...@kleider.name> wrote:
>> On Sat, 2011-07-09 at 17:26 -0500, Zachary Zolton wrote:
>>> Could you change your filter to something like this?
>>>
>>> function(doc) {
>>> return doc._deleted || doc.type == 'foo';
>>> }
>>>
>>> That way you replicate all deleted docs.
>>
>> That would work, thanks. I tried to avoid replicating millions of
>> deleted documents together with a few hundred documents passing the
>> filter.
>>
>> A solution might be this validate_doc_update function on the target
>> database:
>>
>> function(newDoc, oldDoc, userCtx) {
>> if (newDoc._deleted === true && !oldDoc) {
>> throw({forbidden: 'do not create deleted docs'});
>> }
>> }
>>
>> This way all deleted documents will be transmitted, but at least not
>> written if there is no previous revision.
>>
>>
>> The problem for changed values remains: when the source database changes
>> type='foo' to type='bar', the target database keeps the old revision
>> with type='foo'.
>>
>>
>> Mitja
>>
>>
>
Re: Replication filter ignoring deleted docs or changed values
Posted by Zachary Zolton <za...@gmail.com>.
Mitja,
Perhaps you could record in some other field that the doc with type
"bar" used to be type "foo".
Then way you can filter the replication by docs where:
type == "foo" || type == "bar" && old_type == "foo"
This should then replicate docs that change their type from "foo" to "bar".
To give any better advice, I'd need to hear more concretely what
you're trying to do here.
-Zach
On Sunday, July 10, 2011, Mitja Kleider <mi...@kleider.name> wrote:
> On Sat, 2011-07-09 at 17:26 -0500, Zachary Zolton wrote:
>> Could you change your filter to something like this?
>>
>> function(doc) {
>> return doc._deleted || doc.type == 'foo';
>> }
>>
>> That way you replicate all deleted docs.
>
> That would work, thanks. I tried to avoid replicating millions of
> deleted documents together with a few hundred documents passing the
> filter.
>
> A solution might be this validate_doc_update function on the target
> database:
>
> function(newDoc, oldDoc, userCtx) {
> if (newDoc._deleted === true && !oldDoc) {
> throw({forbidden: 'do not create deleted docs'});
> }
> }
>
> This way all deleted documents will be transmitted, but at least not
> written if there is no previous revision.
>
>
> The problem for changed values remains: when the source database changes
> type='foo' to type='bar', the target database keeps the old revision
> with type='foo'.
>
>
> Mitja
>
>
Re: Replication filter ignoring deleted docs or changed values
Posted by Mitja Kleider <mi...@kleider.name>.
On Sat, 2011-07-09 at 17:26 -0500, Zachary Zolton wrote:
> Could you change your filter to something like this?
>
> function(doc) {
> return doc._deleted || doc.type == 'foo';
> }
>
> That way you replicate all deleted docs.
That would work, thanks. I tried to avoid replicating millions of
deleted documents together with a few hundred documents passing the
filter.
A solution might be this validate_doc_update function on the target
database:
function(newDoc, oldDoc, userCtx) {
if (newDoc._deleted === true && !oldDoc) {
throw({forbidden: 'do not create deleted docs'});
}
}
This way all deleted documents will be transmitted, but at least not
written if there is no previous revision.
The problem for changed values remains: when the source database changes
type='foo' to type='bar', the target database keeps the old revision
with type='foo'.
Mitja
Re: Replication filter ignoring deleted docs or changed values
Posted by Zachary Zolton <za...@gmail.com>.
Could you change your filter to something like this?
function(doc) {
return doc._deleted || doc.type == 'foo';
}
That way you replicate all deleted docs.
-Zach
On Saturday, July 9, 2011, Mitja Kleider <mi...@kleider.name> wrote:
> Hi,
>
> as an example I chose the replication filter function from
> http://wiki.apache.org/couchdb/Replication#Filtered_Replication
>> function(doc, req) {
>> if (doc.type && doc.type == "foo") {
>> return true;
>> } else {
>> return false;
>> }
>> }
>
> After replicating to another db, change a document containing type="foo" to type="bar" and replicate again.
> Because the filter function returns false, the document remains unchanged in the replicated db.
> The same happens when deleting the document containing type="foo". The filter function returns false on the deleted document, the deletion is not replicated.
>
> I would like to achieve a different behavior:
> If the filter function returns false, but the document was replicated previously, it should be deleted.
>
> Any ideas how to solve this?
>
>
> Thanks,
> Mitja
>
>