Passing seq_num to filter replication?

[Mike Marino <mm...@gmail.com>]

Hi all,

We have databases that are full of time-based information (e.g.
measurements being saved) and would like to be able to partially replicate
the databases.  I know this can be done with normal filter functions (we do
this already), but this requires running over all documents which can be
very inefficient and not particularly scalable.

I was wondering if there's some (perhaps not well documented) feature
allowing something like since=update_seq or since="now" as it is in
continuous changes feeds?  Very often, we're simply concerned about only
pulling in the newest documents and doing it continuously.

If this doesn't exist, I guess we could consider rolling our own
"replication" based upon a changes feed.

Thanks,
Mike

Re: Passing seq_num to filter replication?

[Alexander Shorin <kx...@gmail.com>]

Afaik, "now" wouldn't work since that sugar is defined on httpd side,
not replicator one.  And such value will be a bit confusing,
especially when replication will have to be restarted.
But any help with docs clarification is welcome! (:
--
,,,^..^,,,


On Mon, May 4, 2015 at 8:43 PM, Mike Marino <mm...@gmail.com> wrote:
> Hi Alexander,
>
> yeah, wow, that's what I'm looking for... not sure how I missed that.
>
> To make this even more obvious, I think it'd be great to add a link from
> the replicator database page (
> https://couchdb.readthedocs.org/en/1.6.1/replication/replicator.html#replicator)
> to the JSON object page, (
> https://couchdb.readthedocs.org/en/1.6.1/json-structure.html#replication-settings
> ).  Some of the info on the former is not up to date.  Also, it's not clear
> if "now" works? (I'm about to try it, but just for future reference needs.)
>
> I'd be happy to make these updates.  :-)
>
> Cheers,
> Mike
>
> On Mon, May 4, 2015 at 7:27 PM, Alexander Shorin <kx...@gmail.com> wrote:
>
>> You can specify replication start sequence with "since_seq" field with
>> the request to /_replicate or in _replicator doc. Is this what you're
>> looking for?
>> --
>> ,,,^..^,,,
>>
>>
>> On Mon, May 4, 2015 at 8:12 PM, Mike Marino <mm...@gmail.com> wrote:
>> > Hi all,
>> >
>> > We have databases that are full of time-based information (e.g.
>> > measurements being saved) and would like to be able to partially
>> replicate
>> > the databases.  I know this can be done with normal filter functions (we
>> do
>> > this already), but this requires running over all documents which can be
>> > very inefficient and not particularly scalable.
>> >
>> > I was wondering if there's some (perhaps not well documented) feature
>> > allowing something like since=update_seq or since="now" as it is in
>> > continuous changes feeds?  Very often, we're simply concerned about only
>> > pulling in the newest documents and doing it continuously.
>> >
>> > If this doesn't exist, I guess we could consider rolling our own
>> > "replication" based upon a changes feed.
>> >
>> > Thanks,
>> > Mike
>>

Re: Passing seq_num to filter replication?

[Mike Marino <mm...@gmail.com>]

Hi Alexander,

yeah, wow, that's what I'm looking for... not sure how I missed that.

To make this even more obvious, I think it'd be great to add a link from
the replicator database page (
https://couchdb.readthedocs.org/en/1.6.1/replication/replicator.html#replicator)
to the JSON object page, (
https://couchdb.readthedocs.org/en/1.6.1/json-structure.html#replication-settings
).  Some of the info on the former is not up to date.  Also, it's not clear
if "now" works? (I'm about to try it, but just for future reference needs.)

I'd be happy to make these updates.  :-)

Cheers,
Mike

On Mon, May 4, 2015 at 7:27 PM, Alexander Shorin <kx...@gmail.com> wrote:

> You can specify replication start sequence with "since_seq" field with
> the request to /_replicate or in _replicator doc. Is this what you're
> looking for?
> --
> ,,,^..^,,,
>
>
> On Mon, May 4, 2015 at 8:12 PM, Mike Marino <mm...@gmail.com> wrote:
> > Hi all,
> >
> > We have databases that are full of time-based information (e.g.
> > measurements being saved) and would like to be able to partially
> replicate
> > the databases.  I know this can be done with normal filter functions (we
> do
> > this already), but this requires running over all documents which can be
> > very inefficient and not particularly scalable.
> >
> > I was wondering if there's some (perhaps not well documented) feature
> > allowing something like since=update_seq or since="now" as it is in
> > continuous changes feeds?  Very often, we're simply concerned about only
> > pulling in the newest documents and doing it continuously.
> >
> > If this doesn't exist, I guess we could consider rolling our own
> > "replication" based upon a changes feed.
> >
> > Thanks,
> > Mike
>

Re: Passing seq_num to filter replication?

[Alexander Shorin <kx...@gmail.com>]

You can specify replication start sequence with "since_seq" field with
the request to /_replicate or in _replicator doc. Is this what you're
looking for?
--
,,,^..^,,,


On Mon, May 4, 2015 at 8:12 PM, Mike Marino <mm...@gmail.com> wrote:
> Hi all,
>
> We have databases that are full of time-based information (e.g.
> measurements being saved) and would like to be able to partially replicate
> the databases.  I know this can be done with normal filter functions (we do
> this already), but this requires running over all documents which can be
> very inefficient and not particularly scalable.
>
> I was wondering if there's some (perhaps not well documented) feature
> allowing something like since=update_seq or since="now" as it is in
> continuous changes feeds?  Very often, we're simply concerned about only
> pulling in the newest documents and doing it continuously.
>
> If this doesn't exist, I guess we could consider rolling our own
> "replication" based upon a changes feed.
>
> Thanks,
> Mike