More View Changes Input

["benjamin.bastian@gmail.com" <be...@gmail.com>]

Hey all,

So, the view changes PRs have been hanging out for a bit (review welcome!)
but things have changed a fair bit since I last emailed wider dev
community. Bob's had a chance to do a bit of review (thanks Bob!) and
voiced his preference for there to be a single view changes row returned in
the response body for each sequence changed (rather than for each seq-key
pair changed). So, that would mean returning the keys added and removed
from the index on each row rather than one key per row. Initially, I
thought a format like:

{"id": "docid", "add": [{"key": "value"},{"key":"value2"}], "remove":
["key_a", "key_b"], "seq": 5, "changes": [{"_rev": "1-abcdef"}]}

(Note: "add" is a list of {"key": "value"} objects rather than a single
multi-KV object because duplicate keys can be emitted in a single change)

Though I realized that keys can be non-strings, which would make that
return format invalid JSON. So, what I'm currently going with is:

{"id": "docid", "add": [["key", "value"],["key", "value2"]], "remove":
["key_a", "key_b"], "seq": 5, "changes": [{"_rev": "1-abcdef"}]}

I can see why one might find this esoteric/yucky, so I thought I'd express
the reasoning and ask for some input/alternatives. So, does anyone have
thoughts/preferences?

In other news, I'll be rebasing/squashing soon to remove some less-useful
commits, but after that I think it'll be mergeable. If anyone feels
comfortable to delve into the changes, I'd appreciate some feedback (and
hopefully some +1s).

Thanks!
Ben

Re: More View Changes Input

[Benoit Chesneau <bc...@gmail.com>]

On Mon, Oct 6, 2014 at 9:47 PM, benjamin.bastian@gmail.com <
benjamin.bastian@gmail.com> wrote:

> Are you talking about this?
> https://github.com/rcouch/rcouch/wiki/Replication-with-view-changes
>
> That's in there.
>
> There are two "view changes" things. There's the optimization to
> db/_changes?filter=_view&view=d/v (for getting a _changes feed filtered by
> a view in an efficient way) and there's db/_design/d/_view_changes/v (for
> getting a feed of the changes to key/value pairs in a view) and I'm talking
> about the latter here. For filtering on the _view_changes feed, there's
> nothing preventing that from happening in the future, but that's not
> currently in the PR (as it has slightly different meanings if you're
> filtering on KVs rather than a doc). For filtering on the seq-indexed
> db/_changes?filter=_view&view=d/v feed, that has been left relatively
> unchanged from what you wrote.
>
> The question I'm asking in this email is with regard to the second endpoint
> for the feed of updated key/value pairs and what that API looks like.
>
> Does that make sense?
>


What i could say. Getting changes for a group / doc makes sense I guess
though it is not what the rcouch merge branch was about and not the reason
i given the code for.

- benoit




>
> Ben
>
> On Fri, Oct 3, 2014 at 2:11 PM, Benoit Chesneau <bc...@gmail.com>
> wrote:
>
> > On Fri, Oct 3, 2014 at 10:57 PM, benjamin.bastian@gmail.com <
> > benjamin.bastian@gmail.com> wrote:
> >
> > > Yeah, what Paul said is correct.
> > >
> > > As far as replication, you would be able to manually mediate a view
> > > replication with the view changes feed, but there's no built-in
> > > functionality to do it for you. Unless you're talking about the
> > > view-filtered replication, which will be optimized if the view is
> > sequence
> > > indexed.
> > >
> > >
> > It is in rcouch. So it was about to reuse the code ...
> >
> > I do think that such change requires a ticket for those who want to track
> > at least, but this quite out of topic so just let's talk about this
> > specific change.
> >
> > How does key filtering work with such things? Or isn't it part of the
> view
> > change any more? If it is what would be the result?
> >
> > On an operational part such changes imply that views changes are about
> > getting index operations on a doc. Which is conceptually different that
> > than getting changes on an index. Maybe we could have both, but I think
> > having index changes is also interesting independently of having index
> > operations on a doc. It is also covering 2 different usages. Imo having
> doc
> > index operations could be done on another resource point. Thoughts?
> >
> > - benoit.
> >
>

Re: More View Changes Input

["benjamin.bastian@gmail.com" <be...@gmail.com>]

Are you talking about this?
https://github.com/rcouch/rcouch/wiki/Replication-with-view-changes

That's in there.

There are two "view changes" things. There's the optimization to
db/_changes?filter=_view&view=d/v (for getting a _changes feed filtered by
a view in an efficient way) and there's db/_design/d/_view_changes/v (for
getting a feed of the changes to key/value pairs in a view) and I'm talking
about the latter here. For filtering on the _view_changes feed, there's
nothing preventing that from happening in the future, but that's not
currently in the PR (as it has slightly different meanings if you're
filtering on KVs rather than a doc). For filtering on the seq-indexed
db/_changes?filter=_view&view=d/v feed, that has been left relatively
unchanged from what you wrote.

The question I'm asking in this email is with regard to the second endpoint
for the feed of updated key/value pairs and what that API looks like.

Does that make sense?

Ben

On Fri, Oct 3, 2014 at 2:11 PM, Benoit Chesneau <bc...@gmail.com> wrote:

> On Fri, Oct 3, 2014 at 10:57 PM, benjamin.bastian@gmail.com <
> benjamin.bastian@gmail.com> wrote:
>
> > Yeah, what Paul said is correct.
> >
> > As far as replication, you would be able to manually mediate a view
> > replication with the view changes feed, but there's no built-in
> > functionality to do it for you. Unless you're talking about the
> > view-filtered replication, which will be optimized if the view is
> sequence
> > indexed.
> >
> >
> It is in rcouch. So it was about to reuse the code ...
>
> I do think that such change requires a ticket for those who want to track
> at least, but this quite out of topic so just let's talk about this
> specific change.
>
> How does key filtering work with such things? Or isn't it part of the view
> change any more? If it is what would be the result?
>
> On an operational part such changes imply that views changes are about
> getting index operations on a doc. Which is conceptually different that
> than getting changes on an index. Maybe we could have both, but I think
> having index changes is also interesting independently of having index
> operations on a doc. It is also covering 2 different usages. Imo having doc
> index operations could be done on another resource point. Thoughts?
>
> - benoit.
>

Re: More View Changes Input

[Benoit Chesneau <bc...@gmail.com>]

On Fri, Oct 3, 2014 at 10:57 PM, benjamin.bastian@gmail.com <
benjamin.bastian@gmail.com> wrote:

> Yeah, what Paul said is correct.
>
> As far as replication, you would be able to manually mediate a view
> replication with the view changes feed, but there's no built-in
> functionality to do it for you. Unless you're talking about the
> view-filtered replication, which will be optimized if the view is sequence
> indexed.
>
>
It is in rcouch. So it was about to reuse the code ...

I do think that such change requires a ticket for those who want to track
at least, but this quite out of topic so just let's talk about this
specific change.

How does key filtering work with such things? Or isn't it part of the view
change any more? If it is what would be the result?

On an operational part such changes imply that views changes are about
getting index operations on a doc. Which is conceptually different that
than getting changes on an index. Maybe we could have both, but I think
having index changes is also interesting independently of having index
operations on a doc. It is also covering 2 different usages. Imo having doc
index operations could be done on another resource point. Thoughts?

- benoit.

Re: More View Changes Input

["benjamin.bastian@gmail.com" <be...@gmail.com>]

Yeah, what Paul said is correct.

As far as replication, you would be able to manually mediate a view
replication with the view changes feed, but there's no built-in
functionality to do it for you. Unless you're talking about the
view-filtered replication, which will be optimized if the view is sequence
indexed.

On Fri, Oct 3, 2014 at 1:43 PM, Paul Davis <pa...@gmail.com>
wrote:

> On Fri, Oct 3, 2014 at 4:17 AM, Benoit Chesneau <bc...@gmail.com>
> wrote:
> > On Fri, Oct 3, 2014 at 11:12 AM, benjamin.bastian@gmail.com <
> > benjamin.bastian@gmail.com> wrote:
> >
> >> Hey all,
> >>
> >> So, the view changes PRs have been hanging out for a bit (review
> welcome!)
> >> but things have changed a fair bit since I last emailed wider dev
> >> community. Bob's had a chance to do a bit of review (thanks Bob!) and
> >> voiced his preference for there to be a single view changes row
> returned in
> >> the response body for each sequence changed (rather than for each
> seq-key
> >> pair changed). So, that would mean returning the keys added and removed
> >> from the index on each row rather than one key per row. Initially, I
> >> thought a format like:
> >>
> >> {"id": "docid", "add": [{"key": "value"},{"key":"value2"}], "remove":
> >> ["key_a", "key_b"], "seq": 5, "changes": [{"_rev": "1-abcdef"}]}
> >>
> >> (Note: "add" is a list of {"key": "value"} objects rather than a single
> >> multi-KV object because duplicate keys can be emitted in a single
> change)
> >>
> >> Though I realized that keys can be non-strings, which would make that
> >> return format invalid JSON. So, what I'm currently going with is:
> >>
> >> {"id": "docid", "add": [["key", "value"],["key", "value2"]], "remove":
> >> ["key_a", "key_b"], "seq": 5, "changes": [{"_rev": "1-abcdef"}]}
> >>
> >> I can see why one might find this esoteric/yucky, so I thought I'd
> express
> >> the reasoning and ask for some input/alternatives. So, does anyone have
> >> thoughts/preferences?
> >>
> >> In other news, I'll be rebasing/squashing soon to remove some
> less-useful
> >> commits, but after that I think it'll be mergeable. If anyone feels
> >> comfortable to delve into the changes, I'd appreciate some feedback (and
> >> hopefully some +1s).
> >>
> >> Thanks!
> >> Ben
> >>
> >
> > Can we have a feature request about that proposal?
> >
>
> I don't think this is a feature so much as asking for input on the
> format to be used for view changes. For instance, if we have a
> document that emit's three keys, do we have there rows in the output,
> or a single row that has a list of the keys that document emitted.
> Given that the database update sequence is per document it seems to me
> that the view sequence should probably also follow that model or else
> we have to decide if we want to create a new update sequence or have
> rows with the same sequence number.
>
> > It is also not clear what it imply, what is the impact on the memory,
> > bandwidth? Why this change is needed?
> >
>
> I'd imagine it'd save a bit on bandwidth just by the fact that its not
> sending the document id multiple times. I wouldn't expect there to be
> a memory difference either way as I'm pretty sure its just formatting
> as the bytes go onto the socket more than anything. Granted I haven't
> gone back and reviewed the structures so I could be missing details
> there as well.
>
> > Also I am lost in the flux of UI commits and i am not sure now, but is
> the
> > replication over view changes is enabled?
> >
> > - benoit
>

Re: More View Changes Input

[Paul Davis <pa...@gmail.com>]

On Fri, Oct 3, 2014 at 4:17 AM, Benoit Chesneau <bc...@gmail.com> wrote:
> On Fri, Oct 3, 2014 at 11:12 AM, benjamin.bastian@gmail.com <
> benjamin.bastian@gmail.com> wrote:
>
>> Hey all,
>>
>> So, the view changes PRs have been hanging out for a bit (review welcome!)
>> but things have changed a fair bit since I last emailed wider dev
>> community. Bob's had a chance to do a bit of review (thanks Bob!) and
>> voiced his preference for there to be a single view changes row returned in
>> the response body for each sequence changed (rather than for each seq-key
>> pair changed). So, that would mean returning the keys added and removed
>> from the index on each row rather than one key per row. Initially, I
>> thought a format like:
>>
>> {"id": "docid", "add": [{"key": "value"},{"key":"value2"}], "remove":
>> ["key_a", "key_b"], "seq": 5, "changes": [{"_rev": "1-abcdef"}]}
>>
>> (Note: "add" is a list of {"key": "value"} objects rather than a single
>> multi-KV object because duplicate keys can be emitted in a single change)
>>
>> Though I realized that keys can be non-strings, which would make that
>> return format invalid JSON. So, what I'm currently going with is:
>>
>> {"id": "docid", "add": [["key", "value"],["key", "value2"]], "remove":
>> ["key_a", "key_b"], "seq": 5, "changes": [{"_rev": "1-abcdef"}]}
>>
>> I can see why one might find this esoteric/yucky, so I thought I'd express
>> the reasoning and ask for some input/alternatives. So, does anyone have
>> thoughts/preferences?
>>
>> In other news, I'll be rebasing/squashing soon to remove some less-useful
>> commits, but after that I think it'll be mergeable. If anyone feels
>> comfortable to delve into the changes, I'd appreciate some feedback (and
>> hopefully some +1s).
>>
>> Thanks!
>> Ben
>>
>
> Can we have a feature request about that proposal?
>

I don't think this is a feature so much as asking for input on the
format to be used for view changes. For instance, if we have a
document that emit's three keys, do we have there rows in the output,
or a single row that has a list of the keys that document emitted.
Given that the database update sequence is per document it seems to me
that the view sequence should probably also follow that model or else
we have to decide if we want to create a new update sequence or have
rows with the same sequence number.

> It is also not clear what it imply, what is the impact on the memory,
> bandwidth? Why this change is needed?
>

I'd imagine it'd save a bit on bandwidth just by the fact that its not
sending the document id multiple times. I wouldn't expect there to be
a memory difference either way as I'm pretty sure its just formatting
as the bytes go onto the socket more than anything. Granted I haven't
gone back and reviewed the structures so I could be missing details
there as well.

> Also I am lost in the flux of UI commits and i am not sure now, but is the
> replication over view changes is enabled?
>
> - benoit

Re: More View Changes Input

[Benoit Chesneau <bc...@gmail.com>]

On Fri, Oct 3, 2014 at 11:12 AM, benjamin.bastian@gmail.com <
benjamin.bastian@gmail.com> wrote:

> Hey all,
>
> So, the view changes PRs have been hanging out for a bit (review welcome!)
> but things have changed a fair bit since I last emailed wider dev
> community. Bob's had a chance to do a bit of review (thanks Bob!) and
> voiced his preference for there to be a single view changes row returned in
> the response body for each sequence changed (rather than for each seq-key
> pair changed). So, that would mean returning the keys added and removed
> from the index on each row rather than one key per row. Initially, I
> thought a format like:
>
> {"id": "docid", "add": [{"key": "value"},{"key":"value2"}], "remove":
> ["key_a", "key_b"], "seq": 5, "changes": [{"_rev": "1-abcdef"}]}
>
> (Note: "add" is a list of {"key": "value"} objects rather than a single
> multi-KV object because duplicate keys can be emitted in a single change)
>
> Though I realized that keys can be non-strings, which would make that
> return format invalid JSON. So, what I'm currently going with is:
>
> {"id": "docid", "add": [["key", "value"],["key", "value2"]], "remove":
> ["key_a", "key_b"], "seq": 5, "changes": [{"_rev": "1-abcdef"}]}
>
> I can see why one might find this esoteric/yucky, so I thought I'd express
> the reasoning and ask for some input/alternatives. So, does anyone have
> thoughts/preferences?
>
> In other news, I'll be rebasing/squashing soon to remove some less-useful
> commits, but after that I think it'll be mergeable. If anyone feels
> comfortable to delve into the changes, I'd appreciate some feedback (and
> hopefully some +1s).
>
> Thanks!
> Ben
>

Can we have a feature request about that proposal?

It is also not clear what it imply, what is the impact on the memory,
bandwidth? Why this change is needed?

Also I am lost in the flux of UI commits and i am not sure now, but is the
replication over view changes is enabled?

- benoit