You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Avi Flax <av...@arc90.com> on 2010/08/01 23:43:35 UTC
startkey_docid and "wildcard" paging
Hi all, quick question — i'm having a little trouble with paging with
startkey_docid
I have a view whose keys look like: ["foo", "bar", "2010-06-10T12:14:35Z"]
I'm querying the view with startkey=["foo", "bar"] and endkey=["foo","bar",{}]
Which is returning what i expect, but i'm having trouble doing paging
with this scenario; it almost seems like startkey_docid is being
ignored.
Any ideas?
Thanks,
Avi
Avi Flax » Partner » Arc90 » http://arc90.com
Kindling: Innovation through Collaboration » http://kindlingapp.com
Re: startkey_docid and "wildcard" paging
Posted by Avi Flax <av...@arc90.com>.
OK, it's working perfectly now.
Thanks again Chris!
Avi
On Sun, Aug 1, 2010 at 20:35, Avi Flax <av...@arc90.com> wrote:
> Actually I've been paging that way for a while, it's only today that I
> added the timestamp to my view and had to start using the "wildcard"
> approach. But now I think I see what I was missing — I only need to
> use the "wildcard" for my first page. I'll give it a try and report
> back.
Re: startkey_docid and "wildcard" paging
Posted by Avi Flax <av...@arc90.com>.
Actually I've been paging that way for a while, it's only today that I
added the timestamp to my view and had to start using the "wildcard"
approach. But now I think I see what I was missing — I only need to
use the "wildcard" for my first page. I'll give it a try and report
back.
Thanks!
On Sun, Aug 1, 2010 at 19:44, J Chris Anderson <jc...@apache.org> wrote:
> Ffirst just stop using startkey_docid at all. It has no relevance for your current use case. It will start to matter, someday when you have billions of rows, such that you start to regularly have more than 11 rows with the same exact tags and timestamp.
>
> Your query string for the second request should be:
>
> ?limit=11&startkey=KEY_OF_THE_LAST_ROW_FROM_PAGE_ONE&descending=true&skip=1&endkey=SAME_OLD_SAME_OLD
>
> Essentially you are building a linked list. You tell Couch: "I know there is a doc with key X, cause it was at the bottom of page 1, so lets put that row at the top of page 2. Oh but since we just saw that, lets skip just that row this time."
>
> Try that, and page through my blog to see how I do it, if you are still stuck.
>
> Chris
>
>> Does that help clarify what my confusion is?
>>
>> Thanks!
>> Avi
>
>
Re: startkey_docid and "wildcard" paging
Posted by J Chris Anderson <jc...@apache.org>.
On Aug 1, 2010, at 4:38 PM, Avi Flax wrote:
> On Sun, Aug 1, 2010 at 19:26, J Chris Anderson <jc...@gmail.com> wrote:
>
>> first query is just ?limit=10
>>
>> second query uses the key of the last row of the first query as the startkey (and the id of that row's doc as startkey_docid), and limit=10, and skip=1 (to avoid getting the old last row as the new first row).
>
> Thanks Chris, but I feel like I'm missing something. Let me get more concrete.
>
> My query for page 1 looks like this:
>
> http://localhost:5984/avi/_design/main/_view/by_label?limit=11&descending=true&startkey=[%22pool%22,%22cooking%22,{}]&endkey=[%22pool%22,%22cooking%22]
>
> The last row, the 11th, has the ID E2A0D3D1-3E37-47D1-BC5E-F84CCD60E1B7.
>
> So I tried just adding
> startkey_docid=E2A0D3D1-3E37-47D1-BC5E-F84CCD60E1B7 to the URL. But
> that didn't work — I got the same set of rows.
>
Ffirst just stop using startkey_docid at all. It has no relevance for your current use case. It will start to matter, someday when you have billions of rows, such that you start to regularly have more than 11 rows with the same exact tags and timestamp.
Your query string for the second request should be:
?limit=11&startkey=KEY_OF_THE_LAST_ROW_FROM_PAGE_ONE&descending=true&skip=1&endkey=SAME_OLD_SAME_OLD
Essentially you are building a linked list. You tell Couch: "I know there is a doc with key X, cause it was at the bottom of page 1, so lets put that row at the top of page 2. Oh but since we just saw that, lets skip just that row this time."
Try that, and page through my blog to see how I do it, if you are still stuck.
Chris
> Does that help clarify what my confusion is?
>
> Thanks!
> Avi
Re: startkey_docid and "wildcard" paging
Posted by Avi Flax <av...@arc90.com>.
On Sun, Aug 1, 2010 at 19:26, J Chris Anderson <jc...@gmail.com> wrote:
> first query is just ?limit=10
>
> second query uses the key of the last row of the first query as the startkey (and the id of that row's doc as startkey_docid), and limit=10, and skip=1 (to avoid getting the old last row as the new first row).
Thanks Chris, but I feel like I'm missing something. Let me get more concrete.
My query for page 1 looks like this:
http://localhost:5984/avi/_design/main/_view/by_label?limit=11&descending=true&startkey=[%22pool%22,%22cooking%22,{}]&endkey=[%22pool%22,%22cooking%22]
The last row, the 11th, has the ID E2A0D3D1-3E37-47D1-BC5E-F84CCD60E1B7.
So I tried just adding
startkey_docid=E2A0D3D1-3E37-47D1-BC5E-F84CCD60E1B7 to the URL. But
that didn't work — I got the same set of rows.
Does that help clarify what my confusion is?
Thanks!
Avi
Re: startkey_docid and "wildcard" paging
Posted by J Chris Anderson <jc...@gmail.com>.
On Aug 1, 2010, at 4:22 PM, Avi Flax wrote:
> Thanks Chris.
>
> I think you're saying that what I was trying isn't going to work.
>
> Maybe if I share my actual use case, you might be able to point me in the
> right direction.
>
> I have keys in my view like this:
>
> ["a", "1", "2010-06-10T12:14:35Z"],
> ["a", "1", "2010-06-10T13:14:35Z"],
> ["a", "2", "2010-06-10T14:14:35Z"],
> ["a", "2", "2010-06-10T15:14:35Z"],
> ["b", "1", "2010-06-10T16:14:35Z"],
> ["b", "1", "2010-06-10T17:14:35Z"],
> ["b", "2", "2010-06-10T18:14:35Z"],
> ["b", "2", "2010-06-10T19:14:35Z"]
>
> The first two elements are present because I want to filter by them — I want
> to get rows for "a" and "2", for example. The third element is present
> because I want the results sorted by date.
>
> So — how can I add paging into this mix? What if I have 20 rows of "a" and
> "2", each with a different date and a different doc_id, and I'm paging in
> groups of 10 — if I can't use startkey_docid, then what can I do?
>
first query is just ?limit=10
second query uses the key of the last row of the first query as the startkey (and the id of that row's doc as startkey_docid), and limit=10, and skip=1 (to avoid getting the old last row as the new first row).
this is how I page on my blog: http://jchrisa.net
Chris
> Thanks!
> Avi
>
> On Sun, Aug 1, 2010 at 19:10, J Chris Anderson <jc...@apache.org> wrote:
>>
>> On Aug 1, 2010, at 2:43 PM, Avi Flax wrote:
>>
>>> Hi all, quick question — i'm having a little trouble with paging with
>>> startkey_docid
>>>
>>> I have a view whose keys look like: ["foo", "bar",
> "2010-06-10T12:14:35Z"]
>>>
>>> I'm querying the view with startkey=["foo", "bar"] and
> endkey=["foo","bar",{}]
>>>
>>> Which is returning what i expect, but i'm having trouble doing paging
>>> with this scenario; it almost seems like startkey_docid is being
>>> ignored.
>>>
>>
>> startkey_docid is only useful when paging through a view where there are
> more rows emitted for key X than your page size.
>>
>> startkey_docid is subservient to the key. the internal implementation of
> collation looks like:
>>
>> [key, docid]
>>
>> so in your case
>>
>> [ ["foo", "bar", "2010-06-10T12:14:35Z"], docid]
>>
>> so the startkey_docid only has any effect when the key is an exact match
> for the emitted key.
>>
>> I hope that helps.
>>
>> Chris
>>
>>> Any ideas?
>>>
>>> Thanks,
>>> Avi
>>>
>>> Avi Flax » Partner » Arc90 » http://arc90.com
>>> Kindling: Innovation through Collaboration » http://kindlingapp.com
>>
>>
Re: startkey_docid and "wildcard" paging
Posted by Avi Flax <av...@arc90.com>.
Thanks Chris.
I think you're saying that what I was trying isn't going to work.
Maybe if I share my actual use case, you might be able to point me in the
right direction.
I have keys in my view like this:
["a", "1", "2010-06-10T12:14:35Z"],
["a", "1", "2010-06-10T13:14:35Z"],
["a", "2", "2010-06-10T14:14:35Z"],
["a", "2", "2010-06-10T15:14:35Z"],
["b", "1", "2010-06-10T16:14:35Z"],
["b", "1", "2010-06-10T17:14:35Z"],
["b", "2", "2010-06-10T18:14:35Z"],
["b", "2", "2010-06-10T19:14:35Z"]
The first two elements are present because I want to filter by them — I want
to get rows for "a" and "2", for example. The third element is present
because I want the results sorted by date.
So — how can I add paging into this mix? What if I have 20 rows of "a" and
"2", each with a different date and a different doc_id, and I'm paging in
groups of 10 — if I can't use startkey_docid, then what can I do?
Thanks!
Avi
On Sun, Aug 1, 2010 at 19:10, J Chris Anderson <jc...@apache.org> wrote:
>
> On Aug 1, 2010, at 2:43 PM, Avi Flax wrote:
>
>> Hi all, quick question — i'm having a little trouble with paging with
>> startkey_docid
>>
>> I have a view whose keys look like: ["foo", "bar",
"2010-06-10T12:14:35Z"]
>>
>> I'm querying the view with startkey=["foo", "bar"] and
endkey=["foo","bar",{}]
>>
>> Which is returning what i expect, but i'm having trouble doing paging
>> with this scenario; it almost seems like startkey_docid is being
>> ignored.
>>
>
> startkey_docid is only useful when paging through a view where there are
more rows emitted for key X than your page size.
>
> startkey_docid is subservient to the key. the internal implementation of
collation looks like:
>
> [key, docid]
>
> so in your case
>
> [ ["foo", "bar", "2010-06-10T12:14:35Z"], docid]
>
> so the startkey_docid only has any effect when the key is an exact match
for the emitted key.
>
> I hope that helps.
>
> Chris
>
>> Any ideas?
>>
>> Thanks,
>> Avi
>>
>> Avi Flax » Partner » Arc90 » http://arc90.com
>> Kindling: Innovation through Collaboration » http://kindlingapp.com
>
>
Re: startkey_docid and "wildcard" paging
Posted by J Chris Anderson <jc...@apache.org>.
On Aug 1, 2010, at 2:43 PM, Avi Flax wrote:
> Hi all, quick question — i'm having a little trouble with paging with
> startkey_docid
>
> I have a view whose keys look like: ["foo", "bar", "2010-06-10T12:14:35Z"]
>
> I'm querying the view with startkey=["foo", "bar"] and endkey=["foo","bar",{}]
>
> Which is returning what i expect, but i'm having trouble doing paging
> with this scenario; it almost seems like startkey_docid is being
> ignored.
>
startkey_docid is only useful when paging through a view where there are more rows emitted for key X than your page size.
startkey_docid is subservient to the key. the internal implementation of collation looks like:
[key, docid]
so in your case
[ ["foo", "bar", "2010-06-10T12:14:35Z"], docid]
so the startkey_docid only has any effect when the key is an exact match for the emitted key.
I hope that helps.
Chris
> Any ideas?
>
> Thanks,
> Avi
>
> Avi Flax » Partner » Arc90 » http://arc90.com
> Kindling: Innovation through Collaboration » http://kindlingapp.com