You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Ross Bates <rb...@gmail.com> on 2009/07/04 20:48:10 UTC
getting unique set of document id's
Hi All - finally got up an running with 0.9.0 and have been experimenting
with the POST {"keys": ["key1", "key2", ...]} feature and have a question
Take the typical example of a set of blog posts which can be tagged with
1...n tags. Pretend I want to find all posts tagged with both "foo" and
"bar".
A simple map would look something like this
function(doc) {
for(i in doc.tags) {
emit(doc.tags[i], doc._id);
}
}
So now when I post {"keys": ["foo", "bar"]} to the view I get all the
documents tagged "foo", "bar", and also duplicates for any document that is
tagged with both.
Is the best option to deduplicate the doc._id on the client and resubmit to
all docs, or can this be handled in a reduce function?
Thank you for any help!
Ross
-----
@rossbates
rossbates.com
Re: getting unique set of document id's
Posted by Robert Newson <ro...@gmail.com>.
Your goal is achievable with couchdb-lucene
(http://github.com/rnewson/couchdb-lucene), fwiw.
That is, you would add all of the tags for each document to a
full-text view with;
{
"_id":"lucene",
"fulltext": {
"tags": {
"index":"function(doc) { var ret=new Document();
ret.add(doc.tags); return ret }"
},
}
}
and query with something like;
http://localhost:5984/dbname/_fti/lucene/tags?q="key:foo AND key:bar"
couchdb-lucene supports include_docs=true too, so it might be a
one-stop shop solution for you, assuming you can bear the indignity of
Java.
B.
On Tue, Jul 7, 2009 at 7:08 PM, Ross Bates<rb...@gmail.com> wrote:
> Thank you for the follow up Brian. After looking at your examples I think I
> understand where I wasn't clear in how I planned to use the POST {"keys":
> ["foo", "bar"]} statement.
>
> What you and Paul suggested is the fastest route to getting documents that
> are tagged with both foo AND bar, but for my search I am hoping to get foo
> AND/OR bar.
>
> The reason I was including the doc id in the emit was that I was trying to
> play with the reduce to see if I could get it to "reduce" it to a unique
> set.
>
> I can use just the map() function then make the result unique on the client,
> but from a performance standpoint I imagine the couchdb view already knows
> what those unique id's are. I could be totally wrong there though.
>
> The original foo/bar example is simple, but extending it out would prove to
> be powerful for set based analysis (database marketing, customer
> segmentation). To be able to pass {"keys": ["foo", "bar","baz","boo","xyz"]}
> to a view and get back a set of docs that matched *one or more* of the keys
> would be fantastic.
>
> -Ross
>
>
> On Tue, Jul 7, 2009 at 4:05 AM, Brian Candler <B....@pobox.com> wrote:
>
>> On Sun, Jul 05, 2009 at 03:18:33PM -0500, Ross Bates wrote:
>> > Hi Paul - thank you for the pointers. Something I'm unclear on though...
>> > using a sum in the reduce returns something like this for all the tags:
>> >
>> > foo, 3
>> > bar, 5
>> > baz, 7
>> >
>> > When I use the multi-key fetch against the view it doesn't return
>> specific
>> > docid's for each tag, just a subset of tags and their counts.
>> >
>> > POST {"keys": ["foo", "bar"]}
>> >
>> > foo, 3
>> > bar, 5
>> >
>> > How can I get access to the list of docid's which make up the total?
>>
>> You query the view again with reduce=false, which turns it back into a
>> regular map view.
>>
>> The idea is you'd query in this case only for key "foo", because this
>> returns the smallest number of documents, and then filter the result set
>> client-side to documents which also include key "bar".
>>
>> You can read the view using include_docs=true to get the whole docs, or in
>> your view you can emit the parts of doc you're interested in as the value.
>>
>> Your original code was:
>>
>> function(doc) {
>> for(i in doc.tags) {
>> emit(doc.tags[i], doc._id);
>> }
>> }
>>
>> Note that emitting doc._id is not useful (every K/V pair also emits the id
>> as well), so you could have
>>
>> // Query with include_docs=true
>> function(doc) {
>> for(i in doc.tags) {
>> emit(doc.tags[i], null);
>> }
>> }
>>
>> Or:
>>
>> // Index is bigger but faster to read
>> function(doc) {
>> for(i in doc.tags) {
>> emit(doc.tags[i], doc);
>> }
>> }
>>
>> Or:
>>
>> // Just emit the pieces you need when processing the view
>> function(doc) {
>> for(i in doc.tags) {
>> emit(doc.tags[i], {tags: doc.tags});
>> }
>> }
>>
>
Re: getting unique set of document id's
Posted by Brian Candler <B....@pobox.com>.
On Tue, Jul 07, 2009 at 01:08:14PM -0500, Ross Bates wrote:
> Thank you for the follow up Brian. After looking at your examples I
> think I understand where I wasn't clear in how I planned to use the
> POST {"keys": ["foo", "bar"]} statement.
> What you and Paul suggested is the fastest route to getting documents
> that are tagged with both foo AND bar, but for my search I am hoping to
> get foo AND/OR bar.
OK. Then POST {"keys": ["foo", "bar"]} will get you all the documents, and
you can unique them in the client.
> The reason I was including the doc id in the emit was that I was trying
> to play with the reduce to see if I could get it to "reduce" it to a
> unique set.
In fact the docid is already passed to the reduce function; the "keys" array
is actually [key,docid] pairs. But a reduce function is the wrong place to
put this. If you have 1,000,000 documents then the root reduce node would
contain 1,000,000 docids, and that would perform really, really badly. (Note
that each Btree node contains the reduced value of all the docs in that
node, and hence the root Btree node contains the reduced value across all
docs in your database)
> I can use just the map() function then make the result unique on the
> client, but from a performance standpoint I imagine the couchdb view
> already knows what those unique id's are. I could be totally wrong
> there though.
It does know the ids, but the view will emit in order of keys. CouchDB would
need to build up a list of unique IDs while traversing the tree, and it
doesn't do that. It just returns them as it sees them.
So it will walk the Btree index from for all keys="foo", then walk the Btree
index for all keys="bar", and emit the docids.
You can use a Hash or similar data structure on the client to mark doc ids
seen. Actually, since they will be emitted in increasing docid order (for
equal keys) then you could perform a merge instead.
> The original foo/bar example is simple, but extending it out would
> prove to be powerful for set based analysis (database marketing,
> customer segmentation). To be able to pass {"keys": ["foo",
> "bar","baz","boo","xyz"]} to a view and get back a set of docs that
> matched one or more of the keys would be fantastic.
That's pretty much what you get now - with the proviso that the doc may be
returned up to N times if it matches N keys.
Regards,
Brian.
Re: getting unique set of document id's
Posted by Ross Bates <rb...@gmail.com>.
Thank you for the follow up Brian. After looking at your examples I think I
understand where I wasn't clear in how I planned to use the POST {"keys":
["foo", "bar"]} statement.
What you and Paul suggested is the fastest route to getting documents that
are tagged with both foo AND bar, but for my search I am hoping to get foo
AND/OR bar.
The reason I was including the doc id in the emit was that I was trying to
play with the reduce to see if I could get it to "reduce" it to a unique
set.
I can use just the map() function then make the result unique on the client,
but from a performance standpoint I imagine the couchdb view already knows
what those unique id's are. I could be totally wrong there though.
The original foo/bar example is simple, but extending it out would prove to
be powerful for set based analysis (database marketing, customer
segmentation). To be able to pass {"keys": ["foo", "bar","baz","boo","xyz"]}
to a view and get back a set of docs that matched *one or more* of the keys
would be fantastic.
-Ross
On Tue, Jul 7, 2009 at 4:05 AM, Brian Candler <B....@pobox.com> wrote:
> On Sun, Jul 05, 2009 at 03:18:33PM -0500, Ross Bates wrote:
> > Hi Paul - thank you for the pointers. Something I'm unclear on though...
> > using a sum in the reduce returns something like this for all the tags:
> >
> > foo, 3
> > bar, 5
> > baz, 7
> >
> > When I use the multi-key fetch against the view it doesn't return
> specific
> > docid's for each tag, just a subset of tags and their counts.
> >
> > POST {"keys": ["foo", "bar"]}
> >
> > foo, 3
> > bar, 5
> >
> > How can I get access to the list of docid's which make up the total?
>
> You query the view again with reduce=false, which turns it back into a
> regular map view.
>
> The idea is you'd query in this case only for key "foo", because this
> returns the smallest number of documents, and then filter the result set
> client-side to documents which also include key "bar".
>
> You can read the view using include_docs=true to get the whole docs, or in
> your view you can emit the parts of doc you're interested in as the value.
>
> Your original code was:
>
> function(doc) {
> for(i in doc.tags) {
> emit(doc.tags[i], doc._id);
> }
> }
>
> Note that emitting doc._id is not useful (every K/V pair also emits the id
> as well), so you could have
>
> // Query with include_docs=true
> function(doc) {
> for(i in doc.tags) {
> emit(doc.tags[i], null);
> }
> }
>
> Or:
>
> // Index is bigger but faster to read
> function(doc) {
> for(i in doc.tags) {
> emit(doc.tags[i], doc);
> }
> }
>
> Or:
>
> // Just emit the pieces you need when processing the view
> function(doc) {
> for(i in doc.tags) {
> emit(doc.tags[i], {tags: doc.tags});
> }
> }
>
Re: getting unique set of document id's
Posted by Brian Candler <B....@pobox.com>.
On Sun, Jul 05, 2009 at 03:18:33PM -0500, Ross Bates wrote:
> Hi Paul - thank you for the pointers. Something I'm unclear on though...
> using a sum in the reduce returns something like this for all the tags:
>
> foo, 3
> bar, 5
> baz, 7
>
> When I use the multi-key fetch against the view it doesn't return specific
> docid's for each tag, just a subset of tags and their counts.
>
> POST {"keys": ["foo", "bar"]}
>
> foo, 3
> bar, 5
>
> How can I get access to the list of docid's which make up the total?
You query the view again with reduce=false, which turns it back into a
regular map view.
The idea is you'd query in this case only for key "foo", because this
returns the smallest number of documents, and then filter the result set
client-side to documents which also include key "bar".
You can read the view using include_docs=true to get the whole docs, or in
your view you can emit the parts of doc you're interested in as the value.
Your original code was:
function(doc) {
for(i in doc.tags) {
emit(doc.tags[i], doc._id);
}
}
Note that emitting doc._id is not useful (every K/V pair also emits the id
as well), so you could have
// Query with include_docs=true
function(doc) {
for(i in doc.tags) {
emit(doc.tags[i], null);
}
}
Or:
// Index is bigger but faster to read
function(doc) {
for(i in doc.tags) {
emit(doc.tags[i], doc);
}
}
Or:
// Just emit the pieces you need when processing the view
function(doc) {
for(i in doc.tags) {
emit(doc.tags[i], {tags: doc.tags});
}
}
Re: getting unique set of document id's
Posted by Paul Davis <pa...@gmail.com>.
On Sun, Jul 5, 2009 at 4:18 PM, Ross Bates<rb...@gmail.com> wrote:
> Hi Paul - thank you for the pointers. Something I'm unclear on though...
> using a sum in the reduce returns something like this for all the tags:
>
> foo, 3
> bar, 5
> baz, 7
>
> When I use the multi-key fetch against the view it doesn't return specific
> docid's for each tag, just a subset of tags and their counts.
>
> POST {"keys": ["foo", "bar"]}
>
> foo, 3
> bar, 5
>
> How can I get access to the list of docid's which make up the total?
>
The idea is that now that you have a set of tags you know what order
to do the client side join. Ie, in this case you start with retrieving
the docids that have the tag 'foo' then 'bar'. this will limit the
number of tags to a maximum of 3 because that's the smallest number.
HTH,
Paul Davis
>
>
> On Sat, Jul 4, 2009 at 3:52 PM, Paul Davis <pa...@gmail.com>wrote:
>
>> On Sat, Jul 4, 2009 at 2:48 PM, Ross Bates<rb...@gmail.com> wrote:
>> > Hi All - finally got up an running with 0.9.0 and have been experimenting
>> > with the POST {"keys": ["key1", "key2", ...]} feature and have a question
>> >
>> > Take the typical example of a set of blog posts which can be tagged with
>> > 1...n tags. Pretend I want to find all posts tagged with both "foo" and
>> > "bar".
>> >
>> > A simple map would look something like this
>> >
>> > function(doc) {
>> > for(i in doc.tags) {
>> > emit(doc.tags[i], doc._id);
>> > }
>> > }
>> >
>> > So now when I post {"keys": ["foo", "bar"]} to the view I get all the
>> > documents tagged "foo", "bar", and also duplicates for any document that
>> is
>> > tagged with both.
>> >
>> > Is the best option to deduplicate the doc._id on the client and resubmit
>> to
>> > all docs, or can this be handled in a reduce function?
>> >
>> > Thank you for any help!
>> >
>> > Ross
>> >
>> > -----
>> > @rossbates
>> > rossbates.com
>> >
>>
>> Assuming the set of tags you want to query isn't known until query
>> time then you need to go with doing the client side intersection. A
>> method for making this process a bit faster is to create a view that
>> lists the tag count using a standard map/reduce of
>> doc.tags.forEach(function(tag) {emit(tag, 1);}); and a reduce of
>> "return sum(values)". Then query this view with the multi-key fetch to
>> get the list of docids for each tag, then query in order of ascending
>> counts.
>>
>> Paul Davis
>>
>
Re: getting unique set of document id's
Posted by Ross Bates <rb...@gmail.com>.
Hi Paul - thank you for the pointers. Something I'm unclear on though...
using a sum in the reduce returns something like this for all the tags:
foo, 3
bar, 5
baz, 7
When I use the multi-key fetch against the view it doesn't return specific
docid's for each tag, just a subset of tags and their counts.
POST {"keys": ["foo", "bar"]}
foo, 3
bar, 5
How can I get access to the list of docid's which make up the total?
On Sat, Jul 4, 2009 at 3:52 PM, Paul Davis <pa...@gmail.com>wrote:
> On Sat, Jul 4, 2009 at 2:48 PM, Ross Bates<rb...@gmail.com> wrote:
> > Hi All - finally got up an running with 0.9.0 and have been experimenting
> > with the POST {"keys": ["key1", "key2", ...]} feature and have a question
> >
> > Take the typical example of a set of blog posts which can be tagged with
> > 1...n tags. Pretend I want to find all posts tagged with both "foo" and
> > "bar".
> >
> > A simple map would look something like this
> >
> > function(doc) {
> > for(i in doc.tags) {
> > emit(doc.tags[i], doc._id);
> > }
> > }
> >
> > So now when I post {"keys": ["foo", "bar"]} to the view I get all the
> > documents tagged "foo", "bar", and also duplicates for any document that
> is
> > tagged with both.
> >
> > Is the best option to deduplicate the doc._id on the client and resubmit
> to
> > all docs, or can this be handled in a reduce function?
> >
> > Thank you for any help!
> >
> > Ross
> >
> > -----
> > @rossbates
> > rossbates.com
> >
>
> Assuming the set of tags you want to query isn't known until query
> time then you need to go with doing the client side intersection. A
> method for making this process a bit faster is to create a view that
> lists the tag count using a standard map/reduce of
> doc.tags.forEach(function(tag) {emit(tag, 1);}); and a reduce of
> "return sum(values)". Then query this view with the multi-key fetch to
> get the list of docids for each tag, then query in order of ascending
> counts.
>
> Paul Davis
>
Re: getting unique set of document id's
Posted by Paul Davis <pa...@gmail.com>.
On Sat, Jul 4, 2009 at 2:48 PM, Ross Bates<rb...@gmail.com> wrote:
> Hi All - finally got up an running with 0.9.0 and have been experimenting
> with the POST {"keys": ["key1", "key2", ...]} feature and have a question
>
> Take the typical example of a set of blog posts which can be tagged with
> 1...n tags. Pretend I want to find all posts tagged with both "foo" and
> "bar".
>
> A simple map would look something like this
>
> function(doc) {
> for(i in doc.tags) {
> emit(doc.tags[i], doc._id);
> }
> }
>
> So now when I post {"keys": ["foo", "bar"]} to the view I get all the
> documents tagged "foo", "bar", and also duplicates for any document that is
> tagged with both.
>
> Is the best option to deduplicate the doc._id on the client and resubmit to
> all docs, or can this be handled in a reduce function?
>
> Thank you for any help!
>
> Ross
>
> -----
> @rossbates
> rossbates.com
>
Assuming the set of tags you want to query isn't known until query
time then you need to go with doing the client side intersection. A
method for making this process a bit faster is to create a view that
lists the tag count using a standard map/reduce of
doc.tags.forEach(function(tag) {emit(tag, 1);}); and a reduce of
"return sum(values)". Then query this view with the multi-key fetch to
get the list of docids for each tag, then query in order of ascending
counts.
Paul Davis