You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by MK <mk...@cognitivedissonance.ca> on 2011/05/22 19:43:28 UTC
couchDB and node.js double bind
I'm pseudo cross posting this to the nodejs list, but the replies will
not cross-post so hopefully no one is too troubled by that.
I'm new to both and I've hit a bit of a conundrum.
I have a node request handler that presents a view of a couchDB
document. The document contains an array of other document id's. In
the view, I want to list those documents, and include a piece of
information from them (call it the "short description"). This means,
in addition to querying couch for the primary doc, I have to query it
for the short description of all the docs listed in the primary doc's
array of related pages.
Sounds simple. I have the array, now all I have to do is
var shortDescs = new Array();
for (i in array) {
//query couch for data
shortDescs.push(data)
}
Except the only way to query couch from node is via asynchronous http
requests. Ie, when this loop completes, it is very unlikely that the
shortDescs array will be properly populated. So this is a dead end.
That's the way nodejs works, and it has that limitation. The way couch
works also results in some limitations. You cannot make a single
request for an arbitrary set of documents. You can set up a "view"
function that will return data from all the documents that match a
specific criteria *but that criteria must be hardcoded*. In other
words, you cannot create a view function that takes arbitrary
parameters, such as an array of ids.
In short, AFAICT, it is simply impossible to do this using couchDB and
nodejs together, which seems a shame.
I can see three choices to work around this:
- Make the primary document's array an array of objects which contains
the short desc from the other docs. Obviously, this will result in a
tremendous amount of reduplication and wasted space if every doc has an
array of related docs that must contain information from those other
docs because I cannot query couch for an array of arbitrary docs by id.
- Do a whole bunch of ajax calls when the page loads to complete the
information for each doc in the array, because I cannot use node to
assemble information from an arbitrary array of http requests to
couchDB. Obviously, this will result in increased server load.
- call a couch view to return all short descriptions +
ids indiscriminately, and then choose the ones I want. Probably the
best choice, but IMO still a very poor compromise.
Is there any other solution anyone sees here?
--
"Enthusiasm is not the enemy of the intellect." (said of Irving Howe)
"The angel of history[...]is turned toward the past." (Walter Benjamin)
Re: couchDB and node.js double bind
Posted by MK <mk...@cognitivedissonance.ca>.
On Sun, 22 May 2011 23:49:38 +0300
afters <af...@gmail.com> wrote:
> Luckily for you, some of your assumptions are wrong.
>
> You should have no problem doing an "async-for-loop" with Node. Read
> this:
> http://metaduck.com/post/2675027550/asynchronous-iteration-patterns-in-node-js
Thanks, dunno why I didn't think of that (using a counter in the
callback and then calling a named func when it hits the total).
--
"Enthusiasm is not the enemy of the intellect." (said of Irving Howe)
"The angel of history[...]is turned toward the past." (Walter Benjamin)
Re: couchDB and node.js double bind
Posted by afters <af...@gmail.com>.
MK,
Luckily for you, some of your assumptions are wrong.
You should have no problem doing an "async-for-loop" with Node. Read this:
http://metaduck.com/post/2675027550/asynchronous-iteration-patterns-in-node-js
Also, you might not need a for-loop, as Couch lets you fetch several
documents at the same time:
http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API#Fetch_Multiple_Documents_With_a_Single_Request
Cheers,
a.
On 22 May 2011 20:43, MK <mk...@cognitivedissonance.ca> wrote:
> I'm pseudo cross posting this to the nodejs list, but the replies will
> not cross-post so hopefully no one is too troubled by that.
>
> I'm new to both and I've hit a bit of a conundrum.
>
> I have a node request handler that presents a view of a couchDB
> document. The document contains an array of other document id's. In
> the view, I want to list those documents, and include a piece of
> information from them (call it the "short description"). This means,
> in addition to querying couch for the primary doc, I have to query it
> for the short description of all the docs listed in the primary doc's
> array of related pages.
>
> Sounds simple. I have the array, now all I have to do is
>
> var shortDescs = new Array();
> for (i in array) {
> //query couch for data
> shortDescs.push(data)
> }
>
> Except the only way to query couch from node is via asynchronous http
> requests. Ie, when this loop completes, it is very unlikely that the
> shortDescs array will be properly populated. So this is a dead end.
>
> That's the way nodejs works, and it has that limitation. The way couch
> works also results in some limitations. You cannot make a single
> request for an arbitrary set of documents. You can set up a "view"
> function that will return data from all the documents that match a
> specific criteria *but that criteria must be hardcoded*. In other
> words, you cannot create a view function that takes arbitrary
> parameters, such as an array of ids.
>
> In short, AFAICT, it is simply impossible to do this using couchDB and
> nodejs together, which seems a shame.
>
> I can see three choices to work around this:
>
> - Make the primary document's array an array of objects which contains
> the short desc from the other docs. Obviously, this will result in a
> tremendous amount of reduplication and wasted space if every doc has an
> array of related docs that must contain information from those other
> docs because I cannot query couch for an array of arbitrary docs by id.
>
> - Do a whole bunch of ajax calls when the page loads to complete the
> information for each doc in the array, because I cannot use node to
> assemble information from an arbitrary array of http requests to
> couchDB. Obviously, this will result in increased server load.
>
> - call a couch view to return all short descriptions +
> ids indiscriminately, and then choose the ones I want. Probably the
> best choice, but IMO still a very poor compromise.
>
> Is there any other solution anyone sees here?
>
> --
> "Enthusiasm is not the enemy of the intellect." (said of Irving Howe)
> "The angel of history[...]is turned toward the past." (Walter Benjamin)
>
>
Re: couchDB and node.js double bind
Posted by MK <mk...@cognitivedissonance.ca>.
On Sun, 22 May 2011 16:52:02 -0400
Sean Copenhaver <se...@gmail.com> wrote:
> _view/view?startkey=[<id>],include_docs=true
"include_docs" will definitely help here, thanks! I had not noticed
that option.
--
"Enthusiasm is not the enemy of the intellect." (said of Irving Howe)
"The angel of history[...]is turned toward the past." (Walter Benjamin)
Re: couchDB and node.js double bind
Posted by Sean Copenhaver <se...@gmail.com>.
If the relationships were bi-directional then perhaps your map function could look like this:
function(doc) {
emit([doc._id], null);
for(var i = 0; i < doc.related.length; i++){
emit([doc.related[i], doc._id], null);
}
}
When you query for a specific doc id you would get that document plus all the other related documents. Something like:
_view/view?startkey=[<id>],include_docs=true
If they were one way then yeah, I believe you would need to do some looping or make a second request.
--
Sean Copenhaver
Sent with Sparrow
On Sunday, May 22, 2011 at 4:38 PM, MK wrote:
> On Sun, 22 May 2011 14:12:05 -0400
> Sean Copenhaver <se...@gmail.com> wrote:
> > I hope I understand what you are after, but here's what I got. You
> > have a parent document that has a list of ids for it's children or
> > some sort of related documents? You want to be able to query for the
> > parent in the relationship and get a list of children ids with a
> > short description?
> >
> > Sounds like you need to reverse the relationship (or make it
> > bi-directional). Have the children keep an id to their parent
>
> All the documents have exactly the same structure -- there is no
> parent/child relation -- which includes a short description and an
> array of other (identically structured docs) to which this one is
> related. I want to display one of these, and in the display, list the
> related docs including their short desc.
>
>
> --
> "Enthusiasm is not the enemy of the intellect." (said of Irving Howe)
> "The angel of history[...]is turned toward the past." (Walter Benjamin)
>
Re: couchDB and node.js double bind
Posted by MK <mk...@cognitivedissonance.ca>.
On Sun, 22 May 2011 14:12:05 -0400
Sean Copenhaver <se...@gmail.com> wrote:
> I hope I understand what you are after, but here's what I got. You
> have a parent document that has a list of ids for it's children or
> some sort of related documents? You want to be able to query for the
> parent in the relationship and get a list of children ids with a
> short description?
>
> Sounds like you need to reverse the relationship (or make it
> bi-directional). Have the children keep an id to their parent
All the documents have exactly the same structure -- there is no
parent/child relation -- which includes a short description and an
array of other (identically structured docs) to which this one is
related. I want to display one of these, and in the display, list the
related docs including their short desc.
--
"Enthusiasm is not the enemy of the intellect." (said of Irving Howe)
"The angel of history[...]is turned toward the past." (Walter Benjamin)
Re: couchDB and node.js double bind
Posted by Sean Copenhaver <se...@gmail.com>.
I hope I understand what you are after, but here's what I got. You have a parent document that has a list of ids for it's children or some sort of related documents? You want to be able to query for the parent in the relationship and get a list of children ids with a short description?
Sounds like you need to reverse the relationship (or make it bi-directional). Have the children keep an id to their parent and use something like the wiki lays out:
http://wiki.apache.org/couchdb/EntityRelationship#One_to_Many:_Separate_documents
With that you should be able to query for the parent and get everything you wanted. For the children your map function would emit a value of the description you are after. For the parent the value emitted might be it's doc. Then if you need to turn that into one big document or some other format you can use a list function:
http://guide.couchdb.org/draft/transforming.html
I hope that helps.
--
Sean Copenhaver
Sent with Sparrow
On Sunday, May 22, 2011 at 1:55 PM, Ryan Ramage wrote:
> You create a view, emitting all the docs you will need and the key
> will be the primary id that you will query on, and then create a list
> function over top? Then you would only have one thing to query.
>
> http://wiki.apache.org/couchdb/Formatting_with_Show_and_List
>
>
>
> On Sun, May 22, 2011 at 11:43 AM, MK <mk...@cognitivedissonance.ca> wrote:
> > I'm pseudo cross posting this to the nodejs list, but the replies will
> > not cross-post so hopefully no one is too troubled by that.
> >
> > I'm new to both and I've hit a bit of a conundrum.
> >
> > I have a node request handler that presents a view of a couchDB
> > document. The document contains an array of other document id's. In
> > the view, I want to list those documents, and include a piece of
> > information from them (call it the "short description"). This means,
> > in addition to querying couch for the primary doc, I have to query it
> > for the short description of all the docs listed in the primary doc's
> > array of related pages.
> >
> > Sounds simple. I have the array, now all I have to do is
> >
> > var shortDescs = new Array();
> > for (i in array) {
> > //query couch for data
> > shortDescs.push(data)
> > }
> >
> > Except the only way to query couch from node is via asynchronous http
> > requests. Ie, when this loop completes, it is very unlikely that the
> > shortDescs array will be properly populated. So this is a dead end.
> >
> > That's the way nodejs works, and it has that limitation. The way couch
> > works also results in some limitations. You cannot make a single
> > request for an arbitrary set of documents. You can set up a "view"
> > function that will return data from all the documents that match a
> > specific criteria *but that criteria must be hardcoded*. In other
> > words, you cannot create a view function that takes arbitrary
> > parameters, such as an array of ids.
> >
> > In short, AFAICT, it is simply impossible to do this using couchDB and
> > nodejs together, which seems a shame.
> >
> > I can see three choices to work around this:
> >
> > - Make the primary document's array an array of objects which contains
> > the short desc from the other docs. Obviously, this will result in a
> > tremendous amount of reduplication and wasted space if every doc has an
> > array of related docs that must contain information from those other
> > docs because I cannot query couch for an array of arbitrary docs by id.
> >
> > - Do a whole bunch of ajax calls when the page loads to complete the
> > information for each doc in the array, because I cannot use node to
> > assemble information from an arbitrary array of http requests to
> > couchDB. Obviously, this will result in increased server load.
> >
> > - call a couch view to return all short descriptions +
> > ids indiscriminately, and then choose the ones I want. Probably the
> > best choice, but IMO still a very poor compromise.
> >
> > Is there any other solution anyone sees here?
> >
> > --
> > "Enthusiasm is not the enemy of the intellect." (said of Irving Howe)
> > "The angel of history[...]is turned toward the past." (Walter Benjamin)
>
Re: couchDB and node.js double bind
Posted by MK <mk...@cognitivedissonance.ca>.
On Sun, 22 May 2011 11:55:42 -0600
Ryan Ramage <ry...@gmail.com> wrote:
> You create a view, emitting all the docs you will need and the key
> will be the primary id that you will query on, and then create a list
> function over top? Then you would only have one thing to query.
>
> http://wiki.apache.org/couchdb/Formatting_with_Show_and_List
This is very tantalizing, I have not played with lists yet and
hopefully can work something out. From looking at the docs, I don't
think using the primary id as a "key" will work, this will simply give
me the data for one doc and not many. But if I can use custom query
parameters in the list function, I can transform the array from the
primary doc into an object and test against the keys from that.
--
"Enthusiasm is not the enemy of the intellect." (said of Irving Howe)
"The angel of history[...]is turned toward the past." (Walter Benjamin)
Re: couchDB and node.js double bind
Posted by Ryan Ramage <ry...@gmail.com>.
You create a view, emitting all the docs you will need and the key
will be the primary id that you will query on, and then create a list
function over top? Then you would only have one thing to query.
http://wiki.apache.org/couchdb/Formatting_with_Show_and_List
On Sun, May 22, 2011 at 11:43 AM, MK <mk...@cognitivedissonance.ca> wrote:
> I'm pseudo cross posting this to the nodejs list, but the replies will
> not cross-post so hopefully no one is too troubled by that.
>
> I'm new to both and I've hit a bit of a conundrum.
>
> I have a node request handler that presents a view of a couchDB
> document. The document contains an array of other document id's. In
> the view, I want to list those documents, and include a piece of
> information from them (call it the "short description"). This means,
> in addition to querying couch for the primary doc, I have to query it
> for the short description of all the docs listed in the primary doc's
> array of related pages.
>
> Sounds simple. I have the array, now all I have to do is
>
> var shortDescs = new Array();
> for (i in array) {
> //query couch for data
> shortDescs.push(data)
> }
>
> Except the only way to query couch from node is via asynchronous http
> requests. Ie, when this loop completes, it is very unlikely that the
> shortDescs array will be properly populated. So this is a dead end.
>
> That's the way nodejs works, and it has that limitation. The way couch
> works also results in some limitations. You cannot make a single
> request for an arbitrary set of documents. You can set up a "view"
> function that will return data from all the documents that match a
> specific criteria *but that criteria must be hardcoded*. In other
> words, you cannot create a view function that takes arbitrary
> parameters, such as an array of ids.
>
> In short, AFAICT, it is simply impossible to do this using couchDB and
> nodejs together, which seems a shame.
>
> I can see three choices to work around this:
>
> - Make the primary document's array an array of objects which contains
> the short desc from the other docs. Obviously, this will result in a
> tremendous amount of reduplication and wasted space if every doc has an
> array of related docs that must contain information from those other
> docs because I cannot query couch for an array of arbitrary docs by id.
>
> - Do a whole bunch of ajax calls when the page loads to complete the
> information for each doc in the array, because I cannot use node to
> assemble information from an arbitrary array of http requests to
> couchDB. Obviously, this will result in increased server load.
>
> - call a couch view to return all short descriptions +
> ids indiscriminately, and then choose the ones I want. Probably the
> best choice, but IMO still a very poor compromise.
>
> Is there any other solution anyone sees here?
>
> --
> "Enthusiasm is not the enemy of the intellect." (said of Irving Howe)
> "The angel of history[...]is turned toward the past." (Walter Benjamin)
>
>