You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Daniel Sheppard <da...@pronto.com.au> on 2008/04/18 07:01:13 UTC
Is the current Map/Reduce plan too restrictive?
Suppose, I have three document types:
{ _id: "1234", type: "category" }
{ _id: "5678", type: "group", :category_id: "1234" }
{ _id: "9012", type: "item", :group_id: "5678" }
Now, to get all the items in a group, I'd have a view like so:
function(doc) {
if(doc.type == "item") {
map([doc.group_id, doc]);
}
}
and to get all the groups in a category:
function(doc) {
if(doc.type == "group") {
map([doc.category_id, doc]);
}
}
However, if I wanted to get all the items in a category, I'd need to
interrogate the db for the list of groups and then grab the list of
items for each group. Is that the only way? I tried thinking of various
ways to map/reduce the data, or double-map the data, but consistently
came up blank.
I know that this could be solved easily by denormalising the data so
that the items stored their category id, but I thought I'd be stubborn
to figure out other ways to do it.
It would be possible if the reducer was no longer required to be a
combiner and could actually emit multiple documents from its data
however it wanted.
First map:
function(doc) {
if(doc.type == "item") {
map([doc.group_id, doc])
} else if (doc.type =="group") {
map([doc._id, doc])
}
}
Then a reduction of each key into new documents:
function(key, values) {
var group;
for(i=0; i < values.length; i++) {
if(values[i].type == "group") {
group = values[i];
}
}
if(group) {
for(i=0; i < values.length; i++) {
if(values[i].type == "item") {
map(group.category_id, item);
}
}
}
}
(and more advanced queries could go through further map/reduce cycles).
Damien's blog discussions on the map/reduce implementation seem to imply
that the reduce function should be reducing the values for a key down
into a single key,value pair, but Map/Reduce does not require that - as
shown in the example above, it's useful to be able to emit multiple
key/value pairs from the 'reduce', actually expanding the key/value set.
It is COMMON for the reduce function to emit a single value - it is not
necessary for Map/Reduce to be restricted in that way.
Dan.
Re: Erroneous output from the trunk?
Posted by Guby <se...@gmail.com>.
Thanks Benoit!
I got it to work now.
I am on r651236 (it reports r651244 now after i applied the patch!).
The couch.js file doesn't exist anymore, but if one ignores that and
only applied the path for the couch_httpd.erl file it works like a
charm!
Thanks
S
On Apr 24, 2008, at 8:37 AM, Benoit Chesneau wrote:
> On Wed, Apr 23, 2008 at 11:53 PM, Guby <gu...@gmail.com> wrote:
>> Read your ticket.
>> Having the exact same problem!
>> Unfortunately the patch doesn't work on the trunk version. A lot of
>> filenames have changed...
>>
>> Which revision are you running? I could just go back to that and
>> apply your
>> patch until the trunk has been fixed.
>>
>
> mm patch apply well here on latest trunk (r650977) and was reported to
> work on irc :)
>
> patch -p0 -i thepatch in top directory of couchdb sources You could
> also use these patches :
> http://babilu.metavers.net/couchdb/patches/patch-share_www_script_couch_js
> http://babilu.metavers.net/couchdb/patches/patch-src_couchdb_couch_httpd_erl
>
> Same patch but splitted in two for my openbsd port.
>
> - benoît
Re: Erroneous output from the trunk?
Posted by Guby <gu...@gmail.com>.
Thanks Benoit!
I got it to work now.
I am on r651236 (it reports r651244 now after i applied the patch!).
Thanks
S
On Apr 24, 2008, at 8:37 AM, Benoit Chesneau wrote:
> On Wed, Apr 23, 2008 at 11:53 PM, Guby <gu...@gmail.com> wrote:
>> Read your ticket.
>> Having the exact same problem!
>> Unfortunately the patch doesn't work on the trunk version. A lot of
>> filenames have changed...
>>
>> Which revision are you running? I could just go back to that and
>> apply your
>> patch until the trunk has been fixed.
>>
>
> mm patch apply well here on latest trunk (r650977) and was reported to
> work on irc :)
>
> patch -p0 -i thepatch in top directory of couchdb sources You could
> also use these patches :
> http://babilu.metavers.net/couchdb/patches/patch-share_www_script_couch_js
> http://babilu.metavers.net/couchdb/patches/patch-src_couchdb_couch_httpd_erl
>
> Same patch but splitted in two for my openbsd port.
>
> - benoît
Re: Erroneous output from the trunk?
Posted by Benoit Chesneau <bc...@gmail.com>.
On Wed, Apr 23, 2008 at 11:53 PM, Guby <gu...@gmail.com> wrote:
> Read your ticket.
> Having the exact same problem!
> Unfortunately the patch doesn't work on the trunk version. A lot of
> filenames have changed...
>
> Which revision are you running? I could just go back to that and apply your
> patch until the trunk has been fixed.
>
mm patch apply well here on latest trunk (r650977) and was reported to
work on irc :)
patch -p0 -i thepatch in top directory of couchdb sources You could
also use these patches :
http://babilu.metavers.net/couchdb/patches/patch-share_www_script_couch_js
http://babilu.metavers.net/couchdb/patches/patch-src_couchdb_couch_httpd_erl
Same patch but splitted in two for my openbsd port.
- benoît
Re: Erroneous output from the trunk?
Posted by Guby <gu...@gmail.com>.
Read your ticket.
Having the exact same problem!
Unfortunately the patch doesn't work on the trunk version. A lot of
filenames have changed...
Which revision are you running? I could just go back to that and apply
your patch until the trunk has been fixed.
Thanks!
S
On Apr 23, 2008, at 3:27 PM, Benoit Chesneau wrote:
> On Wed, Apr 23, 2008 at 8:22 PM, Guby <gu...@gmail.com> wrote:
>> I access it through nginx, but this worked perfectly until I updated
>> couchdb. The nginx config looks like this:
>>
>> upstream db {
>> server 127.0.0.1:5984;
>> }
>>
>> server {
>> listen 80;
>> server_name my_domain.com;
>> location / {
>> proxy_pass http://db;
>> auth_basic "Restricted area";
>> auth_basic_user_file /path_to_password_file/.htpasswd;
>> break;
>> }
>> }
>>
>> Here are the headers from Firefox:
>>
>> Response Headers
>> Transfer-Encoding chunked
>> Date Wed, 23 Apr 2008 16:00:36 GMT
>> Content-Type text/plain;charset=utf-8
>> Server nginx/0.5.26
>> Content-Encoding gzip
>>
>> Request Headers
>> Host db.kle.io
>> User-Agent Mozilla/5.0 (Macintosh; U; Intel Mac OS X; nb-NO;
>> rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14
>> Accept
>> text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/
>> plain;q=0.8,image/png,*/*;q=0.5
>> Accept-Language nb,no;q=0.8,nn;q=0.6,en-us;q=0.4,en;q=0.2
>> Accept-Encoding gzip,deflate
>> Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
>> Keep-Alive 300
>> Connection keep-alive
>> Authorization Basic c2ViYXN0aWFuOnNlYmJhODU=
>>
>>
>> Best regards
>> Sebastian
>>
>
>
> ok. So this is a known problem, nginx converse with couchdb in http
> 1.0 like most of proxies. So you get a non parsed http chunks.
>
> I made a patch for that (though non tested with latest trunk) :
> You can get it here :
> https://issues.apache.org/jira/browse/COUCHDB-40
>
> don't hesitate to comment this issue :)
>
> Benoît
>
>
> --
> - benoît
Re: Erroneous output from the trunk?
Posted by Benoit Chesneau <bc...@gmail.com>.
On Wed, Apr 23, 2008 at 8:22 PM, Guby <gu...@gmail.com> wrote:
> I access it through nginx, but this worked perfectly until I updated
> couchdb. The nginx config looks like this:
>
> upstream db {
> server 127.0.0.1:5984;
> }
>
> server {
> listen 80;
> server_name my_domain.com;
> location / {
> proxy_pass http://db;
> auth_basic "Restricted area";
> auth_basic_user_file /path_to_password_file/.htpasswd;
> break;
> }
> }
>
> Here are the headers from Firefox:
>
> Response Headers
> Transfer-Encoding chunked
> Date Wed, 23 Apr 2008 16:00:36 GMT
> Content-Type text/plain;charset=utf-8
> Server nginx/0.5.26
> Content-Encoding gzip
>
> Request Headers
> Host db.kle.io
> User-Agent Mozilla/5.0 (Macintosh; U; Intel Mac OS X; nb-NO;
> rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14
> Accept
> text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
> Accept-Language nb,no;q=0.8,nn;q=0.6,en-us;q=0.4,en;q=0.2
> Accept-Encoding gzip,deflate
> Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
> Keep-Alive 300
> Connection keep-alive
> Authorization Basic c2ViYXN0aWFuOnNlYmJhODU=
>
>
> Best regards
> Sebastian
>
ok. So this is a known problem, nginx converse with couchdb in http
1.0 like most of proxies. So you get a non parsed http chunks.
I made a patch for that (though non tested with latest trunk) :
You can get it here :
https://issues.apache.org/jira/browse/COUCHDB-40
don't hesitate to comment this issue :)
Benoît
--
- benoît
Re: Erroneous output from the trunk?
Posted by Guby <gu...@gmail.com>.
I access it through nginx, but this worked perfectly until I updated
couchdb. The nginx config looks like this:
upstream db {
server 127.0.0.1:5984;
}
server {
listen 80;
server_name my_domain.com;
location / {
proxy_pass http://db;
auth_basic "Restricted area";
auth_basic_user_file /path_to_password_file/.htpasswd;
break;
}
}
Here are the headers from Firefox:
Response Headers
Transfer-Encoding chunked
Date Wed, 23 Apr 2008 16:00:36 GMT
Content-Type text/plain;charset=utf-8
Server nginx/0.5.26
Content-Encoding gzip
Request Headers
Host db.kle.io
User-Agent Mozilla/5.0 (Macintosh; U; Intel Mac OS X; nb-NO; rv:
1.8.1.14) Gecko/20080404 Firefox/2.0.0.14
Accept text/xml,application/xml,application/xhtml+xml,text/
html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language nb,no;q=0.8,nn;q=0.6,en-us;q=0.4,en;q=0.2
Accept-Encoding gzip,deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 300
Connection keep-alive
Authorization Basic c2ViYXN0aWFuOnNlYmJhODU=
Best regards
Sebastian
On Apr 23, 2008, at 2:15 PM, Benoit Chesneau wrote:
> On Wed, Apr 23, 2008 at 6:10 PM, Guby <gu...@gmail.com> wrote:
>
>> It is when I access the couchdb server from my local machine using
>> safari
>> and firefox that I get the strange output. That never happened
>> before! I
>> haven't changed my nginx setup or config either.
>>
>
> do you access directly to couch or via nginx ? do you have a way to
> check header you get from the server response ? if yes , could you
> paste it here ?
>
>
> - benoît
Re: Erroneous output from the trunk?
Posted by Benoit Chesneau <bc...@gmail.com>.
On Wed, Apr 23, 2008 at 6:10 PM, Guby <gu...@gmail.com> wrote:
> It is when I access the couchdb server from my local machine using safari
> and firefox that I get the strange output. That never happened before! I
> haven't changed my nginx setup or config either.
>
do you access directly to couch or via nginx ? do you have a way to
check header you get from the server response ? if yes , could you
paste it here ?
- benoît
Re: Erroneous output from the trunk?
Posted by Guby <gu...@gmail.com>.
I am using bash.
In bash with curl it works fine when logged into the remote machine.
It is when I access the couchdb server from my local machine using
safari and firefox that I get the strange output. That never happened
before! I haven't changed my nginx setup or config either.
S
On Apr 23, 2008, at 1:08 PM, Benoit Chesneau wrote:
> On Wed, Apr 23, 2008 at 6:01 PM, Guby <gu...@gmail.com> wrote:
>> Thanks for the answers guys.
>> You are right.
>> But I don't understand why I get the raw HTTP/1.1 response chunk
>> all of a
>> sudden and not only the "data" like before?
>>
>> Doing a "curl" from my server works though!
>>
>>
>>
>
> what is the command line you use ?
>
> - benoît
Re: Erroneous output from the trunk?
Posted by Benoit Chesneau <bc...@gmail.com>.
On Wed, Apr 23, 2008 at 6:01 PM, Guby <gu...@gmail.com> wrote:
> Thanks for the answers guys.
> You are right.
> But I don't understand why I get the raw HTTP/1.1 response chunk all of a
> sudden and not only the "data" like before?
>
> Doing a "curl" from my server works though!
>
>
>
what is the command line you use ?
- benoît
Re: Erroneous output from the trunk?
Posted by Guby <gu...@gmail.com>.
Thanks for the answers guys.
You are right.
But I don't understand why I get the raw HTTP/1.1 response chunk all
of a sudden and not only the "data" like before?
Doing a "curl" from my server works though!
So I guess it is not CouchDBs fault really...
But what can it be?
and to benoit: I am not behind a proxy...
Best regards
Sebastian
On Apr 23, 2008, at 12:54 PM, Kristopher Tate wrote:
> I might be wrong, but I think what you're looking at is a HTTP/1.1
> response chunk.
>
> 2e == 46 in hexadecimal, the length of the chunk.
>
> The trailing 0 acts as a terminator for the chunk sequence.
>
> --
> kristopher
>
> On 2008/04/24, at 0:43, Guby wrote:
>>
>> I just updated my install to the trunk version on both my local
>> machine and on my server:
>> On my local machine everything works nice on the trunk, on the
>> server on the other hand I get some strange output from CouchDB...
>> 2e
>> {"couchdb":"Welcome","version":"0.7.3a650904"}
>> 0
>> This behavior also makes it impossible to interact with the
>> database using the _utils/ interface, because of the incorrect
>> output...
>>
>> Any ideas?
>>
>> Other commands and their output:
>>
>> _all_dbs:
>> 9
>> ["kleio"]
>> 0
>> /kleio/
>>
>> 6c
>> {"db_name":"kleio","doc_count":5,"doc_del_count":0,"update_seq":
>> 5,"compact_running":false,"disk_size":15721}
>> 0
>>
>> Best regards
>> Sebastian
>>
>>
>>
>
Re: Erroneous output from the trunk?
Posted by Kristopher Tate <kr...@bluebridge.jp>.
I might be wrong, but I think what you're looking at is a HTTP/1.1
response chunk.
2e == 46 in hexadecimal, the length of the chunk.
The trailing 0 acts as a terminator for the chunk sequence.
--
kristopher
On 2008/04/24, at 0:43, Guby wrote:
>
> I just updated my install to the trunk version on both my local
> machine and on my server:
> On my local machine everything works nice on the trunk, on the
> server on the other hand I get some strange output from CouchDB...
> 2e
> {"couchdb":"Welcome","version":"0.7.3a650904"}
> 0
> This behavior also makes it impossible to interact with the database
> using the _utils/ interface, because of the incorrect output...
>
> Any ideas?
>
> Other commands and their output:
>
> _all_dbs:
> 9
> ["kleio"]
> 0
> /kleio/
>
> 6c
> {"db_name":"kleio","doc_count":5,"doc_del_count":0,"update_seq":
> 5,"compact_running":false,"disk_size":15721}
> 0
>
> Best regards
> Sebastian
>
>
>
Re: Erroneous output from the trunk?
Posted by Guby <gu...@gmail.com>.
No, not that I know.
Same setup worked fine before I updated the couchdb on the server.
Best regards
Sebastian
On Apr 23, 2008, at 12:48 PM, Benoit Chesneau wrote:
> On Wed, Apr 23, 2008 at 5:43 PM, Guby <gu...@gmail.com> wrote:
>>
>> I just updated my install to the trunk version on both my local
>> machine and
>> on my server:
>> On my local machine everything works nice on the trunk, on the
>> server on
>> the other hand I get some strange output from CouchDB...
>> 2e
>> {"couchdb":"Welcome","version":"0.7.3a650904"}
>> 0
>> This behavior also makes it impossible to interact with the
>> database using
>> the _utils/ interface, because of the incorrect output...
>>
>> Any ideas?
>>
>> Other commands and their output:
>>
>> _all_dbs:
>> 9
>> ["kleio"]
>> 0
>> /kleio/
>>
>> 6c
>>
>> {"db_name":"kleio","doc_count":5,"doc_del_count":0,"update_seq":
>> 5,"compact_running":false,"disk_size":15721}
>> 0
>>
>> Best regards
>> Sebastian
>>
>>
>>
>>
>
> are you behind a proxy ? looks like you get unparsed http/1.1 chunk .
>
>
> - benoît
Re: Erroneous output from the trunk?
Posted by Benoit Chesneau <bc...@gmail.com>.
On Wed, Apr 23, 2008 at 5:43 PM, Guby <gu...@gmail.com> wrote:
>
> I just updated my install to the trunk version on both my local machine and
> on my server:
> On my local machine everything works nice on the trunk, on the server on
> the other hand I get some strange output from CouchDB...
> 2e
> {"couchdb":"Welcome","version":"0.7.3a650904"}
> 0
> This behavior also makes it impossible to interact with the database using
> the _utils/ interface, because of the incorrect output...
>
> Any ideas?
>
> Other commands and their output:
>
> _all_dbs:
> 9
> ["kleio"]
> 0
> /kleio/
>
> 6c
>
> {"db_name":"kleio","doc_count":5,"doc_del_count":0,"update_seq":5,"compact_running":false,"disk_size":15721}
> 0
>
> Best regards
> Sebastian
>
>
>
>
are you behind a proxy ? looks like you get unparsed http/1.1 chunk .
- benoît
Erroneous output from the trunk?
Posted by Guby <gu...@gmail.com>.
I just updated my install to the trunk version on both my local
machine and on my server:
On my local machine everything works nice on the trunk, on the server
on the other hand I get some strange output from CouchDB...
2e
{"couchdb":"Welcome","version":"0.7.3a650904"}
0
This behavior also makes it impossible to interact with the database
using the _utils/ interface, because of the incorrect output...
Any ideas?
Other commands and their output:
_all_dbs:
9
["kleio"]
0
/kleio/
6c
{"db_name":"kleio","doc_count":5,"doc_del_count":0,"update_seq":
5,"compact_running":false,"disk_size":15721}
0
Best regards
Sebastian