You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Luca Morandini <lm...@ieee.org> on 2014/09/30 11:05:58 UTC
"keys is null" message when a view is performed on many docs
Folks,
I wrote a view to join docs on a common attribute, making the reduce part store
docs with the same key for both sides of the join.
This works for a few docs, but when I try to run the view on, say 100K docs,
CouchDB's logs keep on saying:
OS Process #Port<0.3458> Log :: function raised exception (new TypeError("keys is
null", "undefined", 5))
The view is as such:
The Map part:
function (doc) {
if (doc.joinside) {
emit([ doc.joinkey, doc.joinside ], doc.feature);
}
}
The Reduce part:
function (keys, values, rereduce) {
var lefts = [];
var rights = [];
for (var i = 0; i < keys.length; i++) {
if (keys[i][0][1] == "left") {
lefts.push(values[i]);
} else {
rights.push(values[i]);
}
}
return {
lefts : lefts,
rights : rights
};
}
...as you may have guessed, I had to set the reduce_limit to false (it is supposed
to run only at group_level=1..
At the moment, the view runs on an Ubuntu 14.04, CouchDB 1.5.0.
View's output with a few dozens docs looks fine:
lefts: [
{
type: "Feature",
properties: {
boundedBy: [
147.2908,
-32.1711,
147.2908,
-32.1711
],
timeName: "2011",
geometryID: "1",
numberOfMeters: null,
numberOfReadings: null
},
id: "CombinedMeterReadings.fid--3d7017ee_14733c15f9f_-2c90"
},
{
type: "Feature",
properties: {
boundedBy: [
147.2908,
-32.1711,
147.2908,
-32.1711
],
timeName: "2011",
geometryID: "1",
numberOfMeters: null,
numberOfReadings: null
},
id: "CombinedMeterReadings.fid--3d7017ee_14733c15f9f_-2c89"
},
{
type: "Feature",
properties: {
boundedBy: [
147.2908,
-32.1711,
147.2908,
-32.1711
],
timeName: "2011",
geometryID: "1",
numberOfMeters: null,
numberOfReadings: null
},
id: "CombinedMeterReadings.fid--3d7017ee_14733c15f9f_-2c82"
}
],
rights: [
{
type: "Feature",
geometry: null,
properties: {
geographicid: "1",
geographicname: "New South Wales",
property: "Unit",
chronological: "ThreeMonth",
daterange: "2014-01-01 to 2014-03-31",
autscnt: 1284
}
}
]
}
Regards,
Luca Morandini
Data Architect - AURIN project
Melbourne eResearch Group
Department of Computing and Information Systems
University of Melbourne
Tel. +61 03 903 58 380
Skype: lmorandini
Re: "keys is null" message when a view is performed on many docs
Posted by Luca Morandini <lm...@ieee.org>.
On 30/09/14 19:09, Mike Marino wrote:
> Hi Luca,
>
> You have to handle the case when rereduce is true (and keys will then be
> null). See e.g.
Duh ! I should have seen it.
Hmm... it worked fine with a few dozens docs though.... ahh! Now I understand:
with small data there were no re-reduce, since every set of values with the same
keys fit into a single "reduce" execution, not the case with sizeable data.
Thanks Mike for letting me see the light ;)
Luca Morandini
Data Architect - AURIN project
Melbourne eResearch Group
Department of Computing and Information Systems
University of Melbourne
Tel. +61 03 903 58 380
Skype: lmorandini
Re: "keys is null" message when a view is performed on many docs
Posted by Mike Marino <mm...@gmail.com>.
Hi Luca,
You have to handle the case when rereduce is true (and keys will then be
null). See e.g.
http://wiki.apache.org/couchdb/Introduction_to_CouchDB_views#Reduce_Functions
Cheers,
Mike
On Tue, Sep 30, 2014 at 11:05 AM, Luca Morandini <lm...@ieee.org>
wrote:
> Folks,
>
> I wrote a view to join docs on a common attribute, making the reduce part
> store docs with the same key for both sides of the join.
>
> This works for a few docs, but when I try to run the view on, say 100K
> docs, CouchDB's logs keep on saying:
> OS Process #Port<0.3458> Log :: function raised exception (new
> TypeError("keys is null", "undefined", 5))
>
> The view is as such:
>
> The Map part:
> function (doc) {
> if (doc.joinside) {
> emit([ doc.joinkey, doc.joinside ], doc.feature);
> }
> }
>
> The Reduce part:
> function (keys, values, rereduce) {
> var lefts = [];
> var rights = [];
>
> for (var i = 0; i < keys.length; i++) {
> if (keys[i][0][1] == "left") {
> lefts.push(values[i]);
> } else {
> rights.push(values[i]);
> }
> }
>
> return {
> lefts : lefts,
> rights : rights
> };
> }
>
> ...as you may have guessed, I had to set the reduce_limit to false (it is
> supposed to run only at group_level=1..
> At the moment, the view runs on an Ubuntu 14.04, CouchDB 1.5.0.
>
> View's output with a few dozens docs looks fine:
> lefts: [
> {
> type: "Feature",
> properties: {
> boundedBy: [
> 147.2908,
> -32.1711,
> 147.2908,
> -32.1711
> ],
> timeName: "2011",
> geometryID: "1",
> numberOfMeters: null,
> numberOfReadings: null
> },
> id: "CombinedMeterReadings.fid--3d7017ee_14733c15f9f_-2c90"
> },
> {
> type: "Feature",
> properties: {
> boundedBy: [
> 147.2908,
> -32.1711,
> 147.2908,
> -32.1711
> ],
> timeName: "2011",
> geometryID: "1",
> numberOfMeters: null,
> numberOfReadings: null
> },
> id: "CombinedMeterReadings.fid--3d7017ee_14733c15f9f_-2c89"
> },
> {
> type: "Feature",
> properties: {
> boundedBy: [
> 147.2908,
> -32.1711,
> 147.2908,
> -32.1711
> ],
> timeName: "2011",
> geometryID: "1",
> numberOfMeters: null,
> numberOfReadings: null
> },
> id: "CombinedMeterReadings.fid--3d7017ee_14733c15f9f_-2c82"
> }
> ],
> rights: [
> {
> type: "Feature",
> geometry: null,
> properties: {
> geographicid: "1",
> geographicname: "New South Wales",
> property: "Unit",
> chronological: "ThreeMonth",
> daterange: "2014-01-01 to 2014-03-31",
> autscnt: 1284
> }
> }
> ]
> }
>
> Regards,
>
> Luca Morandini
> Data Architect - AURIN project
> Melbourne eResearch Group
> Department of Computing and Information Systems
> University of Melbourne
> Tel. +61 03 903 58 380
> Skype: lmorandini
>
>