You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by svilen <az...@svilendobrev.com> on 2012/09/22 12:05:19 UTC
what's the order of docs/keys going into reduce func?
say i have some lamp status monitoring records, i.e.
{ "seq": 1, "name": "a", },
{ "seq": 2, "name": "b", },
{ "seq": 2, "name": "a", "on": true},
{ "seq": 3, "name": "a", },
{ "seq": 7, "name": "a", "on": true},
...
where default state is off, and when on, there's extra bool attribute
for it. Records are sequenced (in time) by some .seq param. And there's
many other here-irrelevant attributes.
say i want a view for all lamps that the currently off.
so, something like:
map_func: function( doc) {
if (doc.type == 'lamp') emit( [doc.name, doc.seq], doc );
}
reduce_func: function( keys, values, rereduce) {
var v = values[ values.length-1];
if (v == null || v.on) return null;
return v;
}
always used with group_level= 1
but i get the very first state instead of very last state...
and in the debug-log, i see that the keys passed to reduce are reversed!
so i have to use descending=True, and only then it works correctly.
the question is, is this expected, or is the default order defined
somewhere else, or what?
ciao
svil
Re: what's the order of docs/keys going into reduce func?
Posted by svilen <az...@svilendobrev.com>.
> > here's something about it
> > http://osdir.com/ml/couchdb-user/2009-05/msg00149.html
>
> I can't access it :(
search for title "Ordering of keys into reduce function", maybe look in
google's cache.
> > function( keys, values, rereduce) {
> > var mx = 0, vmx = null;
> > for (var k in values) {
> > var v = values[k];
> > if (v.seq > mx) { mx= v.seq; vmx = v; }
> > }
> > return vmx;
> > }
> > and since it's not relying on seq order anymore, map func can
> > just emit( doc.name, doc).
>
> Emitting (doc.name, doc.seq) would be enough here.
> Unless you know what you do, you shouldn't emit a full doc; it's a
> waste of space since it is always available when you query with
> &include_docs=true
yeah but that works for non-reduce views only:
couchdb.http.ServerError: (400, ('query_parse_error', 'Query parameter
`include_docs` is invalid for reduce views.'))
svil
Re: what's the order of docs/keys going into reduce func?
Posted by Matthieu Rakotojaona <ma...@gmail.com>.
On Sat, Sep 22, 2012 at 11:18 PM, svilen <az...@svilendobrev.com> wrote:
> well yeah seems one can't rely on key ordering.
> here's something about it
> http://osdir.com/ml/couchdb-user/2009-05/msg00149.html
I can't access it :(
> function( keys, values, rereduce) {
> var mx = 0, vmx = null;
> for (var k in values) {
> var v = values[k];
> if (v.seq > mx) { mx= v.seq; vmx = v; }
> }
> return vmx;
> }
> and since it's not relying on seq order anymore, map func can
> just emit( doc.name, doc).
Emitting (doc.name, doc.seq) would be enough here.
Unless you know what you do, you shouldn't emit a full doc; it's a
waste of space since it is always available when you query with
&include_docs=true
--
Matthieu RAKOTOJAONA
Re: what's the order of docs/keys going into reduce func?
Posted by svilen <az...@svilendobrev.com>.
well yeah seems one can't rely on key ordering.
here's something about it
http://osdir.com/ml/couchdb-user/2009-05/msg00149.html
thanks for proposal, it is close to what i have in python as
postprocessing - essentialy rebuilding the state.
here's variant of what i ended up:
by using group_level=1, i can avoid having multiple names in a reduce
- all values are for same name. Then it's all about yielding the value
with max .seq number:
function( keys, values, rereduce) {
var mx = 0, vmx = null;
for (var k in values) {
var v = values[k];
if (v.seq > mx) { mx= v.seq; vmx = v; }
}
return vmx;
}
and since it's not relying on seq order anymore, map func can
just emit( doc.name, doc).
ok, for now i'll avoid returning null although it seems fine as long the
re/reduce func can handle them as inputs - which would also need the max
seq returned along the null, for later comparison.
svil
On Sat, 22 Sep 2012 22:28:09 +0200
Matthieu Rakotojaona <ma...@gmail.com> wrote:
> I think you shouldn't rely on the order in which your keys and values
> are passed to your reduce function, because there is no way to predict
> it. Plus, the query might run any number of rereduce steps.
>
> I propose the following :
>
> map: func(doc){
> if (doc.type == 'lamp'){
> if (typeof doc.on == "undefined") {
> emit([doc.name, doc.seq],null);
> }
> }
> }
>
> fun(key, value, rereduce) {
>
> last_state_hash = {}
>
> // last_state_hash will look like {"a": 7, "b": 3, "c":12}
>
> if (rereduce == false) {
>
> // key is an array that looks like
> // [
> // [["a",1],<someid1>],
> // [["a",2],<someid2>],
> // [["b",7],<someid3>]
> // ]
> //
> // value is an array of null (this is what we emitted)
>
> for (var i = 0; i < key.length; i++){
> name = key[i][0][0];
> seq = key[i][0][1];
>
> if (typeof(last_state_hash[name]) == 'undefined' ||
> last_state_hash[name] < seq) {
> last_state_hash[name] = seq
> }
> }
>
> } else {
>
> // key is null for a rereduce
> // value is an array of state hashes as defined before
>
> for (var i = 0; i < value.length; i++) {
>
> current_state_hash = value[i];
>
> for (var name in current_state_hash){
> if (current_state_hash.hasOwnProperty(name)){
>
> if (typeof(last_state_hash[name]) == 'undefined' ||
> last_state_hash[name] < current_state_hash[name]) {
> last_state_hash[name] = current_state_hash[name]
> }
>
> }
> }
>
> }
> }
>
> return last_state_hash;
>
> }
>
> Beware of the comparison !
>
> --
> Matthieu RAKOTOJAONA
Re: what's the order of docs/keys going into reduce func?
Posted by Matthieu Rakotojaona <ma...@gmail.com>.
I think you shouldn't rely on the order in which your keys and values
are passed to your reduce function, because there is no way to predict
it. Plus, the query might run any number of rereduce steps.
I propose the following :
map: func(doc){
if (doc.type == 'lamp'){
if (typeof doc.on == "undefined") {
emit([doc.name, doc.seq],null);
}
}
}
fun(key, value, rereduce) {
last_state_hash = {}
// last_state_hash will look like {"a": 7, "b": 3, "c":12}
if (rereduce == false) {
// key is an array that looks like
// [
// [["a",1],<someid1>],
// [["a",2],<someid2>],
// [["b",7],<someid3>]
// ]
//
// value is an array of null (this is what we emitted)
for (var i = 0; i < key.length; i++){
name = key[i][0][0];
seq = key[i][0][1];
if (typeof(last_state_hash[name]) == 'undefined' ||
last_state_hash[name] < seq) {
last_state_hash[name] = seq
}
}
} else {
// key is null for a rereduce
// value is an array of state hashes as defined before
for (var i = 0; i < value.length; i++) {
current_state_hash = value[i];
for (var name in current_state_hash){
if (current_state_hash.hasOwnProperty(name)){
if (typeof(last_state_hash[name]) == 'undefined' ||
last_state_hash[name] < current_state_hash[name]) {
last_state_hash[name] = current_state_hash[name]
}
}
}
}
}
return last_state_hash;
}
Beware of the comparison !
--
Matthieu RAKOTOJAONA
Re: what's the order of docs/keys going into reduce func?
Posted by Octavian Damiean <ma...@gmail.com>.
I don't understand a thing, sorry. You might have to rephrase it.
On Sat, Sep 22, 2012 at 9:23 PM, svilen <az...@svilendobrev.com> wrote:
> > > no, yours is just a simple filter, and it will always have them all
> > > as they all start from being off.
> >
> > I am having trouble understanding this. What do you mean by "simple
> > filter"? A view is a view, not a filter. Once the view is set up it
> > will always be up-to-date.
>
> take 2 steps back and reread my question from start. the important
> notions are "multiple values" with "same-keys" and their "current
> state-per-key". that is what i want.
>
> svil
>
> > On Sat, Sep 22, 2012 at 11:57 AM, svilen <az...@svilendobrev.com> wrote:
> >
> > > no, yours is just a simple filter, and it will always have them all
> > > as they all start from being off.
> > >
> > > and it is somewhat simplified example. The original thing has
> > > multiple states, and i want the last/current one - whatever it is.
> > >
> > > say, i want to reduce things like assigning ( a=1, a=2, b=1, b=2,
> > > a=3 ) into flat last-state ( a=3, b=2 )
> > >
> > > how to do that?
> > >
> > > svil
> > >
> > >
> > > On Sat, 22 Sep 2012 11:43:14 -0700
> > > Mark Hahn <ma...@hahnca.com> wrote:
> > >
> > > > > i want a view for all lamps that the currently off.
> > > >
> > > > You view is wrong. You should only emit when the lamp is off.
> > > >
> > > > Also, you should always return a value in reduce. Returning null
> > > > will screw things up.
> > > >
> > > > I would start by fixing these problems and then go from there.
> > > >
> > > >
> > > > On Sat, Sep 22, 2012 at 3:05 AM, svilen <az...@svilendobrev.com>
> > > > wrote:
> > > >
> > > > > say i have some lamp status monitoring records, i.e.
> > > > >
> > > > > { "seq": 1, "name": "a", },
> > > > > { "seq": 2, "name": "b", },
> > > > > { "seq": 2, "name": "a", "on": true},
> > > > > { "seq": 3, "name": "a", },
> > > > > { "seq": 7, "name": "a", "on": true},
> > > > >
> > > > > ...
> > > > >
> > > > > where default state is off, and when on, there's extra bool
> > > > > attribute for it. Records are sequenced (in time) by some .seq
> > > > > param. And there's many other here-irrelevant attributes.
> > > > >
> > > > > say i want a view for all lamps that the currently off.
> > > > >
> > > > > so, something like:
> > > > >
> > > > > map_func: function( doc) {
> > > > > if (doc.type == 'lamp') emit( [doc.name, doc.seq], doc );
> > > > > }
> > > > >
> > > > > reduce_func: function( keys, values, rereduce) {
> > > > > var v = values[ values.length-1];
> > > > > if (v == null || v.on) return null;
> > > > > return v;
> > > > > }
> > > > >
> > > > > always used with group_level= 1
> > > > >
> > > > > but i get the very first state instead of very last state...
> > > > >
> > > > > and in the debug-log, i see that the keys passed to reduce are
> > > > > reversed!
> > > > >
> > > > > so i have to use descending=True, and only then it works
> > > > > correctly.
> > > > >
> > > > > the question is, is this expected, or is the default order
> > > > > defined somewhere else, or what?
> > > > >
> > > > > ciao
> > > > > svil
> > > > >
> > >
>
Re: what's the order of docs/keys going into reduce func?
Posted by svilen <az...@svilendobrev.com>.
> > no, yours is just a simple filter, and it will always have them all
> > as they all start from being off.
>
> I am having trouble understanding this. What do you mean by "simple
> filter"? A view is a view, not a filter. Once the view is set up it
> will always be up-to-date.
take 2 steps back and reread my question from start. the important
notions are "multiple values" with "same-keys" and their "current
state-per-key". that is what i want.
svil
> On Sat, Sep 22, 2012 at 11:57 AM, svilen <az...@svilendobrev.com> wrote:
>
> > no, yours is just a simple filter, and it will always have them all
> > as they all start from being off.
> >
> > and it is somewhat simplified example. The original thing has
> > multiple states, and i want the last/current one - whatever it is.
> >
> > say, i want to reduce things like assigning ( a=1, a=2, b=1, b=2,
> > a=3 ) into flat last-state ( a=3, b=2 )
> >
> > how to do that?
> >
> > svil
> >
> >
> > On Sat, 22 Sep 2012 11:43:14 -0700
> > Mark Hahn <ma...@hahnca.com> wrote:
> >
> > > > i want a view for all lamps that the currently off.
> > >
> > > You view is wrong. You should only emit when the lamp is off.
> > >
> > > Also, you should always return a value in reduce. Returning null
> > > will screw things up.
> > >
> > > I would start by fixing these problems and then go from there.
> > >
> > >
> > > On Sat, Sep 22, 2012 at 3:05 AM, svilen <az...@svilendobrev.com>
> > > wrote:
> > >
> > > > say i have some lamp status monitoring records, i.e.
> > > >
> > > > { "seq": 1, "name": "a", },
> > > > { "seq": 2, "name": "b", },
> > > > { "seq": 2, "name": "a", "on": true},
> > > > { "seq": 3, "name": "a", },
> > > > { "seq": 7, "name": "a", "on": true},
> > > >
> > > > ...
> > > >
> > > > where default state is off, and when on, there's extra bool
> > > > attribute for it. Records are sequenced (in time) by some .seq
> > > > param. And there's many other here-irrelevant attributes.
> > > >
> > > > say i want a view for all lamps that the currently off.
> > > >
> > > > so, something like:
> > > >
> > > > map_func: function( doc) {
> > > > if (doc.type == 'lamp') emit( [doc.name, doc.seq], doc );
> > > > }
> > > >
> > > > reduce_func: function( keys, values, rereduce) {
> > > > var v = values[ values.length-1];
> > > > if (v == null || v.on) return null;
> > > > return v;
> > > > }
> > > >
> > > > always used with group_level= 1
> > > >
> > > > but i get the very first state instead of very last state...
> > > >
> > > > and in the debug-log, i see that the keys passed to reduce are
> > > > reversed!
> > > >
> > > > so i have to use descending=True, and only then it works
> > > > correctly.
> > > >
> > > > the question is, is this expected, or is the default order
> > > > defined somewhere else, or what?
> > > >
> > > > ciao
> > > > svil
> > > >
> >
Re: what's the order of docs/keys going into reduce func?
Posted by Mark Hahn <ma...@hahnca.com>.
> no, yours is just a simple filter, and it will always have them all as
they all start from being off.
I am having trouble understanding this. What do you mean by "simple
filter"? A view is a view, not a filter. Once the view is set up it will
always be up-to-date.
On Sat, Sep 22, 2012 at 11:57 AM, svilen <az...@svilendobrev.com> wrote:
> no, yours is just a simple filter, and it will always have them all
> as they all start from being off.
>
> and it is somewhat simplified example. The original thing has
> multiple states, and i want the last/current one - whatever it is.
>
> say, i want to reduce things like assigning ( a=1, a=2, b=1, b=2, a=3 )
> into flat last-state ( a=3, b=2 )
>
> how to do that?
>
> svil
>
>
> On Sat, 22 Sep 2012 11:43:14 -0700
> Mark Hahn <ma...@hahnca.com> wrote:
>
> > > i want a view for all lamps that the currently off.
> >
> > You view is wrong. You should only emit when the lamp is off.
> >
> > Also, you should always return a value in reduce. Returning null will
> > screw things up.
> >
> > I would start by fixing these problems and then go from there.
> >
> >
> > On Sat, Sep 22, 2012 at 3:05 AM, svilen <az...@svilendobrev.com> wrote:
> >
> > > say i have some lamp status monitoring records, i.e.
> > >
> > > { "seq": 1, "name": "a", },
> > > { "seq": 2, "name": "b", },
> > > { "seq": 2, "name": "a", "on": true},
> > > { "seq": 3, "name": "a", },
> > > { "seq": 7, "name": "a", "on": true},
> > >
> > > ...
> > >
> > > where default state is off, and when on, there's extra bool
> > > attribute for it. Records are sequenced (in time) by some .seq
> > > param. And there's many other here-irrelevant attributes.
> > >
> > > say i want a view for all lamps that the currently off.
> > >
> > > so, something like:
> > >
> > > map_func: function( doc) {
> > > if (doc.type == 'lamp') emit( [doc.name, doc.seq], doc );
> > > }
> > >
> > > reduce_func: function( keys, values, rereduce) {
> > > var v = values[ values.length-1];
> > > if (v == null || v.on) return null;
> > > return v;
> > > }
> > >
> > > always used with group_level= 1
> > >
> > > but i get the very first state instead of very last state...
> > >
> > > and in the debug-log, i see that the keys passed to reduce are
> > > reversed!
> > >
> > > so i have to use descending=True, and only then it works correctly.
> > >
> > > the question is, is this expected, or is the default order defined
> > > somewhere else, or what?
> > >
> > > ciao
> > > svil
> > >
>
Re: what's the order of docs/keys going into reduce func?
Posted by svilen <az...@svilendobrev.com>.
no, yours is just a simple filter, and it will always have them all
as they all start from being off.
and it is somewhat simplified example. The original thing has
multiple states, and i want the last/current one - whatever it is.
say, i want to reduce things like assigning ( a=1, a=2, b=1, b=2, a=3 )
into flat last-state ( a=3, b=2 )
how to do that?
svil
On Sat, 22 Sep 2012 11:43:14 -0700
Mark Hahn <ma...@hahnca.com> wrote:
> > i want a view for all lamps that the currently off.
>
> You view is wrong. You should only emit when the lamp is off.
>
> Also, you should always return a value in reduce. Returning null will
> screw things up.
>
> I would start by fixing these problems and then go from there.
>
>
> On Sat, Sep 22, 2012 at 3:05 AM, svilen <az...@svilendobrev.com> wrote:
>
> > say i have some lamp status monitoring records, i.e.
> >
> > { "seq": 1, "name": "a", },
> > { "seq": 2, "name": "b", },
> > { "seq": 2, "name": "a", "on": true},
> > { "seq": 3, "name": "a", },
> > { "seq": 7, "name": "a", "on": true},
> >
> > ...
> >
> > where default state is off, and when on, there's extra bool
> > attribute for it. Records are sequenced (in time) by some .seq
> > param. And there's many other here-irrelevant attributes.
> >
> > say i want a view for all lamps that the currently off.
> >
> > so, something like:
> >
> > map_func: function( doc) {
> > if (doc.type == 'lamp') emit( [doc.name, doc.seq], doc );
> > }
> >
> > reduce_func: function( keys, values, rereduce) {
> > var v = values[ values.length-1];
> > if (v == null || v.on) return null;
> > return v;
> > }
> >
> > always used with group_level= 1
> >
> > but i get the very first state instead of very last state...
> >
> > and in the debug-log, i see that the keys passed to reduce are
> > reversed!
> >
> > so i have to use descending=True, and only then it works correctly.
> >
> > the question is, is this expected, or is the default order defined
> > somewhere else, or what?
> >
> > ciao
> > svil
> >
Re: what's the order of docs/keys going into reduce func?
Posted by Mark Hahn <ma...@hahnca.com>.
> i want a view for all lamps that the currently off.
You view is wrong. You should only emit when the lamp is off.
Also, you should always return a value in reduce. Returning null will
screw things up.
I would start by fixing these problems and then go from there.
On Sat, Sep 22, 2012 at 3:05 AM, svilen <az...@svilendobrev.com> wrote:
> say i have some lamp status monitoring records, i.e.
>
> { "seq": 1, "name": "a", },
> { "seq": 2, "name": "b", },
> { "seq": 2, "name": "a", "on": true},
> { "seq": 3, "name": "a", },
> { "seq": 7, "name": "a", "on": true},
>
> ...
>
> where default state is off, and when on, there's extra bool attribute
> for it. Records are sequenced (in time) by some .seq param. And there's
> many other here-irrelevant attributes.
>
> say i want a view for all lamps that the currently off.
>
> so, something like:
>
> map_func: function( doc) {
> if (doc.type == 'lamp') emit( [doc.name, doc.seq], doc );
> }
>
> reduce_func: function( keys, values, rereduce) {
> var v = values[ values.length-1];
> if (v == null || v.on) return null;
> return v;
> }
>
> always used with group_level= 1
>
> but i get the very first state instead of very last state...
>
> and in the debug-log, i see that the keys passed to reduce are reversed!
>
> so i have to use descending=True, and only then it works correctly.
>
> the question is, is this expected, or is the default order defined
> somewhere else, or what?
>
> ciao
> svil
>