Hitting reduce_limit with "good" reduce function

[Michael Jackson <mj...@gmail.com>]

Hi,

I'm currently getting the "Reduce output must shrink more rapidly" error
with what I believe is a "good" reduce function. When I checked the
default.ini config file to tweak the reduce_limit setting, it told me to
ping the mailing list "so we can fine tune the heuristic", so here I am. :)

Without going too deep into the details of my app, I'm basically using my
reduce function to reduce a set of timestamps to the most recent one, and
then appending that value to another doc that I also pass in to the reduce
function. This is probably tripping the reduce_limit guard because the
result of the reduce is essentially the original document plus one extra
property. The thing is, I know that it will only ever add one extra
property, hence I know the reduce function is "good".

My use case is that I've got a bunch of user documents and a bunch of
timestamps of when we last saw that user. When I fetch the user doc, I'd
like to also have present the timestamp when that user was last seen. I may
just be greedy wanting to do this all in one query, but I'll be running
this query very frequently so I'd like to optimize this case.

--
Michael Jackson
@mjackson

Re: Hitting reduce_limit with "good" reduce function

[Michael Jackson <mj...@gmail.com>]

On Thu, Apr 11, 2013 at 11:21 AM, Robert Newson <rn...@apache.org> wrote:

> That's not "good" enough for a reduce. A reduce function has to return
> a value *smaller* than the input values, and preferably a lot smaller.
> Returning something as large as a full document is among the set of
> things reduce_limit is trying to discourage.
>

In most cases I anticipate that the reduce value in this case *will* end up
being a lot smaller than the inputs. For example, after I've seen a user a
thousand times all those values will reduce to just one timestamp - the
most recent one - on the user document.


> For this problem, you should emit your timestamps in the map phase and
> use endkey and limit to find the latest. It's not a reduce problem.
>

Thanks for the suggestion. I can totally accept that this isn't a problem
with reduce.

However, I'd still like to do it. :D The main reason is that I'd like to be
able to fetch many users at a time, along with information about when they
were last seen. Using the method I've described I can query the reduce view
with many keys and group=true.

I may not be understanding correctly but emitting just the timestamps in
the map would require me to make two queries: one for the timestamp and one
for the user document.

For reference, here's my map/reduce:

function map(doc) {
  // A profile document is a user.
  if (doc.type === 'profile') {
    emit(doc._id, doc);
  } else if (doc.type === 'presence' && doc.profile_id) {
    emit(doc.profile_id, doc);
  }
}

function reduce(key, values, rereduce) {
  var profile, timestamp;

  var value;
  for (var i = 0, len = values.length; i < len; ++i) {
    value = values[i];

    // At least one of the values is a profile document. In a
    // rereduce we could see many of these, but we don't care
    // because they are the same document.
    if (value.type === 'profile') {
      profile = value;
    }

    // Find the most recent timestamp. In a rereduce the value
    // may be a profile document with a timestamp property.
    if (value.timestamp && (!timestamp || timestamp < value.timestamp)) {
      timestamp = value.timestamp;
    }
  }

  if (profile && timestamp) {
    profile.timestamp = timestamp;
  }

  return profile;
}

I realize it's a bit messy, but it does save me another trip to the
database and it is easily one of the most common queries I'll be doing, so
it's an optimization that seems worthwhile at this point. If there's a
cleaner/better way to get all the same data in one trip to the database,
I'd love to hear it.

--
Michael Jackson
@mjackson

Re: Hitting reduce_limit with "good" reduce function

[Robert Newson <rn...@apache.org>]

That's not "good" enough for a reduce. A reduce function has to return
a value *smaller* than the input values, and preferably a lot smaller.
Returning something as large as a full document is among the set of
things reduce_limit is trying to discourage.

For this problem, you should emit your timestamps in the map phase and
use endkey and limit to find the latest. It's not a reduce problem.

B.

On 11 April 2013 14:06, Michael Jackson <mj...@gmail.com> wrote:
> Hi,
>
> I'm currently getting the "Reduce output must shrink more rapidly" error
> with what I believe is a "good" reduce function. When I checked the
> default.ini config file to tweak the reduce_limit setting, it told me to
> ping the mailing list "so we can fine tune the heuristic", so here I am. :)
>
> Without going too deep into the details of my app, I'm basically using my
> reduce function to reduce a set of timestamps to the most recent one, and
> then appending that value to another doc that I also pass in to the reduce
> function. This is probably tripping the reduce_limit guard because the
> result of the reduce is essentially the original document plus one extra
> property. The thing is, I know that it will only ever add one extra
> property, hence I know the reduce function is "good".
>
> My use case is that I've got a bunch of user documents and a bunch of
> timestamps of when we last saw that user. When I fetch the user doc, I'd
> like to also have present the timestamp when that user was last seen. I may
> just be greedy wanting to do this all in one query, but I'll be running
> this query very frequently so I'd like to optimize this case.
>
> --
> Michael Jackson
> @mjackson

Re: Hitting reduce_limit with "good" reduce function

[Matthieu Rakotojaona <ma...@gmail.com>]

On 04/11, Michael Jackson wrote:
> My use case is that I've got a bunch of user documents and a bunch of
> timestamps of when we last saw that user. When I fetch the user doc, I'd
> like to also have present the timestamp when that user was last seen. I may
> just be greedy wanting to do this all in one query, but I'll be running
> this query very frequently so I'd like to optimize this case.

According to your use case, you could return the user doc and the
timestamp as two separated fields but in the same response, instead of
building a brand new doc on each rereduce step. The doc will be
transported as-is from rereduce to rereduce, only the last_seen argument
will be changed until all the timestamps are processed.

-- 
Matthieu Rakotojaona