Scale function inefficiency perhaps?

[Erick Erickson <er...@gmail.com>]

Mostly, I'm checking to see if I'm hallucinating or not before going forward....

ScaleFloatFunction.getValues iterates through all the values for a
field in all the documents to find the min and max for a particular
field it's using. So far, so good. But is this really necessary every
time it gets called? I don't think these numbers change if the index
hasn't changed, right? Couldn't the min and max just be cached the
first time around?

If one were to work up a patch (both 3.x and 4.x I think), I assume
that reader.getVersion() and min and max could be cached the first
time around and the list iterated only when the version changes...

The only fly I see in the ointment is that the values would have to be
shared amongst multiple threads, what's the preferred way to
synchronize such a thing? It's been a long time since I've been in
Java synchronization methods....

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: Scale function inefficiency perhaps?

[Greg Bowyer <gb...@fastmail.co.uk>]

I am going to weigh in on this, this function turns up in my profiler 
dumps as being expensive (attached). My profiler output is from 
lucene_3_5, so exacting details might deviate.

With that said, whilst it looks bad, I dont think for the case of 
scaling a field its _that_ bad, essentially (as far as I can tell 
through debugging) it boils down to iterating items from the field 
cache, which is not the most expensive thing going. With that said, I 
was personally thinking if it could be stored in the index, if you are 
suggesting that its stored in the codecs with lucene4 that would be great.

In my case, I am wondering if it has a bad interaction with the grouping 
feature introduced into 3.5; I think that the specific DocValues 
instance it iterates over for groups might be something that is not as 
cheap to iterate as the fieldCache and as a result causes the expensive 
calls I see. I am currently looking at SOLR-3109, as I think the seen 
expense of the Scale Function might be making that problem far worse.

If people want I can get harder better profiling output or debugging here.

-- Greg

On 09/02/12 12:52, Chris Hostetter wrote:
> : can anyone confirm?
>
> everything you said sounded right to me ... scale(...) as a general
> purpose function probably can't be optimized very heaivly.  but it would
> probably be possible to write an optimization specific to
> scale(fieldname).
>
> I seem to recall someone saying something at some point about the idea of
> having the codec record stats like the min/max value of a field (per
> segment) ... assuming i'm not smoking crack, so you wouldn't have to do
> any caching, you could have an optimized "scalefield(f)" function that
> could fail hard if the field didn't use the expected codec type, otherwise
> it just asked all the atomic readers what the min/max where to find the
> overall min/max.
>
>
> -Hoss
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

Re: Scale function inefficiency perhaps?

[Chris Hostetter <ho...@fucit.org>]

: can anyone confirm?

everything you said sounded right to me ... scale(...) as a general 
purpose function probably can't be optimized very heaivly.  but it would 
probably be possible to write an optimization specific to 
scale(fieldname).

I seem to recall someone saying something at some point about the idea of 
having the codec record stats like the min/max value of a field (per 
segment) ... assuming i'm not smoking crack, so you wouldn't have to do 
any caching, you could have an optimized "scalefield(f)" function that 
could fail hard if the field didn't use the expected codec type, otherwise 
it just asked all the atomic readers what the min/max where to find the 
overall min/max.


-Hoss

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: Scale function inefficiency perhaps?

[Erick Erickson <er...@gmail.com>]

Siiiggggh, and I wanted it to be simple. This approach looks doomed;
nested functions apparently don't play nice in this scenario. The test
case that invokes log(val) fails, and digging in a little it looks like the
results of nested functions are in the DocValues array, which makes
the approach below nonsense unless we do some fancy dancing that
would be pretty limited in its utility.

Saved again by someone's diligence with the unit tests <G>.

can anyone confirm?

Thanks,
Erick

On Thu, Feb 9, 2012 at 12:42 PM, Erick Erickson <er...@gmail.com> wrote:
> Mostly, I'm checking to see if I'm hallucinating or not before going forward....
>
> ScaleFloatFunction.getValues iterates through all the values for a
> field in all the documents to find the min and max for a particular
> field it's using. So far, so good. But is this really necessary every
> time it gets called? I don't think these numbers change if the index
> hasn't changed, right? Couldn't the min and max just be cached the
> first time around?
>
> If one were to work up a patch (both 3.x and 4.x I think), I assume
> that reader.getVersion() and min and max could be cached the first
> time around and the list iterated only when the version changes...
>
> The only fly I see in the ointment is that the values would have to be
> shared amongst multiple threads, what's the preferred way to
> synchronize such a thing? It's been a long time since I've been in
> Java synchronization methods....

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org