Intermittent search performance problem

[Ben Dotte <be...@gmail.com>]

Hi,

I'm trying to figure out a way to troubleshoot a performance problem
we're seeing when searching against a memory-based index. What happens
is we will run a search against the index and it generally returns in
1 second or less. But every once in a while it takes 15-20 seconds for
the exact same search for no apparent reason. There is nothing else
going on in the system to cause this behavior.

I have tried hooking up YourKit profiler to see where the time is
going but it doesn't even record the extra time being taken up, even
when I ask for method invocation counts.

This is very strange, we have been using Lucene for years in
production and I've never seen a problem like it. It is also only
exhibited in one particular index, we cannot reproduce the problem
with other indexes. This index has around 170,000 documents in it and
does not have a particularly large amount of data relative to our
other indexes.

I would really appreciate any suggestions for tracking down the
culprit. Since YourKit is missing the extra time it seems like some
sort of lock/synchronized method issue but I've only really seen that
type of problem using disk indexing when the indexes aren't optimized.
We're currently on Lucene 2.0 but I had the same problem with 1.9.1.

Thanks,
Ben

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Intermittent search performance problem

[Yonik Seeley <yo...@apache.org>]

On 11/3/06, Ben Dotte <be...@gmail.com> wrote:
> I'm trying to figure out a way to troubleshoot a performance problem
> we're seeing when searching against a memory-based index. What happens
> is we will run a search against the index and it generally returns in
> 1 second or less. But every once in a while it takes 15-20 seconds for
> the exact same search for no apparent reason.

Are you sure it's not just a big GC?  How big is your heap?

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org