You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Tim Johnson <ti...@saic.com> on 2007/02/21 15:34:36 UTC
Stop long running queries
I'm having issues with some queries taking in excess of 500 secs to run to
completion. The system being used consists of ~100 million docs spilt up
across ~600 indexes. The indexes are of various sizes from 15MB to 8GB and
all searches done in the system require an exact count of matching hits.
The documents are also indexed with various fields and types ranging from 15
to 200 fields depending on the data indexed.
98% of all queries run under 4 sec, but the remaining 2% run for > 15 secs
and sometimes > 500 secs. These long running queries usually consist of
wildcard and range queries where the number of BooleanQuery statements
reaches into the 100,000 range. Some of these queries return a relatively
small number of results (below 300).
I've recently tried to kill the search using a separate thread to wrap the
searcher.search method, which works, but only once. The next query to come
through throws an "IOException: The handle is invalid." in the
FSInputStream.readInternal() method.
Any suggestions on how to stop these long running queries?
Tim Johnson
SAIC
Systems Engineer
757.226.1569
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Stop long running queries
Posted by Chris Hostetter <ho...@fucit.org>.
optimizing away the expensive cases is your best bet if you can do it ...
another option is to use a custom HitCollector which keeps track of how
long it's been running and throws a subclass of RUntimeException which you
explicitly catch and deal with as appropriate if the query has been taking
too long.
: Date: Wed, 21 Feb 2007 09:34:36 -0500
: From: Tim Johnson <ti...@saic.com>
: Reply-To: java-user@lucene.apache.org
: To: java-user@lucene.apache.org
: Subject: Stop long running queries
:
: I'm having issues with some queries taking in excess of 500 secs to run to
: completion. The system being used consists of ~100 million docs spilt up
: across ~600 indexes. The indexes are of various sizes from 15MB to 8GB and
: all searches done in the system require an exact count of matching hits.
: The documents are also indexed with various fields and types ranging from 15
: to 200 fields depending on the data indexed.
:
: 98% of all queries run under 4 sec, but the remaining 2% run for > 15 secs
: and sometimes > 500 secs. These long running queries usually consist of
: wildcard and range queries where the number of BooleanQuery statements
: reaches into the 100,000 range. Some of these queries return a relatively
: small number of results (below 300).
:
: I've recently tried to kill the search using a separate thread to wrap the
: searcher.search method, which works, but only once. The next query to come
: through throws an "IOException: The handle is invalid." in the
: FSInputStream.readInternal() method.
:
: Any suggestions on how to stop these long running queries?
:
: Tim Johnson
: SAIC
: Systems Engineer
: 757.226.1569
:
:
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: For additional commands, e-mail: java-user-help@lucene.apache.org
:
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org