You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Doug Cutting <cu...@lucene.com> on 2003/05/28 23:17:49 UTC
Re: too many hits - OutOfMemoryError
[ Moved to lucene-dev. -drc ]
David_Birthwell@VWR.COM wrote:
> But, wildcard queries that expand to many terms are allways going to be
> memory intensive in Lucene. We ran into this problem and decided to put a
> check on the number of expanded terms and abort the query if the number got
> too high.
Perhaps we should make this a feature of Lucene.
Different types of queries which expand in different ways, but they all
expand into a BooleanQuery. So perhaps BooleanQuery should get:
public static int getMaxClauseCount();
public static void setMaxClauseCount(int maxClauseCount);
When more than the specified number of clauses is added an exception
would be thrown.
Further, I propose that the default for BooleanQuery.getMaxClauseCount()
would be 1024. Each TermQuery requires around 2k bytes to process it.
This would thus limit expansions to around 2MB, however queries with
multiple wildcard terms could use more.
This simple fix would probably stop most OutOfMemory problems, which
affect everyone, while only affecting a very small fraction of queries.
The queries that are affected are in most cases probably not useful
queries anyway. If someone really wishes to permit terms to expand
further, then they can always call BooleanQuery.setMaxClauseCount().
Comments?
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: too many hits - OutOfMemoryError; Low frequency terms
Posted by Ype Kingma <yk...@xs4all.nl>.
Doug,
On Thursday 29 May 2003 14:30, Doug Cutting wrote:
> Ype Kingma wrote:
> > Terms that inadvertantly have a low document frequency (spelling
> > errors for example), get a term relevancy in query execution that
> > is higher than they actually deserve.
> > This problem surfaces when term expansion results in such terms.
> > Is there a way in Lucene to give all expanded terms the same relevancy?
>
> You could override Similarity.idf(Term, Searcher), so that all query
> terms get the same weight. Or, if you only wanted to apply this to
That seems to be overdoing it a bit.
> expanded queries, you could change your term expander so that each
> term's boost is set to 1/Similarity.idf(term, searcher), in order to
> cancel the effect of IDF for just the expanded terms.
> http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/search/Similari
>ty.html#idf(org.apache.lucene.index.Term,%20org.apache.lucene.search.Searche
>r)
I'll have a look at a change in term expansion.
Thanks a lot.
Ype
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: too many hits - OutOfMemoryError; Low frequency terms
Posted by Doug Cutting <cu...@lucene.com>.
Ype Kingma wrote:
> Terms that inadvertantly have a low document frequency (spelling
> errors for example), get a term relevancy in query execution that
> is higher than they actually deserve.
> This problem surfaces when term expansion results in such terms.
> Is there a way in Lucene to give all expanded terms the same relevancy?
You could override Similarity.idf(Term, Searcher), so that all query
terms get the same weight. Or, if you only wanted to apply this to
expanded queries, you could change your term expander so that each
term's boost is set to 1/Similarity.idf(term, searcher), in order to
cancel the effect of IDF for just the expanded terms.
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/search/Similarity.html#idf(org.apache.lucene.index.Term,%20org.apache.lucene.search.Searcher)
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: too many hits - OutOfMemoryError; Low frequency terms
Posted by Ype Kingma <yk...@xs4all.nl>.
Doug,
On Thursday 29 May 2003 09:35, Doug Cutting wrote:
> Ype Kingma wrote:
> > The source of the problem is with the wildcards, so wouldn't be better
> > to enforce a max. nr of expanded terms on these types of queries?
> > That would allow finer control than on 'top level'.
>
> That would provide more flexibility, but also more complexity. There
> are three types of query that expand into BooleanQuery: FuzzyQuery,
> PrefixQuery, and WildcardQuery. FuzzyQuery and WildcardQuery share a
> base class (MultiTermQuery), so they could be controlled by a single
> parameter, or one could make the parameter specific to each, or both.
> PrefixQuery would need a separate parameter.
>
> My inclination is to first add the top-level parameter to BooleanQuery
> that limits all of these. Then, if finer-grained control is desired, we
> could add more parameters.
>
> > Also it would be possible to interact when the number of expanded
> > terms grows out of control: ie. does the user really want
> > all these expanded terms, or would the user prefer to select
> > some of the exanded terms?
>
> That's an interesting thought. What criteria would you use for
> selection? One might limit the expansion to the more frequent terms.
> Do folks think that would be useful? Is someone interested in
> implementing it?
I think the actual interaction needed for term selection by a users
should be left out of Lucene. That leaves an API for subset
selection from a set of terms, which should be straightforward.
Limiting to more frequent terms is very dependant on the the users'
intention. Eg. when one needs high recall, it's not advisable.
> My hunch is that most queries that expand to large numbers of terms are
> not useful queries. They're also very slow, and many (most?) users
> might not wait for results anyway. I think it's better to get an error
> message up front indicating that the query is too vague.
>
> > I realize such interaction features are not needed for the avarage
> > user, so the only thing I'd like to have is that Lucene allows for
> > adding such features without needing to move Lucene functionality
> > though it's class hierarchy.
>
> Lucene allows for adding whatever features folks wish to contribute! So
> if you have a concrete idea for a term expansion API, or, better yet,
> and implementation, please send it.
The implicit good news for me is that you don't think of such features
as infeasible in combination with Lucene.
As I said, I haven't even looked at the actual details of term expansion.
I happen to be familiar with a query language in which user selection
of expanded terms is possible. It also requires prefix queries to at have
least 3 characters before the first truncation in order to limit the term
expansion.
Your mention of selecting terms with high frequency brings me to
another point.
Terms that inadvertantly have a low document frequency (spelling
errors for example), get a term relevancy in query execution that
is higher than they actually deserve.
This problem surfaces when term expansion results in such terms.
Is there a way in Lucene to give all expanded terms the same relevancy?
The problem also surfaces when two synonyms
have a very different document frequency and these synonyms
are used together in a query. In this situation one can compensate by
using appropriate query term weights, but a special OR operator
for synonyms might be preferable.
Kind regards,
Ype Kingma
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: too many hits - OutOfMemoryError
Posted by Doug Cutting <cu...@lucene.com>.
Ype Kingma wrote:
> The source of the problem is with the wildcards, so wouldn't be better
> to enforce a max. nr of expanded terms on these types of queries?
> That would allow finer control than on 'top level'.
That would provide more flexibility, but also more complexity. There
are three types of query that expand into BooleanQuery: FuzzyQuery,
PrefixQuery, and WildcardQuery. FuzzyQuery and WildcardQuery share a
base class (MultiTermQuery), so they could be controlled by a single
parameter, or one could make the parameter specific to each, or both.
PrefixQuery would need a separate parameter.
My inclination is to first add the top-level parameter to BooleanQuery
that limits all of these. Then, if finer-grained control is desired, we
could add more parameters.
> Also it would be possible to interact when the number of expanded
> terms grows out of control: ie. does the user really want
> all these expanded terms, or would the user prefer to select
> some of the exanded terms?
That's an interesting thought. What criteria would you use for
selection? One might limit the expansion to the more frequent terms.
Do folks think that would be useful? Is someone interested in
implementing it?
My hunch is that most queries that expand to large numbers of terms are
not useful queries. They're also very slow, and many (most?) users
might not wait for results anyway. I think it's better to get an error
message up front indicating that the query is too vague.
> I realize such interaction features are not needed for the avarage
> user, so the only thing I'd like to have is that Lucene allows for
> adding such features without needing to move Lucene functionality
> though it's class hierarchy.
Lucene allows for adding whatever features folks wish to contribute! So
if you have a concrete idea for a term expansion API, or, better yet,
and implementation, please send it.
> OTOH a 'top level' control for a max. nr of clauses wouldn't hurt:
> one could always set it very high, not bother about it there,
> and leave the finer control to the wildcard query terms.
Exactly my thoughts. The first thing to add is the top-level control,
then, if and when folks have good ideas for lower-level control, we can
consider adding those.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: too many hits - OutOfMemoryError
Posted by Ype Kingma <yk...@xs4all.nl>.
On Wednesday 28 May 2003 14:17, Doug Cutting wrote:
> [ Moved to lucene-dev. -drc ]
>
> David_Birthwell@VWR.COM wrote:
> > But, wildcard queries that expand to many terms are allways going to be
> > memory intensive in Lucene. We ran into this problem and decided to put
> > a check on the number of expanded terms and abort the query if the number
> > got too high.
>
> Perhaps we should make this a feature of Lucene.
I'm in favour of this.
> Different types of queries which expand in different ways, but they all
> expand into a BooleanQuery. So perhaps BooleanQuery should get:
>
> public static int getMaxClauseCount();
> public static void setMaxClauseCount(int maxClauseCount);
>
> When more than the specified number of clauses is added an exception
> would be thrown.
>
> Further, I propose that the default for BooleanQuery.getMaxClauseCount()
> would be 1024. Each TermQuery requires around 2k bytes to process it.
> This would thus limit expansions to around 2MB, however queries with
> multiple wildcard terms could use more.
>
> This simple fix would probably stop most OutOfMemory problems, which
> affect everyone, while only affecting a very small fraction of queries.
> The queries that are affected are in most cases probably not useful
> queries anyway. If someone really wishes to permit terms to expand
> further, then they can always call BooleanQuery.setMaxClauseCount().
>
> Comments?
I'n not familiar with the details of the query term expansion
in Lucene, so when this doesn't make sense, please correct me.
The source of the problem is with the wildcards, so wouldn't be better
to enforce a max. nr of expanded terms on these types of queries?
That would allow finer control than on 'top level'.
Eg. it would be possible to add a modifier to a wildcard term that
means that the user wants all expanded terms for this particular
wildcard, whatever their number.
Also it would be possible to interact when the number of expanded
terms grows out of control: ie. does the user really want
all these expanded terms, or would the user prefer to select
some of the exanded terms?
I realize such interaction features are not needed for the avarage
user, so the only thing I'd like to have is that Lucene allows for
adding such features without needing to move Lucene functionality
though it's class hierarchy.
OTOH a 'top level' control for a max. nr of clauses wouldn't hurt:
one could always set it very high, not bother about it there,
and leave the finer control to the wildcard query terms.
Kind regards,
Ype
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org