You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by d33mb33 <da...@entity.co.uk> on 2007/12/07 12:33:22 UTC
Post processing to get around TooManyClauses?
I have developed a fuzzy search application over a database of books (titles,
authors etc) and it works really well. (I use Lucene.Net but read the
JavaDocs and forums for java Lucene)
However I've got an interesting use case with "TooManyClauses" and need some
help in solving it.
My users accept that silly title queries like "m*" are going to return too
many results to be useful but they want to combine the wildcard searches
with other search terms.
For example:
Use Case 1
A user wants to search for books by "Charles Dickens"
This works fine using a term query and about 500 results are returned
Use Case 2
A user wants to search for books by "Charles Dickens" where the title starts
with M
This throws a TooManyClauses exceptions (or eats a huge amount of RAM)
because, I guess, Lucene treats the two as independent queries and M* is
expanded across the whole index and not just the books by Charles Dickens.
User's don't understand why the Use Case 1 works but Use Case 2 doesn't.
Use Case 2 as actually being a more restrictive query and will return better
results than Use Case 1.
I've thought a bit about how to solve this but none of them seem very
elegant or efficient.
One solution could be too eliminate one or two character wildcards from the
inital search and then loop through the results doing a String.contains or
something horrible.
Another solution could be through clever use of the QueryFilter classes but
I don't quite understand how they work yet.
Any suggestions would be welcome
--
View this message in context: http://www.nabble.com/Post-processing-to-get-around-TooManyClauses--tf4961564.html#a14210833
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Post processing to get around TooManyClauses?
Posted by d33mb33 <da...@entity.co.uk>.
Ok I'm still struggling with this and a QueryFilter didn't help me one bit
:-(
I'm trying to query for books by "Charles Dickens" that start with "m". I
have constructed a QueryFilter for the author search and a PrefixQuery for
the title search. A simplified version of my code is below.
' Charles Dickens query filter
Dim authorQry As TermQuery = New TermQuery(New Term("ContributorName",
"dickens"))
Dim authorFlter As QueryFilter = New QueryFilter(authorQry)
' M* query
Dim titleQry As PrefixQuery = New PrefixQuery(New Term("Title", "m"))
Dim topDocs As TopDocs = GetIndexSearcher().Search(titleQry,
authorFlter, maxResults)
This still throws a TooManyClauses exception because the PrefixQuery is
being expanded across the entire index.
--
View this message in context: http://www.nabble.com/Post-processing-to-get-around-TooManyClauses--tp14210833p14273737.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Post processing to get around TooManyClauses?
Posted by Erick Erickson <er...@gmail.com>.
I was thinking that you'd assemble a BooleanQuery, perhaps with phrase
queries, perhaps not. The critical thing here is to add in a BooleanClause
that included your filter on the relevant field (s).
Erick
On Dec 7, 2007 9:57 AM, d33mb33 <da...@entity.co.uk> wrote:
>
>
> Erick Erickson wrote:
> >
> > Have you looked at Filters? Essentially, you construct a bitmap where
> each
> > bit corresponds to a document and pass that along into your search.
> > Constructing a filter is surprisingly speedy.
> >
>
> Thanks for the reply.. I'll take a look at filters.
>
> So in my "Charles Dickens starting with m" use case, should I make a
> filter
> for "Charles Dickens" and then do a prefix query on title passing "m"...?
>
> i.e. rewrite the quick query as a filter so the prefix and wildcard
> queries
> work against a smaller index.
>
> Or have I got this the wrong way around :-S
>
> --
> View this message in context:
> http://www.nabble.com/Post-processing-to-get-around-TooManyClauses--tf4961564.html#a14213935
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Post processing to get around TooManyClauses?
Posted by d33mb33 <da...@entity.co.uk>.
Erick Erickson wrote:
>
> Have you looked at Filters? Essentially, you construct a bitmap where each
> bit corresponds to a document and pass that along into your search.
> Constructing a filter is surprisingly speedy.
>
Thanks for the reply.. I'll take a look at filters.
So in my "Charles Dickens starting with m" use case, should I make a filter
for "Charles Dickens" and then do a prefix query on title passing "m"...?
i.e. rewrite the quick query as a filter so the prefix and wildcard queries
work against a smaller index.
Or have I got this the wrong way around :-S
--
View this message in context: http://www.nabble.com/Post-processing-to-get-around-TooManyClauses--tf4961564.html#a14213935
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Post processing to get around TooManyClauses?
Posted by Erick Erickson <er...@gmail.com>.
Have you looked at Filters? Essentially, you construct a bitmap where each
bit corresponds to a document and pass that along into your search.
Constructing
a filter is surprisingly speedy.
What you lose is relevancy for the filtered part of the query. See
ConstantScoreQuery.
Also, search the mail archive for wildcard and you'll find a wealth of
information.
See the thread "I just don't get wildcards at all" for some very worthwhile
discussion.
Best
Erick
On Dec 7, 2007 6:33 AM, d33mb33 <da...@entity.co.uk> wrote:
>
> I have developed a fuzzy search application over a database of books
> (titles,
> authors etc) and it works really well. (I use Lucene.Net but read the
> JavaDocs and forums for java Lucene)
>
> However I've got an interesting use case with "TooManyClauses" and need
> some
> help in solving it.
>
> My users accept that silly title queries like "m*" are going to return too
> many results to be useful but they want to combine the wildcard searches
> with other search terms.
>
> For example:
>
> Use Case 1
> A user wants to search for books by "Charles Dickens"
> This works fine using a term query and about 500 results are returned
>
> Use Case 2
> A user wants to search for books by "Charles Dickens" where the title
> starts
> with M
> This throws a TooManyClauses exceptions (or eats a huge amount of RAM)
> because, I guess, Lucene treats the two as independent queries and M* is
> expanded across the whole index and not just the books by Charles Dickens.
> User's don't understand why the Use Case 1 works but Use Case 2 doesn't.
> Use Case 2 as actually being a more restrictive query and will return
> better
> results than Use Case 1.
>
> I've thought a bit about how to solve this but none of them seem very
> elegant or efficient.
> One solution could be too eliminate one or two character wildcards from
> the
> inital search and then loop through the results doing a String.contains or
> something horrible.
> Another solution could be through clever use of the QueryFilter classes
> but
> I don't quite understand how they work yet.
>
> Any suggestions would be welcome
>
> --
> View this message in context:
> http://www.nabble.com/Post-processing-to-get-around-TooManyClauses--tf4961564.html#a14210833
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>