You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Paul Taylor <pa...@fastmail.fm> on 2013/02/26 17:34:12 UTC
Uable to extends TopTermsRewrite in Lucene 4.1
In Lucene 3.6 I had code that replicated a Dismax Query, and the search
used fuzzy queries in some cases to match values. But I was finding the
score attributed to matches on fuzzy searches was completely different
to the score attributed to matches on exact searches so the total score
returned was not good. I improved this by extends TopTermsRewrite so
that if the query is a prefix query we boost it as if was exact match, I
dont fully understand this but it improved things somewhat, but in
Lucene 4.1 the rewrite() and addClause() methods are final
So how can I implement this in Lucene 4.1, do I even need to - is there
a more intuitive way to improve the scoring.
This is what I currently have that wont compile because of the final methods
//TODO FIXME WAS Overriding methods that are now final
public static class MultiTermUseIdfOfSearchTerm<Q extends
DisjunctionMaxQuery> extends TopTermsRewrite<Query> {
//public static final class MultiTermUseIdfOfSearchTerm extends
TopTermsRewrite<BooleanQuery> {
private final TFIDFSimilarity similarity;
public MultiTermUseIdfOfSearchTerm(int size) {
super(size);
this.similarity = new DefaultSimilarity();
}
@Override
protected int getMaxSize() {
return BooleanQuery.getMaxClauseCount();
}
@Override
protected DisjunctionMaxQuery getTopLevelQuery() {
return new DisjunctionMaxQuery(0.1f);
}
@Override
protected void addClause(Query topLevel, Term term, float boost) {
final Query tq = new ConstantScoreQuery(new TermQuery(term));
tq.setBoost(boost);
((DisjunctionMaxQuery)topLevel).add(tq);
}
protected float getQueryBoost(final IndexReader reader, final
MultiTermQuery query)
throws IOException {
float idf = 1f;
float df;
if (query instanceof PrefixQuery)
{
PrefixQuery fq = (PrefixQuery) query;
df = reader.docFreq(fq.getPrefix());
if(df>=1)
{
//Same as idf value for search term, 0.5 acts as
length norm
idf = (float)Math.pow(similarity.idf((int) df,
reader.numDocs()),2) * 0.5f;
}
}
return idf;
}
@Override
public Query rewrite(final IndexReader reader, final
MultiTermQuery query) throws IOException {
DisjunctionMaxQuery bq =
(DisjunctionMaxQuery)super.rewrite(reader, query);
float idfBoost = getQueryBoost(reader, query);
Iterator<Query> iterator = bq.iterator();
while(iterator.hasNext())
{
Query next = iterator.next();
next.setBoost(next.getBoost() * idfBoost);
}
return bq;
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Uable to extends TopTermsRewrite in Lucene 4.1
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
this looks also fine. If the generics in the FuzzyRewrite from the last mail are correct, the cast in this rewrite is not needed, too (and DisjunctionMaxQuery implements Iterable, so you can use a simple for-loop):
@Override
public Query rewrite(final IndexReader reader, final MultiTermQuery query) throws IOException {
DisjunctionMaxQuery dmq = rewrite.rewrite(reader, query);
float idfBoost = getQueryBoost(reader, query);
for (final Query q : dmq) {
q.setBoost(q.getBoost() * idfBoost);
}
return dmq;
}
}
On the other hand, why not simply boost the whole dmq?
Uwe
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Uable to extends TopTermsRewrite in Lucene 4.1
Posted by Paul Taylor <pa...@fastmail.fm>.
On 04/04/2013 10:59, Paul Taylor wrote:
> On 27/02/2013 10:28, Uwe Schindler wrote:
>> Hi Paul,
>>
>> QueryParser and MTQ's rewrite method have nothing to do with each
>> other. The rewrite method is (explained as simple as possible) a
>> class that is responsible to "rewrite" a MultiTermQuery to another
>> query type (generally a query that allows to add "Term" instances,
>> e.g. BooleanQuery of TermQuery or DisjunctionMaxQuery of Terms). The
>> rewrite method takes the "filtered" terms enum provided by the query
>> and creates a combined query out of it. Lucene ships with some
>> already implemented rewrite methods based on abstract classes that
>> handle the most common cases:
>>
>> - ScoringRewrite handles the case where you want to collect the terms
>> from the termsenum and place them as "clauses" in a top level query
>> (e.g. a scoring BooleanQuery). You have to implement 2 abstract
>> methods that produce the top-level query and create the clauses, that
>> can be added to the top-level query. This class is generic to the
>> top-level query, as the clauses can only be added to the correct
>> top-level query. To make this work without casting, all methods are
>> redefined to take the generics classes. So addClause() takes the
>> generic top level query and a term. The rewrite method by itself
>> returns the top level query
>> - TopTermsRewrite is similar, but has a major difference: It has
>> almost same API, but the internal implementation of this class is
>> different: It never hits the Boolean Max Clause Count, because the
>> collected terms are ordered in a priority queue and only the
>> top-ranking terms are added to the resulting top-level query. This
>> class is also generified against the top-level query. Rewrite returns
>> an instance of the top-level query.
>> - The very base class MultiTermQuery.RewriteMethod is most flexible
>> but has no concrete implementation. It is used to rewrite a MTQ to a
>> query that is not a composite top-level one with a number of terms,
>> e.g. a filter that’s handled in a totally different stage of rewriting.
>>
>> You can use the same MTQ rewrite for different MTQ types, e.g. you
>> can rewrite a FuzzyQuery to a simple ConstantScore Query or a
>> DisjunctionMaxQuery - but only the second one makes sense. On the
>> other hand it makes no sense to rewrite Prefix and Wildcard using
>> TopTermsRewrite, as those queries have terms enums withouth term
>> boosts (only Fuzzy assigns a boost to every term depending on
>> levensthein distance).
>>
>> Things to note:
>> A rewrite method in MTQ would never rewrite to another MTQ like
>> PrefixQuery - it could do this, but only in the lowest base class
>> (see above)! -> If you rely on that, your code has a major problem.
>> In that case the correct behavior would be to create a completely
>> "own"oal.search.Query (that not extends MTQ) and implement a standard
>> rewrite logic. This query could of course rewrite to MTQ's like Fuzzy
>> or Prefix. IndexSearcher rewrites the query until it is completely
>> rewritten, so your custom query would create a PrefixQuery which
>> itself rewrites to something else.
>>
>> QueryParser is just a factory for queries, its not related to MTQ. It
>> only has an option to set a "default" method for common queries. But
>> as you have a custom QueryParser, you can return the queries,
>> configured like you want, to the caller.
>>
>> Uwe
>>
> Hi Uwe
>
> Okay, think I have it now. Now have a working rewrite method for Fuzzy
> Queries
>
> public static class FuzzyTermRewrite<Q extends
> DisjunctionMaxQuery> extends TopTermsRewrite<Query> {
>
> public FuzzyTermRewrite(int size) {
> super(size);
> }
>
> @Override
> protected int getMaxSize() {
> return BooleanQuery.getMaxClauseCount();
> }
>
> @Override
> protected DisjunctionMaxQuery getTopLevelQuery() {
> return new DisjunctionMaxQuery(0.1f);
> }
>
> @Override
> protected void addClause(Query topLevel, Term term, int
> docCount, float boost, TermContext states) {
> final Query tq = new ConstantScoreQuery(new
> TermQuery(term, states));
> tq.setBoost(boost);
> ((DisjunctionMaxQuery)topLevel).add(tq);
> }
> }
>
> and now writing a separate class for Prefix Queries so it does
> actually modify the idf
>
> Paul
>
and this is my prefix rewrite method:
/**
*
* Prefix matches are rewritten to a DisjunctionMaxQuery instead of
the more usual BooleanQuery so that
* if search term matches multiple fields we just take the best
field rather summing all matches like a boolean
* query. The 0.1 for tiebreaker is to favour documents that
contain all words rather than the same word in multiple
* fields.
*
* We set the idf the same as an exact match so that a wildcard
match to a term which happens to be rarer than
* the exact term we were searching for does not get an unfairly
high idf.
*
*/
public static class PrefixTermRewrite extends
MultiTermQuery.RewriteMethod {
private TFIDFSimilarity similarity;
private FuzzyTermRewrite rewrite;
public PrefixTermRewrite(int size) {
this.rewrite = new FuzzyTermRewrite(size);
this.similarity = new DefaultSimilarity();
}
protected float getQueryBoost(final IndexReader reader, final
MultiTermQuery query)
throws IOException {
float idf = 1f;
float df;
PrefixQuery fq = (PrefixQuery) query;
df = reader.docFreq(fq.getPrefix());
if(df>=1)
{
//Same as idf value for search term, 0.5 acts as length
norm
idf = (float)Math.pow(similarity.idf((int) df,
reader.numDocs()),2) * 0.5f;
}
return idf;
}
@Override
public Query rewrite(final IndexReader reader, final
MultiTermQuery query) throws IOException {
DisjunctionMaxQuery dmq =
(DisjunctionMaxQuery)rewrite.rewrite(reader, query);
float idfBoost = getQueryBoost(reader, query);
Iterator<Query> iterator = dmq.iterator();
while(iterator.hasNext())
{
Query next = iterator.next();
next.setBoost(next.getBoost() * idfBoost);
}
return dmq;
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Uable to extends TopTermsRewrite in Lucene 4.1
Posted by Uwe Schindler <uw...@thetaphi.de>.
> Okay, think I have it now. Now have a working rewrite method for Fuzzy
> Queries
>
> public static class FuzzyTermRewrite<Q extends DisjunctionMaxQuery>
> extends TopTermsRewrite<Query> {
>
> public FuzzyTermRewrite(int size) {
> super(size);
> }
>
> @Override
> protected int getMaxSize() {
> return BooleanQuery.getMaxClauseCount();
> }
>
> @Override
> protected DisjunctionMaxQuery getTopLevelQuery() {
> return new DisjunctionMaxQuery(0.1f);
> }
>
> @Override
> protected void addClause(Query topLevel, Term term, int docCount,
> float boost, TermContext states) {
> final Query tq = new ConstantScoreQuery(new TermQuery(term,
> states));
> tq.setBoost(boost);
> ((DisjunctionMaxQuery)topLevel).add(tq);
> }
> }
>
> and now writing a separate class for Prefix Queries so it does actually modify
> the idf
Hi,
looks OK, only the generics are wrong, must be:
public static class FuzzyTermRewrite extends TopTermsRewrite<DisjunctionMaxQuery> {
And then:
protected void addClause(DisjunctionMaxQuery topLevel, Term term, int docCount, float boost, TermContext states) {
This cast is then obsolete: ((DisjunctionMaxQuery)topLevel).add(tq);
Uwe
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Uable to extends TopTermsRewrite in Lucene 4.1
Posted by Paul Taylor <pa...@fastmail.fm>.
On 27/02/2013 10:28, Uwe Schindler wrote:
> Hi Paul,
>
> QueryParser and MTQ's rewrite method have nothing to do with each other. The rewrite method is (explained as simple as possible) a class that is responsible to "rewrite" a MultiTermQuery to another query type (generally a query that allows to add "Term" instances, e.g. BooleanQuery of TermQuery or DisjunctionMaxQuery of Terms). The rewrite method takes the "filtered" terms enum provided by the query and creates a combined query out of it. Lucene ships with some already implemented rewrite methods based on abstract classes that handle the most common cases:
>
> - ScoringRewrite handles the case where you want to collect the terms from the termsenum and place them as "clauses" in a top level query (e.g. a scoring BooleanQuery). You have to implement 2 abstract methods that produce the top-level query and create the clauses, that can be added to the top-level query. This class is generic to the top-level query, as the clauses can only be added to the correct top-level query. To make this work without casting, all methods are redefined to take the generics classes. So addClause() takes the generic top level query and a term. The rewrite method by itself returns the top level query
> - TopTermsRewrite is similar, but has a major difference: It has almost same API, but the internal implementation of this class is different: It never hits the Boolean Max Clause Count, because the collected terms are ordered in a priority queue and only the top-ranking terms are added to the resulting top-level query. This class is also generified against the top-level query. Rewrite returns an instance of the top-level query.
> - The very base class MultiTermQuery.RewriteMethod is most flexible but has no concrete implementation. It is used to rewrite a MTQ to a query that is not a composite top-level one with a number of terms, e.g. a filter that’s handled in a totally different stage of rewriting.
>
> You can use the same MTQ rewrite for different MTQ types, e.g. you can rewrite a FuzzyQuery to a simple ConstantScore Query or a DisjunctionMaxQuery - but only the second one makes sense. On the other hand it makes no sense to rewrite Prefix and Wildcard using TopTermsRewrite, as those queries have terms enums withouth term boosts (only Fuzzy assigns a boost to every term depending on levensthein distance).
>
> Things to note:
> A rewrite method in MTQ would never rewrite to another MTQ like PrefixQuery - it could do this, but only in the lowest base class (see above)! -> If you rely on that, your code has a major problem. In that case the correct behavior would be to create a completely "own"oal.search.Query (that not extends MTQ) and implement a standard rewrite logic. This query could of course rewrite to MTQ's like Fuzzy or Prefix. IndexSearcher rewrites the query until it is completely rewritten, so your custom query would create a PrefixQuery which itself rewrites to something else.
>
> QueryParser is just a factory for queries, its not related to MTQ. It only has an option to set a "default" method for common queries. But as you have a custom QueryParser, you can return the queries, configured like you want, to the caller.
>
> Uwe
>
Hi Uwe
Okay, think I have it now. Now have a working rewrite method for Fuzzy
Queries
public static class FuzzyTermRewrite<Q extends DisjunctionMaxQuery>
extends TopTermsRewrite<Query> {
public FuzzyTermRewrite(int size) {
super(size);
}
@Override
protected int getMaxSize() {
return BooleanQuery.getMaxClauseCount();
}
@Override
protected DisjunctionMaxQuery getTopLevelQuery() {
return new DisjunctionMaxQuery(0.1f);
}
@Override
protected void addClause(Query topLevel, Term term, int
docCount, float boost, TermContext states) {
final Query tq = new ConstantScoreQuery(new TermQuery(term,
states));
tq.setBoost(boost);
((DisjunctionMaxQuery)topLevel).add(tq);
}
}
and now writing a separate class for Prefix Queries so it does actually
modify the idf
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Uable to extends TopTermsRewrite in Lucene 4.1
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Paul,
QueryParser and MTQ's rewrite method have nothing to do with each other. The rewrite method is (explained as simple as possible) a class that is responsible to "rewrite" a MultiTermQuery to another query type (generally a query that allows to add "Term" instances, e.g. BooleanQuery of TermQuery or DisjunctionMaxQuery of Terms). The rewrite method takes the "filtered" terms enum provided by the query and creates a combined query out of it. Lucene ships with some already implemented rewrite methods based on abstract classes that handle the most common cases:
- ScoringRewrite handles the case where you want to collect the terms from the termsenum and place them as "clauses" in a top level query (e.g. a scoring BooleanQuery). You have to implement 2 abstract methods that produce the top-level query and create the clauses, that can be added to the top-level query. This class is generic to the top-level query, as the clauses can only be added to the correct top-level query. To make this work without casting, all methods are redefined to take the generics classes. So addClause() takes the generic top level query and a term. The rewrite method by itself returns the top level query
- TopTermsRewrite is similar, but has a major difference: It has almost same API, but the internal implementation of this class is different: It never hits the Boolean Max Clause Count, because the collected terms are ordered in a priority queue and only the top-ranking terms are added to the resulting top-level query. This class is also generified against the top-level query. Rewrite returns an instance of the top-level query.
- The very base class MultiTermQuery.RewriteMethod is most flexible but has no concrete implementation. It is used to rewrite a MTQ to a query that is not a composite top-level one with a number of terms, e.g. a filter that’s handled in a totally different stage of rewriting.
You can use the same MTQ rewrite for different MTQ types, e.g. you can rewrite a FuzzyQuery to a simple ConstantScore Query or a DisjunctionMaxQuery - but only the second one makes sense. On the other hand it makes no sense to rewrite Prefix and Wildcard using TopTermsRewrite, as those queries have terms enums withouth term boosts (only Fuzzy assigns a boost to every term depending on levensthein distance).
Things to note:
A rewrite method in MTQ would never rewrite to another MTQ like PrefixQuery - it could do this, but only in the lowest base class (see above)! -> If you rely on that, your code has a major problem. In that case the correct behavior would be to create a completely "own"oal.search.Query (that not extends MTQ) and implement a standard rewrite logic. This query could of course rewrite to MTQ's like Fuzzy or Prefix. IndexSearcher rewrites the query until it is completely rewritten, so your custom query would create a PrefixQuery which itself rewrites to something else.
QueryParser is just a factory for queries, its not related to MTQ. It only has an option to set a "default" method for common queries. But as you have a custom QueryParser, you can return the queries, configured like you want, to the caller.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Paul Taylor [mailto:paul_t100@fastmail.fm]
> Sent: Wednesday, February 27, 2013 10:53 AM
> To: java-user@lucene.apache.org
> Cc: Uwe Schindler
> Subject: Re: Uable to extends TopTermsRewrite in Lucene 4.1
>
> On 26/02/2013 18:01, Paul Taylor wrote:
> > On 26/02/2013 17:22, Uwe Schindler wrote:
> >>> Hi,
> >>>
> >>> You cannot override rewrite() because you could easily break the
> >>> logic behind TopTermsRewrite. If you want another behavior, subclass
> >>> another base class and wrap the TopTermsRewrite instead of
> >>> subclassing it (the generics also enforce that the rewrite needs to
> >>> rewrite() to a class that’s specified in the generics parameter).
> >>>
> >>> addClause() is not final, its abstract. There is one "final" helper
> >>> method used by the rewrite itself, but the methods you need to
> >>> override are abstract.
> >>>
> >>> Also your generics seem to be wrong, leading to the above question...
> >> In addition, you cast the call to super.rewrite() to DisjMaxQuery, so
> >> it is definitely a DisjMaxQuery (because getTopLevelQuery() always
> >> returns one, see generics). You then pass this DisjMaxQuery to this
> >> "getQueryBoostMethod", which checks for instanceof PrefixQuery. This
> >> can never return true, so the boost is always 1. You can therefore
> >> nuke the whole rewrite method (as it changes nothing) and only
> >> implement getToplevelQuery() and addClause().
> >>
> >> Uwe
> Not making much sense of this, Im trying to use the same rewritemethod for
>
> QueryParser
>
> and
>
> FuzzyQuery
> PrefixQuery
>
> I'm confused as to whether I should be applying at both stages, and what the
> generic parameter should be as the javadoc for QueryParser.
> setMultiTermRewriteMethod() implies you need to change this to use
> different rewrite for fuzzy and prefix queries but you seem to be saying I
> should be using FuzzyQuery as the generic type whihc would prevent this
> wouldn't it ?
>
> Is there a fuller explanation of rewrite methods anywhere ?
>
> Full class below if it makes things clearer
>
> Paul
>
> package org.musicbrainz.search.servlet;
>
> import org.apache.lucene.index.IndexReader;
> import org.apache.lucene.index.Term;
> import org.apache.lucene.queryparser.classic.ParseException;
> import org.apache.lucene.queryparser.classic.QueryParser;
> import org.apache.lucene.search.*;
> import org.apache.lucene.search.similarities.DefaultSimilarity;
> import org.apache.lucene.search.similarities.Similarity;
> import org.apache.lucene.search.similarities.TFIDFSimilarity;
> import org.musicbrainz.search.LuceneVersion;
>
> import java.io.IOException;
> import java.util.HashMap;
> import java.util.Iterator;
> import java.util.Map;
>
> public class DismaxQueryParser {
>
> public static String IMPOSSIBLE_FIELD_NAME = "\uFFFC\uFFFC\uFFFC";
> protected DisjunctionQueryParser dqp;
>
> protected DismaxQueryParser() {
> }
>
> public DismaxQueryParser(org.apache.lucene.analysis.Analyzer
> analyzer) {
> dqp = new DisjunctionQueryParser(IMPOSSIBLE_FIELD_NAME,
> analyzer);
> //TODO FIXME
> //dqp.setMultiTermRewriteMethod(new
> MultiTermUseIdfOfSearchTerm(100));
> }
>
> /**
> * Create query consists of disjunction queries for each term fields combo,
> and then
> * a phrase search for each field as long as the original query is more than
> one term
> *
> * @param query
> * @return
> *
> */
> public Query parse(String query) throws
> org.apache.lucene.queryparser.classic.ParseException {
>
> Query term = dqp.parse(DismaxQueryParser.IMPOSSIBLE_FIELD_NAME
> + ":(" + query + ")");
> Query phrase =
> dqp.parse(DismaxQueryParser.IMPOSSIBLE_FIELD_NAME + ":\"" + query +
> "\"");
> return buildTopQuery(term, phrase);
> }
>
> /**
> * If a phrase query was built then we create a boolean query that requires
> something to match in
> * the term query, under normal circumstances if nothing matches the
> term query nothing will match the phrase
> * query
> *
> * @param term
> * @param phrase
> * @return
> */
> protected Query buildTopQuery(Query term, Query phrase) {
> if (phrase instanceof DisjunctionMaxQuery) {
> BooleanQuery bq = new BooleanQuery(true);
> bq.add(term, BooleanClause.Occur.MUST);
> bq.add(phrase, BooleanClause.Occur.SHOULD);
> return bq;
> } else {
> return term;
> }
> }
>
>
> public void addAlias(String field, DismaxAlias dismaxAlias) {
> dqp.addAlias(field, dismaxAlias);
> }
>
> static class DisjunctionQueryParser extends QueryParser {
>
> //Only make search terms that are this length fuzzy searchable and only
> match to terms that are also this length
> protected static final int MIN_FIELD_LENGTH_TO_MAKE_FUZZY = 4;
> protected static final float FUZZY_SIMILARITY = 0.5f;
>
> //Reduce boost of wildcard/fuzzy matches compared to exact matches
> protected static final float WILDCARD_BOOST_REDUCER = 0.8f;
>
> //Reduce phrase query scores otherwise there is too much difference
> between a document that matches on
> //phrase and one that doesn't quite.
> protected static final float PHRASE_BOOST_REDUCER = 0.2f;
>
>
> public DisjunctionQueryParser(String defaultField,
> org.apache.lucene.analysis.Analyzer analyzer) {
> super(LuceneVersion.LUCENE_VERSION, defaultField, analyzer);
> }
>
> protected Map<String, DismaxAlias> aliases = new
> HashMap<String, DismaxAlias>(3);
>
> //Field to DismaxAlias
> public void addAlias(String field, DismaxAlias dismaxAlias) {
> aliases.put(field, dismaxAlias);
> }
>
> // TODO FIXME _ Unable to create rewrite using original idf
> // Rewrite Method used by Prefix Search and Fuzzy Search, use
> idf of the original term
> //MultiTermQuery.RewriteMethod
> fuzzyAndPrefixQueryRewriteMethod
> // = new MultiTermUseIdfOfSearchTerm(200);
>
> protected boolean checkQuery(DisjunctionMaxQuery q, Query
> querySub, boolean quoted, DismaxAlias a, String f) {
> if (querySub != null) {
> //if query was quoted but doesn't generate a phrase
> query we reject it
> if ((!quoted) || (querySub instanceof PhraseQuery)) {
> //Reduce phrase because will have matched both
> parts giving far too much score differential
> if (quoted) {
> querySub.setBoost(PHRASE_BOOST_REDUCER);
> } else {
> querySub.setBoost(a.getFields().get(f).getBoost());
> }
> q.add(querySub);
> return true;
> }
> }
> return false;
> }
>
> @Override
> //TODO FIXME was using a FLOAT similarity value of 0.5 but now
> chnaged to integral
> protected Query getFuzzyQuery(String field, String termStr,
> float minSimilarity) {
> Term t = new Term(field, termStr);
> FuzzyQuery fq = new FuzzyQuery(t, 2,
> MIN_FIELD_LENGTH_TO_MAKE_FUZZY);
> //TODO FIXME
> //fq.setRewriteMethod(fuzzyAndPrefixQueryRewriteMethod);
> return fq;
> }
>
>
> protected Query getFieldQuery(String field, String queryText,
> boolean quoted)
> throws ParseException
> {
> //If field is an alias
> if (aliases.containsKey(field)) {
>
> DismaxAlias a = aliases.get(field);
> DisjunctionMaxQuery q = new
> DisjunctionMaxQuery(a.getTie());
> boolean ok = false;
>
> for (String f : a.getFields().keySet()) {
>
> //if query can be created for this field and text
> Query querySub;
> Query queryWildcard = null;
> Query queryFuzzy = null;
>
> DismaxAlias.AliasField af = a.getFields().get(f);
> if (!quoted && queryText.length() >=
> MIN_FIELD_LENGTH_TO_MAKE_FUZZY) {
> querySub = getFieldQuery(f, queryText, quoted);
> if (querySub instanceof TermQuery) {
>
> if (af.isFuzzy()) {
> Term t = ((TermQuery) querySub).getTerm();
> queryWildcard = newPrefixQuery(new
> Term(t.field(), t.text()));
> queryFuzzy = getFuzzyQuery(t.field(),
> t.text(), FUZZY_SIMILARITY);
> queryFuzzy.setBoost(af.getBoost() *
> WILDCARD_BOOST_REDUCER);
> q.add(queryFuzzy);
> queryWildcard.setBoost(af.getBoost() *
> WILDCARD_BOOST_REDUCER);
> q.add(queryWildcard);
> }
> }
> } else {
> querySub = getFieldQuery(f, queryText, quoted);
> }
>
> if (checkQuery(q, querySub, quoted, a, f) && ok ==
> false) {
> ok = true;
> }
> }
> //Something has been added to disjunction query
> return ok ? q : null;
>
> } else {
> //usual Field
> try {
> return super.getFieldQuery(field, queryText, quoted);
> } catch (Exception e) {
> return null;
> }
> }
> }
>
> /**
> * Builds a new PrefixQuery instance
> * @param prefix Prefix term
> * @return new PrefixQuery instance
> */
> protected Query newPrefixQuery(Term prefix){
> PrefixQuery query = new PrefixQuery(prefix);
> //TODO FIXME
> //query.setRewriteMethod(fuzzyAndPrefixQueryRewriteMethod);
> return query;
> }
> }
>
> /*
> TODO FIXME WAS Overriding methods that are now final
> public static class MultiTermUseIdfOfSearchTerm<Q extends
> DisjunctionMaxQuery> extends TopTermsRewrite<Query> {
>
> //public static final class MultiTermUseIdfOfSearchTerm extends
> TopTermsRewrite<BooleanQuery> {
> private final TFIDFSimilarity similarity;
>
> public MultiTermUseIdfOfSearchTerm(int size) {
> super(size);
> this.similarity = new DefaultSimilarity();
>
> }
>
> @Override
> protected int getMaxSize() {
> return BooleanQuery.getMaxClauseCount();
> }
>
> @Override
> protected DisjunctionMaxQuery getTopLevelQuery() {
> return new DisjunctionMaxQuery(0.1f);
> }
>
> @Override
> protected void addClause(Query topLevel, Term term, float boost) {
> final Query tq = new ConstantScoreQuery(new TermQuery(term));
> tq.setBoost(boost);
> ((DisjunctionMaxQuery)topLevel).add(tq);
> }
>
> protected float getQueryBoost(final IndexReader reader, final
> MultiTermQuery query)
> throws IOException {
> float idf = 1f;
> float df;
> if (query instanceof PrefixQuery)
> {
> PrefixQuery fq = (PrefixQuery) query;
> df = reader.docFreq(fq.getPrefix());
> if(df>=1)
> {
> //Same as idf value for search term, 0.5 acts as
> length norm
> idf = (float)Math.pow(similarity.idf((int) df,
> reader.numDocs()),2) * 0.5f;
> }
> }
> return idf;
> }
>
> @Override
> public Query rewrite(final IndexReader reader, final
> MultiTermQuery query) throws IOException {
> DisjunctionMaxQuery bq =
> (DisjunctionMaxQuery)super.rewrite(reader, query);
>
> float idfBoost = getQueryBoost(reader, query);
> Iterator<Query> iterator = bq.iterator();
> while(iterator.hasNext())
> {
> Query next = iterator.next();
> next.setBoost(next.getBoost() * idfBoost);
> }
> return bq;
> }
>
> }
> */
> }
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Uable to extends TopTermsRewrite in Lucene 4.1
Posted by Paul Taylor <pa...@fastmail.fm>.
On 26/02/2013 18:01, Paul Taylor wrote:
> On 26/02/2013 17:22, Uwe Schindler wrote:
>>> Hi,
>>>
>>> You cannot override rewrite() because you could easily break the logic
>>> behind TopTermsRewrite. If you want another behavior, subclass another
>>> base class and wrap the TopTermsRewrite instead of subclassing it (the
>>> generics also enforce that the rewrite needs to rewrite() to a class
>>> that’s
>>> specified in the generics parameter).
>>>
>>> addClause() is not final, its abstract. There is one "final" helper
>>> method used
>>> by the rewrite itself, but the methods you need to override are
>>> abstract.
>>>
>>> Also your generics seem to be wrong, leading to the above question...
>> In addition, you cast the call to super.rewrite() to DisjMaxQuery, so
>> it is definitely a DisjMaxQuery (because getTopLevelQuery() always
>> returns one, see generics). You then pass this DisjMaxQuery to this
>> "getQueryBoostMethod", which checks for instanceof PrefixQuery. This
>> can never return true, so the boost is always 1. You can therefore
>> nuke the whole rewrite method (as it changes nothing) and only
>> implement getToplevelQuery() and addClause().
>>
>> Uwe
Not making much sense of this, Im trying to use the same rewritemethod for
QueryParser
and
FuzzyQuery
PrefixQuery
I'm confused as to whether I should be applying at both stages, and what
the generic parameter should be
as the javadoc for QueryParser. setMultiTermRewriteMethod() implies you
need to change this to use different rewrite for fuzzy and prefix
queries but you seem to be saying I should be using FuzzyQuery as the
generic type whihc would prevent this wouldn't it ?
Is there a fuller explanation of rewrite methods anywhere ?
Full class below if it makes things clearer
Paul
package org.musicbrainz.search.servlet;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.Term;
import org.apache.lucene.queryparser.classic.ParseException;
import org.apache.lucene.queryparser.classic.QueryParser;
import org.apache.lucene.search.*;
import org.apache.lucene.search.similarities.DefaultSimilarity;
import org.apache.lucene.search.similarities.Similarity;
import org.apache.lucene.search.similarities.TFIDFSimilarity;
import org.musicbrainz.search.LuceneVersion;
import java.io.IOException;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
public class DismaxQueryParser {
public static String IMPOSSIBLE_FIELD_NAME = "\uFFFC\uFFFC\uFFFC";
protected DisjunctionQueryParser dqp;
protected DismaxQueryParser() {
}
public DismaxQueryParser(org.apache.lucene.analysis.Analyzer
analyzer) {
dqp = new DisjunctionQueryParser(IMPOSSIBLE_FIELD_NAME, analyzer);
//TODO FIXME
//dqp.setMultiTermRewriteMethod(new
MultiTermUseIdfOfSearchTerm(100));
}
/**
* Create query consists of disjunction queries for each term
fields combo, and then
* a phrase search for each field as long as the original query is
more than one term
*
* @param query
* @return
*
*/
public Query parse(String query) throws
org.apache.lucene.queryparser.classic.ParseException {
Query term = dqp.parse(DismaxQueryParser.IMPOSSIBLE_FIELD_NAME
+ ":(" + query + ")");
Query phrase =
dqp.parse(DismaxQueryParser.IMPOSSIBLE_FIELD_NAME + ":\"" + query + "\"");
return buildTopQuery(term, phrase);
}
/**
* If a phrase query was built then we create a boolean query that
requires something to match in
* the term query, under normal circumstances if nothing matches
the term query nothing will match the phrase
* query
*
* @param term
* @param phrase
* @return
*/
protected Query buildTopQuery(Query term, Query phrase) {
if (phrase instanceof DisjunctionMaxQuery) {
BooleanQuery bq = new BooleanQuery(true);
bq.add(term, BooleanClause.Occur.MUST);
bq.add(phrase, BooleanClause.Occur.SHOULD);
return bq;
} else {
return term;
}
}
public void addAlias(String field, DismaxAlias dismaxAlias) {
dqp.addAlias(field, dismaxAlias);
}
static class DisjunctionQueryParser extends QueryParser {
//Only make search terms that are this length fuzzy searchable
and only match to terms that are also this length
protected static final int MIN_FIELD_LENGTH_TO_MAKE_FUZZY = 4;
protected static final float FUZZY_SIMILARITY = 0.5f;
//Reduce boost of wildcard/fuzzy matches compared to exact matches
protected static final float WILDCARD_BOOST_REDUCER = 0.8f;
//Reduce phrase query scores otherwise there is too much
difference between a document that matches on
//phrase and one that doesn't quite.
protected static final float PHRASE_BOOST_REDUCER = 0.2f;
public DisjunctionQueryParser(String defaultField,
org.apache.lucene.analysis.Analyzer analyzer) {
super(LuceneVersion.LUCENE_VERSION, defaultField, analyzer);
}
protected Map<String, DismaxAlias> aliases = new
HashMap<String, DismaxAlias>(3);
//Field to DismaxAlias
public void addAlias(String field, DismaxAlias dismaxAlias) {
aliases.put(field, dismaxAlias);
}
// TODO FIXME _ Unable to create rewrite using original idf
// Rewrite Method used by Prefix Search and Fuzzy Search, use
idf of the original term
//MultiTermQuery.RewriteMethod fuzzyAndPrefixQueryRewriteMethod
// = new MultiTermUseIdfOfSearchTerm(200);
protected boolean checkQuery(DisjunctionMaxQuery q, Query
querySub, boolean quoted, DismaxAlias a, String f) {
if (querySub != null) {
//if query was quoted but doesn't generate a phrase
query we reject it
if ((!quoted) || (querySub instanceof PhraseQuery)) {
//Reduce phrase because will have matched both
parts giving far too much score differential
if (quoted) {
querySub.setBoost(PHRASE_BOOST_REDUCER);
} else {
querySub.setBoost(a.getFields().get(f).getBoost());
}
q.add(querySub);
return true;
}
}
return false;
}
@Override
//TODO FIXME was using a FLOAT similarity value of 0.5 but now
chnaged to integral
protected Query getFuzzyQuery(String field, String termStr,
float minSimilarity) {
Term t = new Term(field, termStr);
FuzzyQuery fq = new FuzzyQuery(t, 2,
MIN_FIELD_LENGTH_TO_MAKE_FUZZY);
//TODO FIXME
//fq.setRewriteMethod(fuzzyAndPrefixQueryRewriteMethod);
return fq;
}
protected Query getFieldQuery(String field, String queryText,
boolean quoted)
throws ParseException
{
//If field is an alias
if (aliases.containsKey(field)) {
DismaxAlias a = aliases.get(field);
DisjunctionMaxQuery q = new
DisjunctionMaxQuery(a.getTie());
boolean ok = false;
for (String f : a.getFields().keySet()) {
//if query can be created for this field and text
Query querySub;
Query queryWildcard = null;
Query queryFuzzy = null;
DismaxAlias.AliasField af = a.getFields().get(f);
if (!quoted && queryText.length() >=
MIN_FIELD_LENGTH_TO_MAKE_FUZZY) {
querySub = getFieldQuery(f, queryText, quoted);
if (querySub instanceof TermQuery) {
if (af.isFuzzy()) {
Term t = ((TermQuery) querySub).getTerm();
queryWildcard = newPrefixQuery(new
Term(t.field(), t.text()));
queryFuzzy = getFuzzyQuery(t.field(),
t.text(), FUZZY_SIMILARITY);
queryFuzzy.setBoost(af.getBoost() *
WILDCARD_BOOST_REDUCER);
q.add(queryFuzzy);
queryWildcard.setBoost(af.getBoost() *
WILDCARD_BOOST_REDUCER);
q.add(queryWildcard);
}
}
} else {
querySub = getFieldQuery(f, queryText, quoted);
}
if (checkQuery(q, querySub, quoted, a, f) && ok ==
false) {
ok = true;
}
}
//Something has been added to disjunction query
return ok ? q : null;
} else {
//usual Field
try {
return super.getFieldQuery(field, queryText, quoted);
} catch (Exception e) {
return null;
}
}
}
/**
* Builds a new PrefixQuery instance
* @param prefix Prefix term
* @return new PrefixQuery instance
*/
protected Query newPrefixQuery(Term prefix){
PrefixQuery query = new PrefixQuery(prefix);
//TODO FIXME
//query.setRewriteMethod(fuzzyAndPrefixQueryRewriteMethod);
return query;
}
}
/*
TODO FIXME WAS Overriding methods that are now final
public static class MultiTermUseIdfOfSearchTerm<Q extends
DisjunctionMaxQuery> extends TopTermsRewrite<Query> {
//public static final class MultiTermUseIdfOfSearchTerm extends
TopTermsRewrite<BooleanQuery> {
private final TFIDFSimilarity similarity;
public MultiTermUseIdfOfSearchTerm(int size) {
super(size);
this.similarity = new DefaultSimilarity();
}
@Override
protected int getMaxSize() {
return BooleanQuery.getMaxClauseCount();
}
@Override
protected DisjunctionMaxQuery getTopLevelQuery() {
return new DisjunctionMaxQuery(0.1f);
}
@Override
protected void addClause(Query topLevel, Term term, float boost) {
final Query tq = new ConstantScoreQuery(new TermQuery(term));
tq.setBoost(boost);
((DisjunctionMaxQuery)topLevel).add(tq);
}
protected float getQueryBoost(final IndexReader reader, final
MultiTermQuery query)
throws IOException {
float idf = 1f;
float df;
if (query instanceof PrefixQuery)
{
PrefixQuery fq = (PrefixQuery) query;
df = reader.docFreq(fq.getPrefix());
if(df>=1)
{
//Same as idf value for search term, 0.5 acts as
length norm
idf = (float)Math.pow(similarity.idf((int) df,
reader.numDocs()),2) * 0.5f;
}
}
return idf;
}
@Override
public Query rewrite(final IndexReader reader, final
MultiTermQuery query) throws IOException {
DisjunctionMaxQuery bq =
(DisjunctionMaxQuery)super.rewrite(reader, query);
float idfBoost = getQueryBoost(reader, query);
Iterator<Query> iterator = bq.iterator();
while(iterator.hasNext())
{
Query next = iterator.next();
next.setBoost(next.getBoost() * idfBoost);
}
return bq;
}
}
*/
}
Re: Uable to extends TopTermsRewrite in Lucene 4.1
Posted by Paul Taylor <pa...@fastmail.fm>.
On 26/02/2013 17:22, Uwe Schindler wrote:
>> Hi,
>>
>> You cannot override rewrite() because you could easily break the logic
>> behind TopTermsRewrite. If you want another behavior, subclass another
>> base class and wrap the TopTermsRewrite instead of subclassing it (the
>> generics also enforce that the rewrite needs to rewrite() to a class that’s
>> specified in the generics parameter).
>>
>> addClause() is not final, its abstract. There is one "final" helper method used
>> by the rewrite itself, but the methods you need to override are abstract.
>>
>> Also your generics seem to be wrong, leading to the above question...
> In addition, you cast the call to super.rewrite() to DisjMaxQuery, so it is definitely a DisjMaxQuery (because getTopLevelQuery() always returns one, see generics). You then pass this DisjMaxQuery to this "getQueryBoostMethod", which checks for instanceof PrefixQuery. This can never return true, so the boost is always 1. You can therefore nuke the whole rewrite method (as it changes nothing) and only implement getToplevelQuery() and addClause().
>
> Uwe
Thanks Uwe, like I said I didn't really understand what I was doing but
it did seem to do the job, I'll try out your recommendations
Paul
>
>>> -----Original Message-----
>>> From: Paul Taylor [mailto:paul_t100@fastmail.fm]
>>> Sent: Tuesday, February 26, 2013 5:34 PM
>>> To: java-user@lucene.apache.org
>>> Subject: Uable to extends TopTermsRewrite in Lucene 4.1
>>>
>>> In Lucene 3.6 I had code that replicated a Dismax Query, and the
>>> search used fuzzy queries in some cases to match values. But I was
>>> finding the score attributed to matches on fuzzy searches was
>>> completely different to the score attributed to matches on exact
>>> searches so the total score returned was not good. I improved this by
>>> extends TopTermsRewrite so that if the query is a prefix query we
>>> boost it as if was exact match, I dont fully understand this but it
>>> improved things somewhat, but in Lucene 4.1 the
>>> rewrite() and addClause() methods are final
>>>
>>> So how can I implement this in Lucene 4.1, do I even need to - is
>>> there a more intuitive way to improve the scoring.
>>>
>>> This is what I currently have that wont compile because of the final
>>> methods
>>>
>>> //TODO FIXME WAS Overriding methods that are now final
>>> public static class MultiTermUseIdfOfSearchTerm<Q extends
>>> DisjunctionMaxQuery> extends TopTermsRewrite<Query> {
>>>
>>> //public static final class MultiTermUseIdfOfSearchTerm extends
>>> TopTermsRewrite<BooleanQuery> {
>>> private final TFIDFSimilarity similarity;
>>>
>>> public MultiTermUseIdfOfSearchTerm(int size) {
>>> super(size);
>>> this.similarity = new DefaultSimilarity();
>>>
>>> }
>>>
>>> @Override
>>> protected int getMaxSize() {
>>> return BooleanQuery.getMaxClauseCount();
>>> }
>>>
>>> @Override
>>> protected DisjunctionMaxQuery getTopLevelQuery() {
>>> return new DisjunctionMaxQuery(0.1f);
>>> }
>>>
>>> @Override
>>> protected void addClause(Query topLevel, Term term, float boost) {
>>> final Query tq = new ConstantScoreQuery(new TermQuery(term));
>>> tq.setBoost(boost);
>>> ((DisjunctionMaxQuery)topLevel).add(tq);
>>> }
>>>
>>> protected float getQueryBoost(final IndexReader reader, final
>>> MultiTermQuery query)
>>> throws IOException {
>>> float idf = 1f;
>>> float df;
>>> if (query instanceof PrefixQuery)
>>> {
>>> PrefixQuery fq = (PrefixQuery) query;
>>> df = reader.docFreq(fq.getPrefix());
>>> if(df>=1)
>>> {
>>> //Same as idf value for search term, 0.5 acts as length norm
>>> idf = (float)Math.pow(similarity.idf((int) df,
>>> reader.numDocs()),2) * 0.5f;
>>> }
>>> }
>>> return idf;
>>> }
>>>
>>> @Override
>>> public Query rewrite(final IndexReader reader, final
>>> MultiTermQuery
>>> query) throws IOException {
>>> DisjunctionMaxQuery bq =
>>> (DisjunctionMaxQuery)super.rewrite(reader, query);
>>>
>>> float idfBoost = getQueryBoost(reader, query);
>>> Iterator<Query> iterator = bq.iterator();
>>> while(iterator.hasNext())
>>> {
>>> Query next = iterator.next();
>>> next.setBoost(next.getBoost() * idfBoost);
>>> }
>>> return bq;
>>> }
>>>
>>> }
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Uable to extends TopTermsRewrite in Lucene 4.1
Posted by Uwe Schindler <uw...@thetaphi.de>.
> Hi,
>
> You cannot override rewrite() because you could easily break the logic
> behind TopTermsRewrite. If you want another behavior, subclass another
> base class and wrap the TopTermsRewrite instead of subclassing it (the
> generics also enforce that the rewrite needs to rewrite() to a class that’s
> specified in the generics parameter).
>
> addClause() is not final, its abstract. There is one "final" helper method used
> by the rewrite itself, but the methods you need to override are abstract.
>
> Also your generics seem to be wrong, leading to the above question...
In addition, you cast the call to super.rewrite() to DisjMaxQuery, so it is definitely a DisjMaxQuery (because getTopLevelQuery() always returns one, see generics). You then pass this DisjMaxQuery to this "getQueryBoostMethod", which checks for instanceof PrefixQuery. This can never return true, so the boost is always 1. You can therefore nuke the whole rewrite method (as it changes nothing) and only implement getToplevelQuery() and addClause().
Uwe
>
> > -----Original Message-----
> > From: Paul Taylor [mailto:paul_t100@fastmail.fm]
> > Sent: Tuesday, February 26, 2013 5:34 PM
> > To: java-user@lucene.apache.org
> > Subject: Uable to extends TopTermsRewrite in Lucene 4.1
> >
> > In Lucene 3.6 I had code that replicated a Dismax Query, and the
> > search used fuzzy queries in some cases to match values. But I was
> > finding the score attributed to matches on fuzzy searches was
> > completely different to the score attributed to matches on exact
> > searches so the total score returned was not good. I improved this by
> > extends TopTermsRewrite so that if the query is a prefix query we
> > boost it as if was exact match, I dont fully understand this but it
> > improved things somewhat, but in Lucene 4.1 the
> > rewrite() and addClause() methods are final
> >
> > So how can I implement this in Lucene 4.1, do I even need to - is
> > there a more intuitive way to improve the scoring.
> >
> > This is what I currently have that wont compile because of the final
> > methods
> >
> > //TODO FIXME WAS Overriding methods that are now final
> > public static class MultiTermUseIdfOfSearchTerm<Q extends
> > DisjunctionMaxQuery> extends TopTermsRewrite<Query> {
> >
> > //public static final class MultiTermUseIdfOfSearchTerm extends
> > TopTermsRewrite<BooleanQuery> {
> > private final TFIDFSimilarity similarity;
> >
> > public MultiTermUseIdfOfSearchTerm(int size) {
> > super(size);
> > this.similarity = new DefaultSimilarity();
> >
> > }
> >
> > @Override
> > protected int getMaxSize() {
> > return BooleanQuery.getMaxClauseCount();
> > }
> >
> > @Override
> > protected DisjunctionMaxQuery getTopLevelQuery() {
> > return new DisjunctionMaxQuery(0.1f);
> > }
> >
> > @Override
> > protected void addClause(Query topLevel, Term term, float boost) {
> > final Query tq = new ConstantScoreQuery(new TermQuery(term));
> > tq.setBoost(boost);
> > ((DisjunctionMaxQuery)topLevel).add(tq);
> > }
> >
> > protected float getQueryBoost(final IndexReader reader, final
> > MultiTermQuery query)
> > throws IOException {
> > float idf = 1f;
> > float df;
> > if (query instanceof PrefixQuery)
> > {
> > PrefixQuery fq = (PrefixQuery) query;
> > df = reader.docFreq(fq.getPrefix());
> > if(df>=1)
> > {
> > //Same as idf value for search term, 0.5 acts as length norm
> > idf = (float)Math.pow(similarity.idf((int) df,
> > reader.numDocs()),2) * 0.5f;
> > }
> > }
> > return idf;
> > }
> >
> > @Override
> > public Query rewrite(final IndexReader reader, final
> > MultiTermQuery
> > query) throws IOException {
> > DisjunctionMaxQuery bq =
> > (DisjunctionMaxQuery)super.rewrite(reader, query);
> >
> > float idfBoost = getQueryBoost(reader, query);
> > Iterator<Query> iterator = bq.iterator();
> > while(iterator.hasNext())
> > {
> > Query next = iterator.next();
> > next.setBoost(next.getBoost() * idfBoost);
> > }
> > return bq;
> > }
> >
> > }
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Uable to extends TopTermsRewrite in Lucene 4.1
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
You cannot override rewrite() because you could easily break the logic behind TopTermsRewrite. If you want another behavior, subclass another base class and wrap the TopTermsRewrite instead of subclassing it (the generics also enforce that the rewrite needs to rewrite() to a class that’s specified in the generics parameter).
addClause() is not final, its abstract. There is one "final" helper method used by the rewrite itself, but the methods you need to override are abstract.
Also your generics seem to be wrong, leading to the above question...
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Paul Taylor [mailto:paul_t100@fastmail.fm]
> Sent: Tuesday, February 26, 2013 5:34 PM
> To: java-user@lucene.apache.org
> Subject: Uable to extends TopTermsRewrite in Lucene 4.1
>
> In Lucene 3.6 I had code that replicated a Dismax Query, and the search used
> fuzzy queries in some cases to match values. But I was finding the score
> attributed to matches on fuzzy searches was completely different to the
> score attributed to matches on exact searches so the total score returned
> was not good. I improved this by extends TopTermsRewrite so that if the
> query is a prefix query we boost it as if was exact match, I dont fully
> understand this but it improved things somewhat, but in Lucene 4.1 the
> rewrite() and addClause() methods are final
>
> So how can I implement this in Lucene 4.1, do I even need to - is there a
> more intuitive way to improve the scoring.
>
> This is what I currently have that wont compile because of the final methods
>
> //TODO FIXME WAS Overriding methods that are now final
> public static class MultiTermUseIdfOfSearchTerm<Q extends
> DisjunctionMaxQuery> extends TopTermsRewrite<Query> {
>
> //public static final class MultiTermUseIdfOfSearchTerm extends
> TopTermsRewrite<BooleanQuery> {
> private final TFIDFSimilarity similarity;
>
> public MultiTermUseIdfOfSearchTerm(int size) {
> super(size);
> this.similarity = new DefaultSimilarity();
>
> }
>
> @Override
> protected int getMaxSize() {
> return BooleanQuery.getMaxClauseCount();
> }
>
> @Override
> protected DisjunctionMaxQuery getTopLevelQuery() {
> return new DisjunctionMaxQuery(0.1f);
> }
>
> @Override
> protected void addClause(Query topLevel, Term term, float boost) {
> final Query tq = new ConstantScoreQuery(new TermQuery(term));
> tq.setBoost(boost);
> ((DisjunctionMaxQuery)topLevel).add(tq);
> }
>
> protected float getQueryBoost(final IndexReader reader, final
> MultiTermQuery query)
> throws IOException {
> float idf = 1f;
> float df;
> if (query instanceof PrefixQuery)
> {
> PrefixQuery fq = (PrefixQuery) query;
> df = reader.docFreq(fq.getPrefix());
> if(df>=1)
> {
> //Same as idf value for search term, 0.5 acts as length norm
> idf = (float)Math.pow(similarity.idf((int) df,
> reader.numDocs()),2) * 0.5f;
> }
> }
> return idf;
> }
>
> @Override
> public Query rewrite(final IndexReader reader, final MultiTermQuery
> query) throws IOException {
> DisjunctionMaxQuery bq =
> (DisjunctionMaxQuery)super.rewrite(reader, query);
>
> float idfBoost = getQueryBoost(reader, query);
> Iterator<Query> iterator = bq.iterator();
> while(iterator.hasNext())
> {
> Query next = iterator.next();
> next.setBoost(next.getBoost() * idfBoost);
> }
> return bq;
> }
>
> }
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org