You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Erik Hatcher <er...@ehatchersolutions.com> on 2003/09/21 11:54:07 UTC
sloppyFreq - why on Similarity?
I'm just trying to come to "terms" (haha - sorry, bad pun) with all the
glory that is Lucene. I'm deep into PhraseQuery at the moment (big
test case being committed soon) and have an API question. Why is this
method:
public abstract float sloppyFreq(int distance);
on Similarity?
The only place it is called is in SloppyPhraseScorer. After thinking
about this for a bit, it sort of seems out of place and not really
something the main scoring formula needs to concern itself with. I'm
guessing the main reason its here is to allow custom Similarity
implementations to control phrase queries scoring. Is that correct?
Sorry if I fire off dumb questions from time to time - I'm just
thinking out loud and working as quickly as I can to understand all
that is going on under this gem.
Erik
Re: sloppyFreq - why on Similarity?
Posted by Terry Steichen <te...@net-frame.com>.
----- Original Message -----
From: "Erik Hatcher" <er...@ehatchersolutions.com>
To: "Lucene Developers List" <lu...@jakarta.apache.org>
Sent: Monday, September 29, 2003 11:26 PM
Subject: Re: sloppyFreq - why on Similarity?
> I'm hesitant to tinker with QueryParser and add
> more bells and whistles - its got more than its fair share of them
> already.
+1
Regards,
Terry
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: sloppyFreq - why on Similarity?
Posted by Terry Steichen <te...@net-frame.com>.
----- Original Message -----
From: "Erik Hatcher" <er...@ehatchersolutions.com>
To: "Lucene Developers List" <lu...@jakarta.apache.org>
Sent: Monday, September 29, 2003 11:26 PM
Subject: Re: sloppyFreq - why on Similarity?
> I'm hesitant to tinker with QueryParser and add
> more bells and whistles - its got more than its fair share of them
> already.
+1
Regards,
Terry
Re: sloppyFreq - why on Similarity?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Monday, September 29, 2003, at 11:26 PM, Erik Hatcher wrote:
> On Monday, September 29, 2003, at 04:56 PM, Doug Cutting wrote:
>> As a start, we could add parameters to the query parser to disable
>> these features. If we disable them by default then lots of people
>> will howl: they're popular features. Perhaps in a future major
>> release they could be disabled by default, but, for now, I think it
>> would at least be good to be able to disable them.
>
> I haven't actually tried this yet, but I suspect one could subclass
> QueryParser and override the factory methods for creating fuzzy and
> wildcard queries to effectively disable them. Although I suspect
> adding a 'throws ParseException' clause should be added to those
> factory getters to allow subclasses to reject. Or maybe returning
> null is sufficient?
>
> I will likely try this out soon and document this along with
> QueryParser somehow. I'm hesitant to tinker with QueryParser and add
> more bells and whistles - its got more than its fair share of them
> already.
Well, I hate to leave things hanging, so I went ahead and prototyped a
QueryParser subclass to see how it could reject fuzzy and wildcard
queries. Thankfully the factory methods were added, and all that was
needed was to add the throws clause to each of those methods which
makes general good sense anyway.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: sloppyFreq - why on Similarity?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Monday, September 29, 2003, at 11:26 PM, Erik Hatcher wrote:
> On Monday, September 29, 2003, at 04:56 PM, Doug Cutting wrote:
>> As a start, we could add parameters to the query parser to disable
>> these features. If we disable them by default then lots of people
>> will howl: they're popular features. Perhaps in a future major
>> release they could be disabled by default, but, for now, I think it
>> would at least be good to be able to disable them.
>
> I haven't actually tried this yet, but I suspect one could subclass
> QueryParser and override the factory methods for creating fuzzy and
> wildcard queries to effectively disable them. Although I suspect
> adding a 'throws ParseException' clause should be added to those
> factory getters to allow subclasses to reject. Or maybe returning
> null is sufficient?
>
> I will likely try this out soon and document this along with
> QueryParser somehow. I'm hesitant to tinker with QueryParser and add
> more bells and whistles - its got more than its fair share of them
> already.
Well, I hate to leave things hanging, so I went ahead and prototyped a
QueryParser subclass to see how it could reject fuzzy and wildcard
queries. Thankfully the factory methods were added, and all that was
needed was to add the throws clause to each of those methods which
makes general good sense anyway.
Erik
Re: sloppyFreq - why on Similarity?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Monday, September 29, 2003, at 04:56 PM, Doug Cutting wrote:
> As a start, we could add parameters to the query parser to disable
> these features. If we disable them by default then lots of people
> will howl: they're popular features. Perhaps in a future major
> release they could be disabled by default, but, for now, I think it
> would at least be good to be able to disable them.
I haven't actually tried this yet, but I suspect one could subclass
QueryParser and override the factory methods for creating fuzzy and
wildcard queries to effectively disable them. Although I suspect
adding a 'throws ParseException' clause should be added to those
factory getters to allow subclasses to reject. Or maybe returning null
is sufficient?
I will likely try this out soon and document this along with
QueryParser somehow. I'm hesitant to tinker with QueryParser and add
more bells and whistles - its got more than its fair share of them
already.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: sloppyFreq - why on Similarity?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Monday, September 29, 2003, at 04:56 PM, Doug Cutting wrote:
> As a start, we could add parameters to the query parser to disable
> these features. If we disable them by default then lots of people
> will howl: they're popular features. Perhaps in a future major
> release they could be disabled by default, but, for now, I think it
> would at least be good to be able to disable them.
I haven't actually tried this yet, but I suspect one could subclass
QueryParser and override the factory methods for creating fuzzy and
wildcard queries to effectively disable them. Although I suspect
adding a 'throws ParseException' clause should be added to those
factory getters to allow subclasses to reject. Or maybe returning null
is sufficient?
I will likely try this out soon and document this along with
QueryParser somehow. I'm hesitant to tinker with QueryParser and add
more bells and whistles - its got more than its fair share of them
already.
Erik
Re: sloppyFreq - why on Similarity?
Posted by Doug Cutting <cu...@lucene.com>.
Erik Hatcher wrote:
>> I think this is historical. Most of the other query classes are ones
>> I've implemented, and I've added relevant methods to Similarity for
>> them. WildcardQuery and FuzzyQuery were contributed. I've never used
>> them in an application, as I think they're potential performance
>> pitfalls, so I've probably ignored them when maintaining Similarity.
>
>
> Yeah, I'm digging deep into these implementations and see the
> performance pitfall too. Its kind of scary, actually. With a wide open
> QueryParser, it could be a potential DoS attack to force a ton of fuzzy
> or wildcard queries through. It's almost as if we should force the
> enabling of those features in QueryParser and have it off by default.
> Just an idea.
Note that BooleanQuery.maxClauseCount was added to keep these sorts of
queries from blowing things up too much. But, still, if every query
contains a generous wildcard, or was fuzzy at all, things can quickly
grind to a halt. This mainly addresses out-of-memory issues. A fuzzy
query, even if it doesn't match many terms, can still be very slow,
consuming large amounts of i/o and CPU.
As a start, we could add parameters to the query parser to disable these
features. If we disable them by default then lots of people will howl:
they're popular features. Perhaps in a future major release they could
be disabled by default, but, for now, I think it would at least be good
to be able to disable them.
Doug
Re: sloppyFreq - why on Similarity?
Posted by Doug Cutting <cu...@lucene.com>.
Erik Hatcher wrote:
>> I think this is historical. Most of the other query classes are ones
>> I've implemented, and I've added relevant methods to Similarity for
>> them. WildcardQuery and FuzzyQuery were contributed. I've never used
>> them in an application, as I think they're potential performance
>> pitfalls, so I've probably ignored them when maintaining Similarity.
>
>
> Yeah, I'm digging deep into these implementations and see the
> performance pitfall too. Its kind of scary, actually. With a wide open
> QueryParser, it could be a potential DoS attack to force a ton of fuzzy
> or wildcard queries through. It's almost as if we should force the
> enabling of those features in QueryParser and have it off by default.
> Just an idea.
Note that BooleanQuery.maxClauseCount was added to keep these sorts of
queries from blowing things up too much. But, still, if every query
contains a generous wildcard, or was fuzzy at all, things can quickly
grind to a halt. This mainly addresses out-of-memory issues. A fuzzy
query, even if it doesn't match many terms, can still be very slow,
consuming large amounts of i/o and CPU.
As a start, we could add parameters to the query parser to disable these
features. If we disable them by default then lots of people will howl:
they're popular features. Perhaps in a future major release they could
be disabled by default, but, for now, I think it would at least be good
to be able to disable them.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: sloppyFreq - why on Similarity?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Wednesday, September 24, 2003, at 02:31 PM, Doug Cutting wrote:
> Erik Hatcher wrote:
>> WildcardQuery and FuzzyQuery do have the capability of affecting the
>> scoring, although only FuzzyQuery seems to take advantage of this.
>> There is no way on a Similarity implementation to affect the factors
>> applied by these queries though. So there is some inconsistency on
>> these types of things.
>
> I think this is historical. Most of the other query classes are ones
> I've implemented, and I've added relevant methods to Similarity for
> them. WildcardQuery and FuzzyQuery were contributed. I've never used
> them in an application, as I think they're potential performance
> pitfalls, so I've probably ignored them when maintaining Similarity.
Yeah, I'm digging deep into these implementations and see the
performance pitfall too. Its kind of scary, actually. With a wide
open QueryParser, it could be a potential DoS attack to force a ton of
fuzzy or wildcard queries through. It's almost as if we should force
the enabling of those features in QueryParser and have it off by
default. Just an idea.
> It is also a judgement call as to when something should be specified
> per-query (e.g., boost, phrase slop, etc.) and when it is a policy to
> be set for all queries (IDF computation, document length
> normalization, etc.).
Hmmm..... more to think about for me.
>> So now we need a getFuzzyFreq and getWildcardFreq?! :)
>
> I think you're being sarcastic, but, to be consistent, yes, if we
> believe these have parameters that are more about ranking policy than
> are query-specific.
Yes, I was being sarcastic. I think for fun I'll try my hand out at
creating a WildcardQuery variant that scores by closeness.
> The idea is to centrally locate the ranking policy. I guess you could
> alternately make these all methods on various query classes. But if,
> e.g. idf() were a method on TermQuery, it would make construction of
> generic query parsers more difficult.
>
> Do you have another design to propose?
No, not at all. I'm just thinking out loud as I learn more and more
about the internals. Way cool stuff, and the design is superb.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: sloppyFreq - why on Similarity?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Wednesday, September 24, 2003, at 02:31 PM, Doug Cutting wrote:
> Erik Hatcher wrote:
>> WildcardQuery and FuzzyQuery do have the capability of affecting the
>> scoring, although only FuzzyQuery seems to take advantage of this.
>> There is no way on a Similarity implementation to affect the factors
>> applied by these queries though. So there is some inconsistency on
>> these types of things.
>
> I think this is historical. Most of the other query classes are ones
> I've implemented, and I've added relevant methods to Similarity for
> them. WildcardQuery and FuzzyQuery were contributed. I've never used
> them in an application, as I think they're potential performance
> pitfalls, so I've probably ignored them when maintaining Similarity.
Yeah, I'm digging deep into these implementations and see the
performance pitfall too. Its kind of scary, actually. With a wide
open QueryParser, it could be a potential DoS attack to force a ton of
fuzzy or wildcard queries through. It's almost as if we should force
the enabling of those features in QueryParser and have it off by
default. Just an idea.
> It is also a judgement call as to when something should be specified
> per-query (e.g., boost, phrase slop, etc.) and when it is a policy to
> be set for all queries (IDF computation, document length
> normalization, etc.).
Hmmm..... more to think about for me.
>> So now we need a getFuzzyFreq and getWildcardFreq?! :)
>
> I think you're being sarcastic, but, to be consistent, yes, if we
> believe these have parameters that are more about ranking policy than
> are query-specific.
Yes, I was being sarcastic. I think for fun I'll try my hand out at
creating a WildcardQuery variant that scores by closeness.
> The idea is to centrally locate the ranking policy. I guess you could
> alternately make these all methods on various query classes. But if,
> e.g. idf() were a method on TermQuery, it would make construction of
> generic query parsers more difficult.
>
> Do you have another design to propose?
No, not at all. I'm just thinking out loud as I learn more and more
about the internals. Way cool stuff, and the design is superb.
Erik
Re: sloppyFreq - why on Similarity?
Posted by Doug Cutting <cu...@lucene.com>.
Erik Hatcher wrote:
> WildcardQuery and FuzzyQuery do have the capability of affecting the
> scoring, although only FuzzyQuery seems to take advantage of this.
> There is no way on a Similarity implementation to affect the factors
> applied by these queries though. So there is some inconsistency on
> these types of things.
I think this is historical. Most of the other query classes are ones
I've implemented, and I've added relevant methods to Similarity for
them. WildcardQuery and FuzzyQuery were contributed. I've never used
them in an application, as I think they're potential performance
pitfalls, so I've probably ignored them when maintaining Similarity. It
is also a judgement call as to when something should be specified
per-query (e.g., boost, phrase slop, etc.) and when it is a policy to be
set for all queries (IDF computation, document length normalization, etc.).
> So now we need a getFuzzyFreq and getWildcardFreq?! :)
I think you're being sarcastic, but, to be consistent, yes, if we
believe these have parameters that are more about ranking policy than
are query-specific.
The idea is to centrally locate the ranking policy. I guess you could
alternately make these all methods on various query classes. But if,
e.g. idf() were a method on TermQuery, it would make construction of
generic query parsers more difficult.
Do you have another design to propose?
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: sloppyFreq - why on Similarity?
Posted by Doug Cutting <cu...@lucene.com>.
Erik Hatcher wrote:
> WildcardQuery and FuzzyQuery do have the capability of affecting the
> scoring, although only FuzzyQuery seems to take advantage of this.
> There is no way on a Similarity implementation to affect the factors
> applied by these queries though. So there is some inconsistency on
> these types of things.
I think this is historical. Most of the other query classes are ones
I've implemented, and I've added relevant methods to Similarity for
them. WildcardQuery and FuzzyQuery were contributed. I've never used
them in an application, as I think they're potential performance
pitfalls, so I've probably ignored them when maintaining Similarity. It
is also a judgement call as to when something should be specified
per-query (e.g., boost, phrase slop, etc.) and when it is a policy to be
set for all queries (IDF computation, document length normalization, etc.).
> So now we need a getFuzzyFreq and getWildcardFreq?! :)
I think you're being sarcastic, but, to be consistent, yes, if we
believe these have parameters that are more about ranking policy than
are query-specific.
The idea is to centrally locate the ranking policy. I guess you could
alternately make these all methods on various query classes. But if,
e.g. idf() were a method on TermQuery, it would make construction of
generic query parsers more difficult.
Do you have another design to propose?
Doug
Re: sloppyFreq - why on Similarity?
Posted by Ype Kingma <yk...@xs4all.nl>.
On Monday 22 September 2003 11:15, Erik Hatcher wrote:
> On Sunday, September 21, 2003, at 03:00 PM, ykingma@xs4all.nl wrote:\
>
> > That surprises me. I would have expected that sloppyFreq() would also
> > be called for fuzzy terms. In both cases there is an distance
> > that influences the effective term frequency.
>
> WildcardQuery and FuzzyQuery do have the capability of affecting the
> scoring, although only FuzzyQuery seems to take advantage of this.
> There is no way on a Similarity implementation to affect the factors
> applied by these queries though. So there is some inconsistency on
> these types of things.
>
> If I'm wrong about WildcardQuery, let me know, but I don't see that
> searching for "luc*" gives higher weight to "luck" than "lucene",
> although it seems that it should. (it uses a 1.0 multiplier hardcoded
> for the boost factor of the rewritten TermQuery).
>
> > Actually I would prefer to have two different scoring
> > methods for sloppy frases and fuzzy terms.
>
> And a different one for wildcard queries? It seems, at least to my
There is already the idf() term weight for the matching terms.
However, I'd like to have the possibility to have the same weight
for each term matching a wildcard: one use of a wildcard is
that you don't care which term matches, but you do consider
every possible match equally important.
AFAIK such equal weighting is not possible with the current
Similarity interface.
> newbie mindset, that Similarity is carrying around too much, although
> it is a one-stop place (or seems to sell itself that way) for all score
> related tweaks. But there are exceptions like MultiTermQuery
> subclasses like WildcardQuery and FuzzyQuery.
>
> Is there a need to unify these types of tweaks into Similarity?
The nice thing about an interface is that it allows you to fall
back to a default. In that sence a unification is ok I think.
> > The Similarity interface is used for determining how similar a document
> > is to a query. I think sloppyFreq() is well placed there, given
> > the current default implementation that 'works back' from (sloppy)
> > phrase frequencies (and fuzzy term frequencies?) to normal term
> > frequencies.
>
> So now we need a getFuzzyFreq and getWildcardFreq?! :)
Thinking about it, yes. Something like fuzzyFreq(int distance)
would make sense.
For the wildcards there is a problem, though. Currently they work
just like an OR query with idf() weights. However, i'd like the
possibility to have the same weights for OR'ed terms, too,
just like for the wildcards I mentioned above.
It seems that this would require to add an operator to the query
language, eg. WOR, Weighted OR, which would give
all OR'ed terms the same weight.
A first step could be wildcards with equal weights,
this would at least cover the case of very low frequency
spelling errors, that currently get a too high
idf() weight.
A rather drastic way to accomplish this would be
to have idf() always return 1 leaving all the
weighting to the query weights.
Re: sloppyFreq - why on Similarity?
Posted by Ype Kingma <yk...@xs4all.nl>.
On Monday 22 September 2003 11:15, Erik Hatcher wrote:
> On Sunday, September 21, 2003, at 03:00 PM, ykingma@xs4all.nl wrote:\
>
> > That surprises me. I would have expected that sloppyFreq() would also
> > be called for fuzzy terms. In both cases there is an distance
> > that influences the effective term frequency.
>
> WildcardQuery and FuzzyQuery do have the capability of affecting the
> scoring, although only FuzzyQuery seems to take advantage of this.
> There is no way on a Similarity implementation to affect the factors
> applied by these queries though. So there is some inconsistency on
> these types of things.
>
> If I'm wrong about WildcardQuery, let me know, but I don't see that
> searching for "luc*" gives higher weight to "luck" than "lucene",
> although it seems that it should. (it uses a 1.0 multiplier hardcoded
> for the boost factor of the rewritten TermQuery).
>
> > Actually I would prefer to have two different scoring
> > methods for sloppy frases and fuzzy terms.
>
> And a different one for wildcard queries? It seems, at least to my
There is already the idf() term weight for the matching terms.
However, I'd like to have the possibility to have the same weight
for each term matching a wildcard: one use of a wildcard is
that you don't care which term matches, but you do consider
every possible match equally important.
AFAIK such equal weighting is not possible with the current
Similarity interface.
> newbie mindset, that Similarity is carrying around too much, although
> it is a one-stop place (or seems to sell itself that way) for all score
> related tweaks. But there are exceptions like MultiTermQuery
> subclasses like WildcardQuery and FuzzyQuery.
>
> Is there a need to unify these types of tweaks into Similarity?
The nice thing about an interface is that it allows you to fall
back to a default. In that sence a unification is ok I think.
> > The Similarity interface is used for determining how similar a document
> > is to a query. I think sloppyFreq() is well placed there, given
> > the current default implementation that 'works back' from (sloppy)
> > phrase frequencies (and fuzzy term frequencies?) to normal term
> > frequencies.
>
> So now we need a getFuzzyFreq and getWildcardFreq?! :)
Thinking about it, yes. Something like fuzzyFreq(int distance)
would make sense.
For the wildcards there is a problem, though. Currently they work
just like an OR query with idf() weights. However, i'd like the
possibility to have the same weights for OR'ed terms, too,
just like for the wildcards I mentioned above.
It seems that this would require to add an operator to the query
language, eg. WOR, Weighted OR, which would give
all OR'ed terms the same weight.
A first step could be wildcards with equal weights,
this would at least cover the case of very low frequency
spelling errors, that currently get a too high
idf() weight.
A rather drastic way to accomplish this would be
to have idf() always return 1 leaving all the
weighting to the query weights.
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: sloppyFreq - why on Similarity?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Sunday, September 21, 2003, at 03:00 PM, ykingma@xs4all.nl wrote:\
> That surprises me. I would have expected that sloppyFreq() would also
> be called for fuzzy terms. In both cases there is an distance
> that influences the effective term frequency.
WildcardQuery and FuzzyQuery do have the capability of affecting the
scoring, although only FuzzyQuery seems to take advantage of this.
There is no way on a Similarity implementation to affect the factors
applied by these queries though. So there is some inconsistency on
these types of things.
If I'm wrong about WildcardQuery, let me know, but I don't see that
searching for "luc*" gives higher weight to "luck" than "lucene",
although it seems that it should. (it uses a 1.0 multiplier hardcoded
for the boost factor of the rewritten TermQuery).
> Actually I would prefer to have two different scoring
> methods for sloppy frases and fuzzy terms.
And a different one for wildcard queries? It seems, at least to my
newbie mindset, that Similarity is carrying around too much, although
it is a one-stop place (or seems to sell itself that way) for all score
related tweaks. But there are exceptions like MultiTermQuery
subclasses like WildcardQuery and FuzzyQuery.
Is there a need to unify these types of tweaks into Similarity?
> The Similarity interface is used for determining how similar a document
> is to a query. I think sloppyFreq() is well placed there, given
> the current default implementation that 'works back' from (sloppy)
> phrase frequencies (and fuzzy term frequencies?) to normal term
> frequencies.
So now we need a getFuzzyFreq and getWildcardFreq?! :)
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: sloppyFreq - why on Similarity?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Sunday, September 21, 2003, at 03:00 PM, ykingma@xs4all.nl wrote:\
> That surprises me. I would have expected that sloppyFreq() would also
> be called for fuzzy terms. In both cases there is an distance
> that influences the effective term frequency.
WildcardQuery and FuzzyQuery do have the capability of affecting the
scoring, although only FuzzyQuery seems to take advantage of this.
There is no way on a Similarity implementation to affect the factors
applied by these queries though. So there is some inconsistency on
these types of things.
If I'm wrong about WildcardQuery, let me know, but I don't see that
searching for "luc*" gives higher weight to "luck" than "lucene",
although it seems that it should. (it uses a 1.0 multiplier hardcoded
for the boost factor of the rewritten TermQuery).
> Actually I would prefer to have two different scoring
> methods for sloppy frases and fuzzy terms.
And a different one for wildcard queries? It seems, at least to my
newbie mindset, that Similarity is carrying around too much, although
it is a one-stop place (or seems to sell itself that way) for all score
related tweaks. But there are exceptions like MultiTermQuery
subclasses like WildcardQuery and FuzzyQuery.
Is there a need to unify these types of tweaks into Similarity?
> The Similarity interface is used for determining how similar a document
> is to a query. I think sloppyFreq() is well placed there, given
> the current default implementation that 'works back' from (sloppy)
> phrase frequencies (and fuzzy term frequencies?) to normal term
> frequencies.
So now we need a getFuzzyFreq and getWildcardFreq?! :)
Erik
Re: sloppyFreq - why on Similarity?
Posted by yk...@xs4all.nl.
Erik,
> I'm just trying to come to "terms" (haha - sorry, bad pun) with all the
> glory that is Lucene. I'm deep into PhraseQuery at the moment (big
> test case being committed soon) and have an API question. Why is this
> method:
>
> public abstract float sloppyFreq(int distance);
>
> on Similarity?
>
> The only place it is called is in SloppyPhraseScorer. After thinking
That surprises me. I would have expected that sloppyFreq() would also
be called for fuzzy terms. In both cases there is an distance
that influences the effective term frequency.
(See 'Fuzzy searches' and 'Proximity searches' in
http://jakarta.apache.org/lucene/docs/queryparsersyntax.html)
Actually I would prefer to have two different scoring
methods for sloppy frases and fuzzy terms.
> about this for a bit, it sort of seems out of place and not really
> something the main scoring formula needs to concern itself with. I'm
> guessing the main reason its here is to allow custom Similarity
> implementations to control phrase queries scoring. Is that correct?
Yes.
The Similarity interface is used for determining how similar a document
is to a query. I think sloppyFreq() is well placed there, given
the current default implementation that 'works back' from (sloppy)
phrase frequencies (and fuzzy term frequencies?) to normal term frequencies.
Regards,
Ype
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: sloppyFreq - why on Similarity?
Posted by yk...@xs4all.nl.
Erik,
> I'm just trying to come to "terms" (haha - sorry, bad pun) with all the
> glory that is Lucene. I'm deep into PhraseQuery at the moment (big
> test case being committed soon) and have an API question. Why is this
> method:
>
> public abstract float sloppyFreq(int distance);
>
> on Similarity?
>
> The only place it is called is in SloppyPhraseScorer. After thinking
That surprises me. I would have expected that sloppyFreq() would also
be called for fuzzy terms. In both cases there is an distance
that influences the effective term frequency.
(See 'Fuzzy searches' and 'Proximity searches' in
http://jakarta.apache.org/lucene/docs/queryparsersyntax.html)
Actually I would prefer to have two different scoring
methods for sloppy frases and fuzzy terms.
> about this for a bit, it sort of seems out of place and not really
> something the main scoring formula needs to concern itself with. I'm
> guessing the main reason its here is to allow custom Similarity
> implementations to control phrase queries scoring. Is that correct?
Yes.
The Similarity interface is used for determining how similar a document
is to a query. I think sloppyFreq() is well placed there, given
the current default implementation that 'works back' from (sloppy)
phrase frequencies (and fuzzy term frequencies?) to normal term frequencies.
Regards,
Ype