You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Erik Hatcher <er...@ehatchersolutions.com> on 2003/09/21 11:54:07 UTC

sloppyFreq - why on Similarity?

I'm just trying to come to "terms" (haha - sorry, bad pun) with all the 
glory that is Lucene.  I'm deep into PhraseQuery at the moment (big 
test case being committed soon) and have an API question.  Why is this 
method:

   public abstract float sloppyFreq(int distance);

on Similarity?

The only place it is called is in SloppyPhraseScorer.  After thinking 
about this for a bit, it sort of seems out of place and not really 
something the main scoring formula needs to concern itself with.  I'm 
guessing the main reason its here is to allow custom Similarity 
implementations to control phrase queries scoring.  Is that correct?

Sorry if I fire off dumb questions from time to time - I'm just 
thinking out loud and working as quickly as I can to understand all 
that is going on under this gem.

	Erik


Re: sloppyFreq - why on Similarity?

Posted by Terry Steichen <te...@net-frame.com>.
----- Original Message ----- 
From: "Erik Hatcher" <er...@ehatchersolutions.com>
To: "Lucene Developers List" <lu...@jakarta.apache.org>
Sent: Monday, September 29, 2003 11:26 PM
Subject: Re: sloppyFreq - why on Similarity?


> I'm hesitant to tinker with QueryParser and add 
> more bells and whistles - its got more than its fair share of them 
> already.

+1

Regards,

Terry


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: sloppyFreq - why on Similarity?

Posted by Terry Steichen <te...@net-frame.com>.
----- Original Message ----- 
From: "Erik Hatcher" <er...@ehatchersolutions.com>
To: "Lucene Developers List" <lu...@jakarta.apache.org>
Sent: Monday, September 29, 2003 11:26 PM
Subject: Re: sloppyFreq - why on Similarity?


> I'm hesitant to tinker with QueryParser and add 
> more bells and whistles - its got more than its fair share of them 
> already.

+1

Regards,

Terry


Re: sloppyFreq - why on Similarity?

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Monday, September 29, 2003, at 11:26  PM, Erik Hatcher wrote:
> On Monday, September 29, 2003, at 04:56  PM, Doug Cutting wrote:
>> As a start, we could add parameters to the query parser to disable 
>> these features.  If we disable them by default then lots of people 
>> will howl: they're popular features.  Perhaps in a future major 
>> release they could be disabled by default, but, for now, I think it 
>> would at least be good to be able to disable them.
>
> I haven't actually tried this yet, but I suspect one could subclass 
> QueryParser and override the factory methods for creating fuzzy and 
> wildcard queries to effectively disable them.  Although I suspect 
> adding a 'throws ParseException' clause should be added to those 
> factory getters to allow subclasses to reject.  Or maybe returning 
> null is sufficient?
>
> I will likely try this out soon and document this along with 
> QueryParser somehow.  I'm hesitant to tinker with QueryParser and add 
> more bells and whistles - its got more than its fair share of them 
> already.

Well, I hate to leave things hanging, so I went ahead and prototyped a 
QueryParser subclass to see how it could reject fuzzy and wildcard 
queries.  Thankfully the factory methods were added, and all that was 
needed was to add the throws clause to each of those methods which 
makes general good sense anyway.

	Erik


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: sloppyFreq - why on Similarity?

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Monday, September 29, 2003, at 11:26  PM, Erik Hatcher wrote:
> On Monday, September 29, 2003, at 04:56  PM, Doug Cutting wrote:
>> As a start, we could add parameters to the query parser to disable 
>> these features.  If we disable them by default then lots of people 
>> will howl: they're popular features.  Perhaps in a future major 
>> release they could be disabled by default, but, for now, I think it 
>> would at least be good to be able to disable them.
>
> I haven't actually tried this yet, but I suspect one could subclass 
> QueryParser and override the factory methods for creating fuzzy and 
> wildcard queries to effectively disable them.  Although I suspect 
> adding a 'throws ParseException' clause should be added to those 
> factory getters to allow subclasses to reject.  Or maybe returning 
> null is sufficient?
>
> I will likely try this out soon and document this along with 
> QueryParser somehow.  I'm hesitant to tinker with QueryParser and add 
> more bells and whistles - its got more than its fair share of them 
> already.

Well, I hate to leave things hanging, so I went ahead and prototyped a 
QueryParser subclass to see how it could reject fuzzy and wildcard 
queries.  Thankfully the factory methods were added, and all that was 
needed was to add the throws clause to each of those methods which 
makes general good sense anyway.

	Erik


Re: sloppyFreq - why on Similarity?

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Monday, September 29, 2003, at 04:56  PM, Doug Cutting wrote:
> As a start, we could add parameters to the query parser to disable 
> these features.  If we disable them by default then lots of people 
> will howl: they're popular features.  Perhaps in a future major 
> release they could be disabled by default, but, for now, I think it 
> would at least be good to be able to disable them.

I haven't actually tried this yet, but I suspect one could subclass 
QueryParser and override the factory methods for creating fuzzy and 
wildcard queries to effectively disable them.  Although I suspect 
adding a 'throws ParseException' clause should be added to those 
factory getters to allow subclasses to reject.  Or maybe returning null 
is sufficient?

I will likely try this out soon and document this along with 
QueryParser somehow.  I'm hesitant to tinker with QueryParser and add 
more bells and whistles - its got more than its fair share of them 
already.

	Erik


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: sloppyFreq - why on Similarity?

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Monday, September 29, 2003, at 04:56  PM, Doug Cutting wrote:
> As a start, we could add parameters to the query parser to disable 
> these features.  If we disable them by default then lots of people 
> will howl: they're popular features.  Perhaps in a future major 
> release they could be disabled by default, but, for now, I think it 
> would at least be good to be able to disable them.

I haven't actually tried this yet, but I suspect one could subclass 
QueryParser and override the factory methods for creating fuzzy and 
wildcard queries to effectively disable them.  Although I suspect 
adding a 'throws ParseException' clause should be added to those 
factory getters to allow subclasses to reject.  Or maybe returning null 
is sufficient?

I will likely try this out soon and document this along with 
QueryParser somehow.  I'm hesitant to tinker with QueryParser and add 
more bells and whistles - its got more than its fair share of them 
already.

	Erik


Re: sloppyFreq - why on Similarity?

Posted by Doug Cutting <cu...@lucene.com>.
Erik Hatcher wrote:
>> I think this is historical.  Most of the other query classes are ones 
>> I've implemented, and I've added relevant methods to Similarity for 
>> them.  WildcardQuery and FuzzyQuery were contributed.  I've never used 
>> them in an application, as I think they're potential performance 
>> pitfalls, so I've probably ignored them when maintaining Similarity.
> 
> 
> Yeah, I'm digging deep into these implementations and see the 
> performance pitfall too.  Its kind of scary, actually.  With a wide open 
> QueryParser, it could be a potential DoS attack to force a ton of fuzzy 
> or wildcard queries through.  It's almost as if we should force the 
> enabling of those features in QueryParser and have it off by default.  
> Just an idea.

Note that BooleanQuery.maxClauseCount was added to keep these sorts of 
queries from blowing things up too much.  But, still, if every query 
contains a generous wildcard, or was fuzzy at all, things can quickly 
grind to a halt.  This mainly addresses out-of-memory issues.  A fuzzy 
query, even if it doesn't match many terms, can still be very slow, 
consuming large amounts of i/o and CPU.

As a start, we could add parameters to the query parser to disable these 
features.  If we disable them by default then lots of people will howl: 
they're popular features.  Perhaps in a future major release they could 
be disabled by default, but, for now, I think it would at least be good 
to be able to disable them.

Doug


Re: sloppyFreq - why on Similarity?

Posted by Doug Cutting <cu...@lucene.com>.
Erik Hatcher wrote:
>> I think this is historical.  Most of the other query classes are ones 
>> I've implemented, and I've added relevant methods to Similarity for 
>> them.  WildcardQuery and FuzzyQuery were contributed.  I've never used 
>> them in an application, as I think they're potential performance 
>> pitfalls, so I've probably ignored them when maintaining Similarity.
> 
> 
> Yeah, I'm digging deep into these implementations and see the 
> performance pitfall too.  Its kind of scary, actually.  With a wide open 
> QueryParser, it could be a potential DoS attack to force a ton of fuzzy 
> or wildcard queries through.  It's almost as if we should force the 
> enabling of those features in QueryParser and have it off by default.  
> Just an idea.

Note that BooleanQuery.maxClauseCount was added to keep these sorts of 
queries from blowing things up too much.  But, still, if every query 
contains a generous wildcard, or was fuzzy at all, things can quickly 
grind to a halt.  This mainly addresses out-of-memory issues.  A fuzzy 
query, even if it doesn't match many terms, can still be very slow, 
consuming large amounts of i/o and CPU.

As a start, we could add parameters to the query parser to disable these 
features.  If we disable them by default then lots of people will howl: 
they're popular features.  Perhaps in a future major release they could 
be disabled by default, but, for now, I think it would at least be good 
to be able to disable them.

Doug


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: sloppyFreq - why on Similarity?

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Wednesday, September 24, 2003, at 02:31  PM, Doug Cutting wrote:
> Erik Hatcher wrote:
>> WildcardQuery and FuzzyQuery do have the capability of affecting the 
>> scoring, although only FuzzyQuery seems to take advantage of this.  
>> There is no way on a Similarity implementation to affect the factors 
>> applied by these queries though.  So there is some inconsistency on 
>> these types of things.
>
> I think this is historical.  Most of the other query classes are ones 
> I've implemented, and I've added relevant methods to Similarity for 
> them.  WildcardQuery and FuzzyQuery were contributed.  I've never used 
> them in an application, as I think they're potential performance 
> pitfalls, so I've probably ignored them when maintaining Similarity.

Yeah, I'm digging deep into these implementations and see the 
performance pitfall too.  Its kind of scary, actually.  With a wide 
open QueryParser, it could be a potential DoS attack to force a ton of 
fuzzy or wildcard queries through.  It's almost as if we should force 
the enabling of those features in QueryParser and have it off by 
default.  Just an idea.


>   It is also a judgement call as to when something should be specified 
> per-query (e.g., boost, phrase slop, etc.) and when it is a policy to 
> be set for all queries (IDF computation, document length 
> normalization, etc.).

Hmmm..... more to think about for me.

>> So now we need a getFuzzyFreq and getWildcardFreq?!  :)
>
> I think you're being sarcastic, but, to be consistent, yes, if we 
> believe these have parameters that are more about ranking policy than 
> are query-specific.

Yes, I was being sarcastic.  I think for fun I'll try my hand out at 
creating a WildcardQuery variant that scores by closeness.

> The idea is to centrally locate the ranking policy.  I guess you could 
> alternately make these all methods on various query classes.  But if, 
> e.g. idf() were a method on TermQuery, it would make construction of 
> generic query parsers more difficult.
>
> Do you have another design to propose?

No, not at all.  I'm just thinking out loud as I learn more and more 
about the internals.  Way cool stuff, and the design is superb.

	Erik


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: sloppyFreq - why on Similarity?

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Wednesday, September 24, 2003, at 02:31  PM, Doug Cutting wrote:
> Erik Hatcher wrote:
>> WildcardQuery and FuzzyQuery do have the capability of affecting the 
>> scoring, although only FuzzyQuery seems to take advantage of this.  
>> There is no way on a Similarity implementation to affect the factors 
>> applied by these queries though.  So there is some inconsistency on 
>> these types of things.
>
> I think this is historical.  Most of the other query classes are ones 
> I've implemented, and I've added relevant methods to Similarity for 
> them.  WildcardQuery and FuzzyQuery were contributed.  I've never used 
> them in an application, as I think they're potential performance 
> pitfalls, so I've probably ignored them when maintaining Similarity.

Yeah, I'm digging deep into these implementations and see the 
performance pitfall too.  Its kind of scary, actually.  With a wide 
open QueryParser, it could be a potential DoS attack to force a ton of 
fuzzy or wildcard queries through.  It's almost as if we should force 
the enabling of those features in QueryParser and have it off by 
default.  Just an idea.


>   It is also a judgement call as to when something should be specified 
> per-query (e.g., boost, phrase slop, etc.) and when it is a policy to 
> be set for all queries (IDF computation, document length 
> normalization, etc.).

Hmmm..... more to think about for me.

>> So now we need a getFuzzyFreq and getWildcardFreq?!  :)
>
> I think you're being sarcastic, but, to be consistent, yes, if we 
> believe these have parameters that are more about ranking policy than 
> are query-specific.

Yes, I was being sarcastic.  I think for fun I'll try my hand out at 
creating a WildcardQuery variant that scores by closeness.

> The idea is to centrally locate the ranking policy.  I guess you could 
> alternately make these all methods on various query classes.  But if, 
> e.g. idf() were a method on TermQuery, it would make construction of 
> generic query parsers more difficult.
>
> Do you have another design to propose?

No, not at all.  I'm just thinking out loud as I learn more and more 
about the internals.  Way cool stuff, and the design is superb.

	Erik


Re: sloppyFreq - why on Similarity?

Posted by Doug Cutting <cu...@lucene.com>.
Erik Hatcher wrote:
> WildcardQuery and FuzzyQuery do have the capability of affecting the 
> scoring, although only FuzzyQuery seems to take advantage of this.  
> There is no way on a Similarity implementation to affect the factors 
> applied by these queries though.  So there is some inconsistency on 
> these types of things.

I think this is historical.  Most of the other query classes are ones 
I've implemented, and I've added relevant methods to Similarity for 
them.  WildcardQuery and FuzzyQuery were contributed.  I've never used 
them in an application, as I think they're potential performance 
pitfalls, so I've probably ignored them when maintaining Similarity.  It 
is also a judgement call as to when something should be specified 
per-query (e.g., boost, phrase slop, etc.) and when it is a policy to be 
set for all queries (IDF computation, document length normalization, etc.).

> So now we need a getFuzzyFreq and getWildcardFreq?!  :)

I think you're being sarcastic, but, to be consistent, yes, if we 
believe these have parameters that are more about ranking policy than 
are query-specific.

The idea is to centrally locate the ranking policy.  I guess you could 
alternately make these all methods on various query classes.  But if, 
e.g. idf() were a method on TermQuery, it would make construction of 
generic query parsers more difficult.

Do you have another design to propose?

Doug


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: sloppyFreq - why on Similarity?

Posted by Doug Cutting <cu...@lucene.com>.
Erik Hatcher wrote:
> WildcardQuery and FuzzyQuery do have the capability of affecting the 
> scoring, although only FuzzyQuery seems to take advantage of this.  
> There is no way on a Similarity implementation to affect the factors 
> applied by these queries though.  So there is some inconsistency on 
> these types of things.

I think this is historical.  Most of the other query classes are ones 
I've implemented, and I've added relevant methods to Similarity for 
them.  WildcardQuery and FuzzyQuery were contributed.  I've never used 
them in an application, as I think they're potential performance 
pitfalls, so I've probably ignored them when maintaining Similarity.  It 
is also a judgement call as to when something should be specified 
per-query (e.g., boost, phrase slop, etc.) and when it is a policy to be 
set for all queries (IDF computation, document length normalization, etc.).

> So now we need a getFuzzyFreq and getWildcardFreq?!  :)

I think you're being sarcastic, but, to be consistent, yes, if we 
believe these have parameters that are more about ranking policy than 
are query-specific.

The idea is to centrally locate the ranking policy.  I guess you could 
alternately make these all methods on various query classes.  But if, 
e.g. idf() were a method on TermQuery, it would make construction of 
generic query parsers more difficult.

Do you have another design to propose?

Doug


Re: sloppyFreq - why on Similarity?

Posted by Ype Kingma <yk...@xs4all.nl>.
On Monday 22 September 2003 11:15, Erik Hatcher wrote:
> On Sunday, September 21, 2003, at 03:00  PM, ykingma@xs4all.nl wrote:\
>
> > That surprises me. I would have expected that sloppyFreq() would also
> > be called for fuzzy terms. In both cases there is an distance
> > that influences the effective term frequency.
>
> WildcardQuery and FuzzyQuery do have the capability of affecting the
> scoring, although only FuzzyQuery seems to take advantage of this.
> There is no way on a Similarity implementation to affect the factors
> applied by these queries though.  So there is some inconsistency on
> these types of things.
>
> If I'm wrong about WildcardQuery, let me know, but I don't see that
> searching for "luc*" gives higher weight to "luck" than "lucene",
> although it seems that it should.  (it uses a 1.0 multiplier hardcoded
> for the boost factor of the rewritten TermQuery).
>
> > Actually I would prefer to have two different scoring
> > methods for sloppy frases and fuzzy terms.
>
> And a different one for wildcard queries?  It seems, at least to my

There is already the idf() term weight for the matching terms.
However, I'd like to have the possibility to have the same weight
for each term matching a wildcard: one use of a wildcard is
that you don't care which term matches, but you do consider
every possible match equally important.
AFAIK such equal weighting is not possible with the current
Similarity interface.

> newbie mindset, that Similarity is carrying around too much, although
> it is a one-stop place (or seems to sell itself that way) for all score
> related tweaks.  But there are exceptions like MultiTermQuery
> subclasses like WildcardQuery and FuzzyQuery.
>
> Is there a need to unify these types of tweaks into Similarity?

The nice thing about an interface is that it allows you to fall
back to a default. In that sence a unification is ok I think.

> > The Similarity interface is used for determining how similar a document
> > is to a query. I think sloppyFreq() is well placed there, given
> > the current default implementation that 'works back' from (sloppy)
> > phrase frequencies (and fuzzy term frequencies?) to normal term
> > frequencies.
>
> So now we need a getFuzzyFreq and getWildcardFreq?!  :)

Thinking about it, yes. Something like fuzzyFreq(int distance)
would make sense.
For the wildcards there is a problem, though. Currently they work
just like an OR query with idf() weights. However, i'd like the 
possibility to have the same weights for OR'ed terms, too,
just like for the wildcards I mentioned above.
It seems that this would require to add an operator to the query
language, eg. WOR, Weighted OR, which would give
all OR'ed terms the same weight.
A first step could be wildcards with equal weights,
this would at least cover the case of very low frequency
spelling errors, that currently get a too high
idf() weight.

A rather drastic way to accomplish this would be
to have idf() always return 1 leaving all the 
weighting to the query weights.



Re: sloppyFreq - why on Similarity?

Posted by Ype Kingma <yk...@xs4all.nl>.
On Monday 22 September 2003 11:15, Erik Hatcher wrote:
> On Sunday, September 21, 2003, at 03:00  PM, ykingma@xs4all.nl wrote:\
>
> > That surprises me. I would have expected that sloppyFreq() would also
> > be called for fuzzy terms. In both cases there is an distance
> > that influences the effective term frequency.
>
> WildcardQuery and FuzzyQuery do have the capability of affecting the
> scoring, although only FuzzyQuery seems to take advantage of this.
> There is no way on a Similarity implementation to affect the factors
> applied by these queries though.  So there is some inconsistency on
> these types of things.
>
> If I'm wrong about WildcardQuery, let me know, but I don't see that
> searching for "luc*" gives higher weight to "luck" than "lucene",
> although it seems that it should.  (it uses a 1.0 multiplier hardcoded
> for the boost factor of the rewritten TermQuery).
>
> > Actually I would prefer to have two different scoring
> > methods for sloppy frases and fuzzy terms.
>
> And a different one for wildcard queries?  It seems, at least to my

There is already the idf() term weight for the matching terms.
However, I'd like to have the possibility to have the same weight
for each term matching a wildcard: one use of a wildcard is
that you don't care which term matches, but you do consider
every possible match equally important.
AFAIK such equal weighting is not possible with the current
Similarity interface.

> newbie mindset, that Similarity is carrying around too much, although
> it is a one-stop place (or seems to sell itself that way) for all score
> related tweaks.  But there are exceptions like MultiTermQuery
> subclasses like WildcardQuery and FuzzyQuery.
>
> Is there a need to unify these types of tweaks into Similarity?

The nice thing about an interface is that it allows you to fall
back to a default. In that sence a unification is ok I think.

> > The Similarity interface is used for determining how similar a document
> > is to a query. I think sloppyFreq() is well placed there, given
> > the current default implementation that 'works back' from (sloppy)
> > phrase frequencies (and fuzzy term frequencies?) to normal term
> > frequencies.
>
> So now we need a getFuzzyFreq and getWildcardFreq?!  :)

Thinking about it, yes. Something like fuzzyFreq(int distance)
would make sense.
For the wildcards there is a problem, though. Currently they work
just like an OR query with idf() weights. However, i'd like the 
possibility to have the same weights for OR'ed terms, too,
just like for the wildcards I mentioned above.
It seems that this would require to add an operator to the query
language, eg. WOR, Weighted OR, which would give
all OR'ed terms the same weight.
A first step could be wildcards with equal weights,
this would at least cover the case of very low frequency
spelling errors, that currently get a too high
idf() weight.

A rather drastic way to accomplish this would be
to have idf() always return 1 leaving all the 
weighting to the query weights.



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: sloppyFreq - why on Similarity?

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Sunday, September 21, 2003, at 03:00  PM, ykingma@xs4all.nl wrote:\
> That surprises me. I would have expected that sloppyFreq() would also
> be called for fuzzy terms. In both cases there is an distance
> that influences the effective term frequency.

WildcardQuery and FuzzyQuery do have the capability of affecting the 
scoring, although only FuzzyQuery seems to take advantage of this.  
There is no way on a Similarity implementation to affect the factors 
applied by these queries though.  So there is some inconsistency on 
these types of things.

If I'm wrong about WildcardQuery, let me know, but I don't see that 
searching for "luc*" gives higher weight to "luck" than "lucene", 
although it seems that it should.  (it uses a 1.0 multiplier hardcoded 
for the boost factor of the rewritten TermQuery).

> Actually I would prefer to have two different scoring
> methods for sloppy frases and fuzzy terms.

And a different one for wildcard queries?  It seems, at least to my 
newbie mindset, that Similarity is carrying around too much, although 
it is a one-stop place (or seems to sell itself that way) for all score 
related tweaks.  But there are exceptions like MultiTermQuery 
subclasses like WildcardQuery and FuzzyQuery.

Is there a need to unify these types of tweaks into Similarity?

> The Similarity interface is used for determining how similar a document
> is to a query. I think sloppyFreq() is well placed there, given
> the current default implementation that 'works back' from (sloppy)
> phrase frequencies (and fuzzy term frequencies?) to normal term 
> frequencies.

So now we need a getFuzzyFreq and getWildcardFreq?!  :)

	Erik


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: sloppyFreq - why on Similarity?

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Sunday, September 21, 2003, at 03:00  PM, ykingma@xs4all.nl wrote:\
> That surprises me. I would have expected that sloppyFreq() would also
> be called for fuzzy terms. In both cases there is an distance
> that influences the effective term frequency.

WildcardQuery and FuzzyQuery do have the capability of affecting the 
scoring, although only FuzzyQuery seems to take advantage of this.  
There is no way on a Similarity implementation to affect the factors 
applied by these queries though.  So there is some inconsistency on 
these types of things.

If I'm wrong about WildcardQuery, let me know, but I don't see that 
searching for "luc*" gives higher weight to "luck" than "lucene", 
although it seems that it should.  (it uses a 1.0 multiplier hardcoded 
for the boost factor of the rewritten TermQuery).

> Actually I would prefer to have two different scoring
> methods for sloppy frases and fuzzy terms.

And a different one for wildcard queries?  It seems, at least to my 
newbie mindset, that Similarity is carrying around too much, although 
it is a one-stop place (or seems to sell itself that way) for all score 
related tweaks.  But there are exceptions like MultiTermQuery 
subclasses like WildcardQuery and FuzzyQuery.

Is there a need to unify these types of tweaks into Similarity?

> The Similarity interface is used for determining how similar a document
> is to a query. I think sloppyFreq() is well placed there, given
> the current default implementation that 'works back' from (sloppy)
> phrase frequencies (and fuzzy term frequencies?) to normal term 
> frequencies.

So now we need a getFuzzyFreq and getWildcardFreq?!  :)

	Erik


Re: sloppyFreq - why on Similarity?

Posted by yk...@xs4all.nl.
Erik,

> I'm just trying to come to "terms" (haha - sorry, bad pun) with all the
> glory that is Lucene.  I'm deep into PhraseQuery at the moment (big
> test case being committed soon) and have an API question.  Why is this
> method:
>
>    public abstract float sloppyFreq(int distance);
>
> on Similarity?
>
> The only place it is called is in SloppyPhraseScorer.  After thinking

That surprises me. I would have expected that sloppyFreq() would also
be called for fuzzy terms. In both cases there is an distance
that influences the effective term frequency.
(See 'Fuzzy searches' and 'Proximity searches' in
http://jakarta.apache.org/lucene/docs/queryparsersyntax.html)

Actually I would prefer to have two different scoring
methods for sloppy frases and fuzzy terms.

> about this for a bit, it sort of seems out of place and not really
> something the main scoring formula needs to concern itself with.  I'm
> guessing the main reason its here is to allow custom Similarity
> implementations to control phrase queries scoring.  Is that correct?

Yes.
The Similarity interface is used for determining how similar a document
is to a query. I think sloppyFreq() is well placed there, given
the current default implementation that 'works back' from (sloppy)
phrase frequencies (and fuzzy term frequencies?) to normal term frequencies.


Regards,
Ype





---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: sloppyFreq - why on Similarity?

Posted by yk...@xs4all.nl.
Erik,

> I'm just trying to come to "terms" (haha - sorry, bad pun) with all the
> glory that is Lucene.  I'm deep into PhraseQuery at the moment (big
> test case being committed soon) and have an API question.  Why is this
> method:
>
>    public abstract float sloppyFreq(int distance);
>
> on Similarity?
>
> The only place it is called is in SloppyPhraseScorer.  After thinking

That surprises me. I would have expected that sloppyFreq() would also
be called for fuzzy terms. In both cases there is an distance
that influences the effective term frequency.
(See 'Fuzzy searches' and 'Proximity searches' in
http://jakarta.apache.org/lucene/docs/queryparsersyntax.html)

Actually I would prefer to have two different scoring
methods for sloppy frases and fuzzy terms.

> about this for a bit, it sort of seems out of place and not really
> something the main scoring formula needs to concern itself with.  I'm
> guessing the main reason its here is to allow custom Similarity
> implementations to control phrase queries scoring.  Is that correct?

Yes.
The Similarity interface is used for determining how similar a document
is to a query. I think sloppyFreq() is well placed there, given
the current default implementation that 'works back' from (sloppy)
phrase frequencies (and fuzzy term frequencies?) to normal term frequencies.


Regards,
Ype