You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Hoss Man (JIRA)" <ji...@apache.org> on 2006/05/24 02:55:30 UTC
[jira] Updated: (LUCENE-577) SweetSpotSimiliarity
[ http://issues.apache.org/jira/browse/LUCENE-577?page=all ]
Hoss Man updated LUCENE-577:
----------------------------
Attachment: ss.patch
patch file containing class and test cases
> SweetSpotSimiliarity
> --------------------
>
> Key: LUCENE-577
> URL: http://issues.apache.org/jira/browse/LUCENE-577
> Project: Lucene - Java
> Type: New Feature
> Components: Search
> Reporter: Hoss Man
> Priority: Minor
> Attachments: ss.patch
>
> This is a new Similarity implimention for the contrib/miscellaneous/ package, it provides a Similiarty designed for people who know the "sweetspot" of their data. three major pieces of functionality are included:
> 1) a lengthNorm which creates a "platuea" of values.
> 2) a baseline tf that provides a fixed value for tf's up to a minimum, at which point it becomes a sqrt curve (this is used by the tf(int) function.
> 3) a hyperbolic tf function which is best explained by graphing the equation. this isn't used by default, but is available for subclasses to call from their own tf functions.
> All constants used in all functions are configurable. In the case of lengthNorm, the constants are configurable per field, as well as allowing for defaults for unspecified fields.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: SweetSpotSimiliarity
Posted by Doug Cutting <cu...@apache.org>.
Marvin Humphrey wrote:
> The only answer seems to be to apply different lengthNorm algos to
> different fields.
FYI, Nutch uses the following:
http://svn.apache.org/viewvc/lucene/nutch/trunk/src/java/org/apache/nutch/indexer/NutchSimilarity.java?view=markup
All of this is seat-of-the-pants, developed by hand-tuning a few
queries. Like code optimization, relevance tuning is better done with
large amounts of real data. If you have trusted relevant/non-relevant
judgements for a representative sample of queries, then you can do a
much better job of setting these parameters. Unfortunately, such
judgements are expensive to generate.
For Web data, one source of relevance judgements is:
http://ir.dcs.gla.ac.uk/test_collections/
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: SweetSpotSimiliarity
Posted by Marvin Humphrey <ma...@rectangular.com>.
On May 24, 2006, at 2:12 PM, Chris Hostetter wrote:
> : Adding a fieldName argument to similarity.tf(freq) would add
> : significant overhead, since it gets called a *lot*.
>
> ...why would having the extra param add signifianct overhead? ...
> or is
> the point just that if someone wants customized tf based on field
> name, it
> would be better to make that choice once when starting to score a
> query,
> (since the choice is going to be the same for all docs) rather then
> everytime tf is called becuase the way a user chooses *might*
> involve a
> lot of overhead?
I suppose that's true that the default wouldn't suffer much if at all
-- it'd just ignore the fieldName param.
public float tf(float freq, String fieldName) {
return tf(freq);
}
However, if you wanted to override that behavior, you'd have to apply
at least one conditional for each doc that the Scorer plows through.
public float tf(float freq, String fieldName) {
if ("title".equals(fieldName))
return 1.0f;
else
return tf(freq);
}
That's going to be less efficient than overriding that method in an
alternative Similarity instance for the field "title" and retrieving
it once. You never know how much until you benchmark it, of course.
> Should a similar change be made to
> IndexWriter, and replace Similarity.lengthNorm(String,int) with
> lengthNorm(int) ?
I like it. <evilgrin> That's one step closer towards assigning each
Field a pluggable, comprehensive codec. </evilgrin>
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: SweetSpotSimiliarity
Posted by Chris Hostetter <ho...@fucit.org>.
: > Searcher.getSimilarity() is documented as a public method, but maybe
: > it's still OK to add Searcher.getSimilarity(String field)
:
: Right. We can add a new method that, by default, punts to the old method.
it's not really clear to me why Marvin said...
: Adding a fieldName argument to similarity.tf(freq) would add
: significant overhead, since it gets called a *lot*.
...why would having the extra param add signifianct overhead? ... or is
the point just that if someone wants customized tf based on field name, it
would be better to make that choice once when starting to score a query,
(since the choice is going to be the same for all docs) rather then
everytime tf is called becuase the way a user chooses *might* involve a
lot of overhead?
Basically what we're discussing is adding this to Searcher...
public Similarity getSimilarity(String fieldName) {
return this.getSimilarity();
}
And then modifying any Lucene core Query class that has a field associated
with it to call that method from it's own getSimilarity(Searcher) method.
Which raises the question in my mind: Should a similar change be made to
IndexWriter, and replace Similarity.lengthNorm(String,int) with
lengthNorm(int) ?
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: SweetSpotSimiliarity
Posted by Chris Hostetter <ho...@fucit.org>.
: into Scorers, I feel like I have to go spelunk all of them to make
: sure that no Query ever propagates its Similarity instance to a child
: in such a way. Making the broad statement that all instances of
i don't think there is any situation where a Query passes it's similarity
down into any subqueries it may have -- there isn't even a mechanism for
it other then to wrap each subquery in some sort of delegating proxy class
at rewrite time.
it would be possibe to write a query class A which would rewrite to an
anonymous subclass of B with getSimilarity overrien to use A's similarity,
but that would be a lot of work, and i don't think any Query goes to that
trouble.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: SweetSpotSimiliarity
Posted by Marvin Humphrey <ma...@rectangular.com>.
On May 24, 2006, at 12:49 PM, Doug Cutting wrote:
>> and change over the various Query subclasses to request it via a
>> new Query.getSimilarity(Searcher searcher, String Field) method.
>
> I don't see the need for the latter. TermQuery.getSimilarity
> (Searcher) could simply call Searcher.getSimilarity(String field), no?
Yes, that'd work.
>> That would work for TermQuery and PhraseQuery, as they're both
>> limited to a single field. Assessing whether it would work for
>> BooleanQuery is confusing. Various clauses in BooleanQuery can
>> have different fields. The main concern I have is that each
>> subquery not have an incorrect Similarity instance imposed on it
>> from above by the parent BooleanQuery.
>
> I don't follow. BooleanQuery doesn't call tf(), so its okay if it
> still calls Searcher.getSimilarity(), right?
I think I agree.
I'm imagining that there would be some API along the lines of
setSimilarity(String field, Similarity sim) at the Searcher level.
What I'm worried about is that some parent would have its Similarity
instance determined by Query.getSimilarity(Searcher searcher) and
that it would then lose track of the override that was performed at
the Searcher level and supply the wrong Similarity to a child.
Not being familiar with all of the Query classes and how they expand
into Scorers, I feel like I have to go spelunk all of them to make
sure that no Query ever propagates its Similarity instance to a child
in such a way. Making the broad statement that all instances of
Query.getSimilarity(Searcher searcher) should be replaced with
Query.getSimilarity(Searcher searcher, String fieldName) was easier.
It saved me from having to do the spelunking now, by imposing the
requirement that we'd have to do that spelunking later.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: SweetSpotSimiliarity
Posted by Doug Cutting <cu...@apache.org>.
Marvin Humphrey wrote:
> Searcher.getSimilarity() is documented as a public method, but maybe
> it's still OK to add Searcher.getSimilarity(String field)
Right. We can add a new method that, by default, punts to the old method.
> and change
> over the various Query subclasses to request it via a new
> Query.getSimilarity(Searcher searcher, String Field) method.
I don't see the need for the latter. TermQuery.getSimilarity(Searcher)
could simply call Searcher.getSimilarity(String field), no?
> That would work for TermQuery and PhraseQuery, as they're both limited
> to a single field. Assessing whether it would work for BooleanQuery is
> confusing. Various clauses in BooleanQuery can have different fields.
> The main concern I have is that each subquery not have an incorrect
> Similarity instance imposed on it from above by the parent BooleanQuery.
I don't follow. BooleanQuery doesn't call tf(), so its okay if it still
calls Searcher.getSimilarity(), right?
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: SweetSpotSimiliarity
Posted by Marvin Humphrey <ma...@rectangular.com>.
On May 24, 2006, at 11:33 AM, Doug Cutting wrote:
> Might it work to instead change Searcher.getSimilarity to take a
> Query or field parameter? Then one could add Searcher.setSimilarity
> (String field) method?
Searcher.getSimilarity() is documented as a public method, but maybe
it's still OK to add Searcher.getSimilarity(String field) and change
over the various Query subclasses to request it via a new
Query.getSimilarity(Searcher searcher, String Field) method.
That would work for TermQuery and PhraseQuery, as they're both
limited to a single field. Assessing whether it would work for
BooleanQuery is confusing. Various clauses in BooleanQuery can have
different fields. The main concern I have is that each subquery not
have an incorrect Similarity instance imposed on it from above by the
parent BooleanQuery.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: SweetSpotSimiliarity
Posted by Doug Cutting <cu...@apache.org>.
Chris Hostetter wrote:
> certianly .. but it's not just TermQueries right ... PhraseQueries,
> SpanQueries .. lots of things use tf -- and even things like WildCardQuery
> would need to be subclasses, so when they rewrite to a collection of
> TermQueries, they propogate the information about what Similarity to use.
Might it work to instead change Searcher.getSimilarity to take a Query
or field parameter? Then one could add Searcher.setSimilarity(String
field) method?
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: SweetSpotSimiliarity
Posted by Chris Hostetter <ho...@fucit.org>.
: > ...but good lord if that isn't a pain.
:
: There's no reason you have to copy this everywhere. You could use a
: regular, non-anonymous subclass of TermQuery and use that everywhere,
: no? QueryParser should probably have a getTermQuery() method that can
: be overridden to support this kind of thing...
certianly .. but it's not just TermQueries right ... PhraseQueries,
SpanQueries .. lots of things use tf -- and even things like WildCardQuery
would need to be subclasses, so when they rewrite to a collection of
TermQueries, they propogate the information about what Similarity to use.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: SweetSpotSimiliarity
Posted by Doug Cutting <cu...@apache.org>.
Chris Hostetter wrote:
> Alas ... tf() doesn't take in a field name, to do this, you'd have to
> override the Similarity each time your construct a query object,
> something like this i believe...
>
> Query q = new TermQuery(t) {
> public Similarity getSimilarity(Searcher s) {
> return new SimilarityDelegator
> (TermQuery.this.super.getSimilarity(s)) {
> public float tf(freq) {
> ...
> }
> }
> }
> }
> }
>
> ...but good lord if that isn't a pain.
There's no reason you have to copy this everywhere. You could use a
regular, non-anonymous subclass of TermQuery and use that everywhere,
no? QueryParser should probably have a getTermQuery() method that can
be overridden to support this kind of thing...
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: SweetSpotSimiliarity
Posted by Marvin Humphrey <ma...@rectangular.com>.
On May 23, 2006, at 11:38 PM, Chris Hostetter wrote:
> it has the nice property of giving small increases as the frequency
> increases a small amount, then increasing faster once you reach the
> point
> where you think small increases are significant, and then grows slower
> again once you are above the point where you think more occurances are
> acctually significant.
Gotcha.
"Normalization" and "norms" are funny words to use in this context,
since you're aggressively manipulating a score multiplier rather than
normalizing in the usual sense.
> : I tried to graph the hyperbolic function (tip for OS X users: check
> : out Grapher.app, in Utilities). It looks like by default,
> everything
> : cancels out it returns a constant 2. But it's pretty
> complicated, so
> : maybe I missed something.
>
> Hmm... maybe i screwed up the defaults at some point ...
Nah, I found my error -- just a typo that happened somewhere while I
was swapping in the default values. I now see something similar to
what you describe, though the plateaus above and below the transition
look completely flat.
> Alas ... tf() doesn't take in a field name, to do this, you'd have to
> override the Similarity each time your construct a query object,
> something like this i believe...
>
> Query q = new TermQuery(t) {
> public Similarity getSimilarity(Searcher s) {
> return new SimilarityDelegator
> (TermQuery.this.super.getSimilarity(s)) {
> public float tf(freq) {
> ...
> }
> }
> }
> }
> }
>
> ...but good lord if that isn't a pain.
Well, let's toss aside backwards-compatibility concerns for the
purposes of discussion, and see what it would take to make tf()
change per-Field.
Adding a fieldName argument to similarity.tf(freq) would add
significant overhead, since it gets called a *lot*.
To avoid that, my first thought is that you'd need to supply a
different Similarity object for each field, by adding a fieldName
argument to searcher.getSimilarity(). I doubt this would work,
because Lucene's freq/prox files -- unlike it's norms -- are
consolidated, with terms from multiple fields in one file. It would
be hard for the Scorer to know what field it was operating on.
My gut is telling me that this is another reason to consolidate freq,
prox, and norm/boost into a single stream.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: SweetSpotSimiliarity
Posted by Chris Hostetter <ho...@fucit.org>.
: Presumably you had this in the can, and didn't just implement it
: today. :) For those of you who didn't see this afternoon's thread
correct, I've been using it for a few months, and ment to contribute it
last week .. but i forgot until today's discusion of customizing per
field.
: originally, though I didn't see that thread. Earlier discussion at
: http://xrl.us/mpkp (Link to mail-archives.apache.org). Mark's nifty
: graph is still up (linked from his email).
Wow ... i was arround for that thread, and looking at it now, i remember
the graph -- but at the time i was on sabatical and hadn't even started
thinking about score issues (i was only worried about Filters and BitSet
intersections). Had i remembered that thread when i start looking at how
Similarity worked ~Nov2005 I would have saved myself a lot of time and
headaches.
: "stub" documents Lucene tends to favor. However, it also stopped
: excellent matches in fields which are supposed to be short -- like
: title -- from getting a good solid lift.
: The only answer seems to be to apply different lengthNorm algos to
: different fields.
or just use the same formula, but with different constants... at which
points they aren't constants, but you know what i mean.
: What uses have you found a plateau lengthNorm, Hoss?
Primarily bad data: I want fields that are not too short, not too
long ... just right.
If i get data from a source i can't trust i want to make sure fields that
are typically short are rewarded for being short, but penalized for being
trivial (one word RSS titles from the thread you mentioned are a perfect
example of what i mean)
: > 2) a baseline tf that provides a fixed value for tf's up to a
: > minimum, at which point it becomes a sqrt curve (this is used by
: > the tf(int) function.
: > 3) a hyperbolic tf function which is best explained by graphing the
: > equation. this isn't used by default, but is available for
: > subclasses to call from their own tf functions.
:
: ... and when do you use these custom tf's?
honestly, i don't rememebr if i even use the baselineTf anymore (I think i
outgrew it) but it's a simple step up from the default that comes
in handy when 2 isn't really much better then 3 ... without
requiring you to buy in to the crazy hyperbolic tf thing that i came up
with on a whim and discovered it worked pretty well for me.
it has the nice property of giving small increases as the frequency
increases a small amount, then increasing faster once you reach the point
where you think small increases are significant, and then grows slower
again once you are above the point where you think more occurances are
acctually significant.
: I tried to graph the hyperbolic function (tip for OS X users: check
: out Grapher.app, in Utilities). It looks like by default, everything
: cancels out it returns a constant 2. But it's pretty complicated, so
: maybe I missed something.
Hmm... maybe i screwed up the defaults at some point ... i use other
values myself, gnuplot shows the defaults returning ~2 for all values
greater then 15, and ~0 for all values less then 5 and a gradient from 0
to 2 between 5 and 15. "e" probably isn't the best base.
: design though no formal IR training. Today, he wrote, "The title is
: not a discussion. It's binary; this is being considered or it
: isn't. The more words that are being considered, the less
: significant any one is, but you can't get more considered by being
: mentioned more than once in the title."
Very well put.
: I think I would implement this by having tf always return 1 for the
: title field.
Alas ... tf() doesn't take in a field name, to do this, you'd have to
override the Similarity each time your construct a query object,
something like this i believe...
Query q = new TermQuery(t) {
public Similarity getSimilarity(Searcher s) {
return new SimilarityDelegator
(TermQuery.this.super.getSimilarity(s)) {
public float tf(freq) {
...
}
}
}
}
}
...but good lord if that isn't a pain.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
SweetSpotSimiliarity
Posted by Marvin Humphrey <ma...@rectangular.com>.
[note: subject header changed from "Re: [jira] Updated: (LUCENE-577)
SweetSpotSimiliarity"]
Thought-provoking stuff, Hoss...
On May 23, 2006, at 5:55 PM, Hoss Man (JIRA) wrote:
> This is a new Similarity implimention for the contrib/
> miscellaneous/ package, it provides a Similiarty designed for
> people who know the "sweetspot" of their data. three major pieces
> of functionality are included:
> 1) a lengthNorm which creates a "plateau" of values.
Presumably you had this in the can, and didn't just implement it
today. :) For those of you who didn't see this afternoon's thread
"Per-Field Analyzer" on java-user, KinoSearch has used a plateau
lengthNorm since version 0.06...
1 / sqrt(max(100, numTerms))
... and it's been a mixed bag.
The suggestion came my way via Mark Bennett apparently from Doug
originally, though I didn't see that thread. Earlier discussion at
http://xrl.us/mpkp (Link to mail-archives.apache.org). Mark's nifty
graph is still up (linked from his email).
Making that algo the default achieved my goal: downgrade the type of
"stub" documents Lucene tends to favor. However, it also stopped
excellent matches in fields which are supposed to be short -- like
title -- from getting a good solid lift.
The only answer seems to be to apply different lengthNorm algos to
different fields.
What uses have you found a plateau lengthNorm, Hoss?
> 2) a baseline tf that provides a fixed value for tf's up to a
> minimum, at which point it becomes a sqrt curve (this is used by
> the tf(int) function.
> 3) a hyperbolic tf function which is best explained by graphing the
> equation. this isn't used by default, but is available for
> subclasses to call from their own tf functions.
... and when do you use these custom tf's?
I tried to graph the hyperbolic function (tip for OS X users: check
out Grapher.app, in Utilities). It looks like by default, everything
cancels out it returns a constant 2. But it's pretty complicated, so
maybe I missed something.
My interest in this is being driven by a really savvy client with a
formal mathematics background and a good feel for search engine
design though no formal IR training. Today, he wrote, "The title is
not a discussion. It's binary; this is being considered or it
isn't. The more words that are being considered, the less
significant any one is, but you can't get more considered by being
mentioned more than once in the title."
I think I would implement this by having tf always return 1 for the
title field.
Thought: It would be really handy if we had a benchmarking test for
IR precision.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org