You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Marvin Humphrey <ma...@rectangular.com> on 2006/05/23 23:29:12 UTC
Per-Field Similarity
Greets,
Is it possible to have an IndexWriter apply different Similarity
models to different Fields?
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Per-Field Similarity
Posted by karl wettin <ka...@snigel.net>.
On Tue, 2006-05-23 at 14:49 -0700, Chris Hostetter wrote:
> i've definitely wished more then once that they took in a field name
> as a parameter.
+1 for starting a branch with non-depricated radically reconstructed
fields after release of 2.0.
I'd be happy to document all design discussions with UML. If it works
well in theory someone is bound to type in the code. Maybe even I.
I'm sure it will find its way back to the trunk if it is good enough.
And maybe some Scorer-people should join in to glance at a more
pluggable per-search scoring strategy without all the hardwired code.
Have a look at per-term position boosting, et.c.
That would be really great.
Anyone with six months of vacation?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Per-Field Similarity
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Tue, May 23, 2006 at 02:49:40PM -0700, Chris Hostetter wrote:
>
> : Is it possible to have an IndexWriter apply different Similarity
> : models to different Fields?
>
> As far as i know, the only way Similarity comes into play when using an
> IndexWriter is lengthNorm, and it is passed the fieldName so it's easy to
> make it's behavior field specific (SimilarityDelegator makes it easy to do
> this)
Ah. KinoSearch's implementation inadvertently differs: lengthNorm takes only
the number of tokens. It didn't occur to me that the problem had been solved
that way.
A while ago, I changed lengthNorm to use either the number of tokens or 100,
whichever was greater. That had the intended effect, which was to downgrade
the type of "stub" documents Lucene tends to boost. However, it had the
undesirable effect of no longer boosting exact title matches.
The answer is to use Lucene's default lengthNorm for title and the modified
lengthNorm for bodytext. Wasn't sure how to do that in Lucene; now I know.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Per-Field Similarity
Posted by Chris Hostetter <ho...@fucit.org>.
: Is it possible to have an IndexWriter apply different Similarity
: models to different Fields?
As far as i know, the only way Similarity comes into play when using an
IndexWriter is lengthNorm, and it is passed the fieldName so it's easy to
make it's behavior field specific (SimilarityDelegator makes it easy to do
this)
At query time things like coord and queryNorm don't erreally make sense as
field specific methods; the idf method on the other hand takes in a Term
(or a Collection of Terms) so it's also easy to customize.
tf and sloppyFreq however are somewhat frustrating ... i've definitely
wished more then once that they took in a field name as a parameter.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Per-Field Similarity
Posted by karl wettin <ka...@snigel.net>.
On Tue, 2006-05-23 at 15:03 -0700, Chris Hostetter wrote:
>
> Why wouldn't you just provide your own Similarity instance that looked
> at the fieldName passed to the lengthNorm method?
Perhaps one reason could be.. hmm. that it would make it one similarity
per field and IndexWriter and he really want per document and field.
But really, I'm just reaching out to save my face.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Per-Field Similarity
Posted by Chris Hostetter <ho...@fucit.org>.
: Refactor :
:
: class DocumentWriter {
: private final void writeNorms(String segment) throws IOException {
: for(int n = 0; n < fieldInfos.size(); n++){
: FieldInfo fi = fieldInfos.fieldInfo(n);
: if(fi.isIndexed && !fi.omitNorms){
: float norm = fieldBoosts[n] * similarity.lengthNorm(fi.name,
:
: to this:
: float norm = fieldBoosts[n] * fi.similarity.lengthNorm(fi.name,
Why wouldn't you just provide your own Similarity instance that looked at
the fieldName passed to the lengthNorm method?
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Per-Field Similarity
Posted by karl wettin <ka...@snigel.net>.
On Tue, 2006-05-23 at 14:29 -0700, Marvin Humphrey wrote:
> Greets,
>
> Is it possible to have an IndexWriter apply different Similarity
> models to different Fields?
You only want to apply this to the norms? Are up for an ad-hoc solution?
Refactor :
class DocumentWriter {
private final void writeNorms(String segment) throws IOException {
for(int n = 0; n < fieldInfos.size(); n++){
FieldInfo fi = fieldInfos.fieldInfo(n);
if(fi.isIndexed && !fi.omitNorms){
float norm = fieldBoosts[n] * similarity.lengthNorm(fi.name,
to this:
float norm = fieldBoosts[n] * fi.similarity.lengthNorm(fi.name,
You'll have to hack the similarity in FieldInfos.add(Document) and Field
or Document. If I was to add more than one Field with the same name in a
Document I'd probably hack it in the Document.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org