You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Mark Miller <ma...@gmail.com> on 2006/12/16 18:07:55 UTC

sorting by per doc hit count

I have not really looked into this yet, but maybe you can save me some time
-- Is it feasible/simple to sort by the number of hits found per document?
Would this require changing the scoring system (remove idf etc etc) and
doing a normal relevancy search? Could it be done with functionquery? Any
Hints? If it is a lot of work I am not interested in doing it, but if it is
somewhat simple it would make a few customers feel fuzzy.

Thanks,

Mark

Re: sorting by per doc hit count

Posted by Mark Miller <ma...@gmail.com>.
Thanks for the tip Doron,

What if I replace the decode static method in Similiarity so that it 
returns 1 always for the HitPerDocSimiliarity? This would not require a 
re-index right?

Doron Cohen wrote:
> "Mark Miller" <ma...@gmail.com> wrote on 19/12/2006 09:21:00:
>   
>> LIA mentioned something about needing to rebuild the
>> index if you change Similarity's. That does not make
>> sense to me yet. It would seem you could alternate them.
>> What does scoring have to do with indexing?
>>     
>
> For this part of your question, the length norms of fields are computed at
> indexing time and stored in the index (as field norms). If your modified
> Similarity changes the length norm computation, re-indexing is required.
> See
> http://lucene.apache.org/java/docs/api/org/apache/lucene/search/Similarity.html#formula_norm
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>   

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: sorting by per doc hit count

Posted by Chris Hostetter <ho...@fucit.org>.
: this thread that Hoss's solution was perfect and I indeed was able to add a
: new dynamically changeable Term frequency relevance scoring system. The

Cool ... "PINE is my IDE."




-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: sorting by per doc hit count

Posted by Mark Miller <ma...@gmail.com>.
Mr. Hostetter, you are a Godsend. Just wanted to report to anyone following
this thread that Hoss's solution was perfect and I indeed was able to add a
new dynamically changeable Term frequency relevance scoring system. The
value of such a thing may not be high, but man do I love Lucene for making
it all possible.

Thanks Hoss.

- Mark

On 12/20/06, Chris Hostetter <ho...@fucit.org> wrote:
>
> :
> : problem reamins that I would like to be able to switch between the hits
> : per doc Similarity and the default Similarity on  any given search. I
> : was hoping that I could index with DefaultSimilarity and store the norms
> : for normal relevancy searching. Then I would need to ignore or make
> : constant the norms with a custom Similarity for hits per doc searching.
> : It would seem that this is not as easy as I had hoped it might be.
>
> not quite ... but it's not super tough.  if "reader" is what you use to
> construct your IndexSearcher when you wnat norms, something like this
> should work when you don't want lengthNorms...
>
>    IndexReader noNormsReader = new FilteredIndexReader(reader) {
>      byte[] ones = SegmentReader.createFakeNorms(maxDoc());
>      public synchronized byte[] norms(String field) throws IOException {
>        return ones;
>      }
>      public synchronized void norms(String field, byte[] result, int
> offset) {
>        System.arraycopy(ones, 0, result, offset, maxDoc());
>      }
>    }
>
> it would be nice if encodeNorm, decodeNorm, and getNormDecoder weren't
> static, so you could not only use whatever encoding you want at index
> time, but could also ignore it alltogether at search time in the way you
> describe ... but alas, that's just not the way it works -- changing it now
> in a backwards compatible way would be a bit tricky.
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: sorting by per doc hit count

Posted by Chris Hostetter <ho...@fucit.org>.
:
: problem reamins that I would like to be able to switch between the hits
: per doc Similarity and the default Similarity on  any given search. I
: was hoping that I could index with DefaultSimilarity and store the norms
: for normal relevancy searching. Then I would need to ignore or make
: constant the norms with a custom Similarity for hits per doc searching.
: It would seem that this is not as easy as I had hoped it might be.

not quite ... but it's not super tough.  if "reader" is what you use to
construct your IndexSearcher when you wnat norms, something like this
should work when you don't want lengthNorms...

   IndexReader noNormsReader = new FilteredIndexReader(reader) {
     byte[] ones = SegmentReader.createFakeNorms(maxDoc());
     public synchronized byte[] norms(String field) throws IOException {
       return ones;
     }
     public synchronized void norms(String field, byte[] result, int offset) {
       System.arraycopy(ones, 0, result, offset, maxDoc());
     }
   }

it would be nice if encodeNorm, decodeNorm, and getNormDecoder weren't
static, so you could not only use whatever encoding you want at index
time, but could also ignore it alltogether at search time in the way you
describe ... but alas, that's just not the way it works -- changing it now
in a backwards compatible way would be a bit tricky.



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: sorting by per doc hit count

Posted by Mark Miller <ma...@gmail.com>.
I appreciate your help Hoss. That has cleared up some things for me. The 
problem reamins that I would like to be able to switch between the hits 
per doc Similarity and the default Similarity on  any given search. I 
was hoping that I could index with DefaultSimilarity and store the norms 
for normal relevancy searching. Then I would need to ignore or make 
constant the norms with a custom Similarity for hits per doc searching. 
It would seem that this is not as easy as I had hoped it might be.

- Mark

Chris Hostetter wrote:
> : Foolish me...override a static method...silly silly. Still, I think
> : there must be some way. I don't care about the field
> : normalization...there must be some way to make it return a constant 1
> : when using a new Similarity class.
>
> as discussed: norms are a value explicitly stored in your index, so the
> easiest way to ensure that they don't factor into the score is to use
> OMIT_NORMS (or use your custom SImilarity with your customlengthNorm
> function) when indexing.
>
> once your index is built, the only way to ignore norms is with a custom
> IndexReader (it's the "IndexReader.norms" method you need to worry about)
>
> fortunately FilterIndexReader solves a lot of the busy work so you can
> focus on just the norms method.
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>   

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: sorting by per doc hit count

Posted by Chris Hostetter <ho...@fucit.org>.
: Foolish me...override a static method...silly silly. Still, I think
: there must be some way. I don't care about the field
: normalization...there must be some way to make it return a constant 1
: when using a new Similarity class.

as discussed: norms are a value explicitly stored in your index, so the
easiest way to ensure that they don't factor into the score is to use
OMIT_NORMS (or use your custom SImilarity with your customlengthNorm
function) when indexing.

once your index is built, the only way to ignore norms is with a custom
IndexReader (it's the "IndexReader.norms" method you need to worry about)

fortunately FilterIndexReader solves a lot of the busy work so you can
focus on just the norms method.



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: sorting by per doc hit count

Posted by Mark Miller <ma...@gmail.com>.
Foolish me...override a static method...silly silly. Still, I think 
there must be some way. I don't care about the field 
normalization...there must be some way to make it return a constant 1 
when using a new Similarity class.

Doron Cohen wrote:
> "Mark Miller" <ma...@gmail.com> wrote on 19/12/2006 09:21:00:
>   
>> LIA mentioned something about needing to rebuild the
>> index if you change Similarity's. That does not make
>> sense to me yet. It would seem you could alternate them.
>> What does scoring have to do with indexing?
>>     
>
> For this part of your question, the length norms of fields are computed at
> indexing time and stored in the index (as field norms). If your modified
> Similarity changes the length norm computation, re-indexing is required.
> See
> http://lucene.apache.org/java/docs/api/org/apache/lucene/search/Similarity.html#formula_norm
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>   

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: sorting by per doc hit count

Posted by Doron Cohen <DO...@il.ibm.com>.
"Mark Miller" <ma...@gmail.com> wrote on 19/12/2006 09:21:00:
> LIA mentioned something about needing to rebuild the
> index if you change Similarity's. That does not make
> sense to me yet. It would seem you could alternate them.
> What does scoring have to do with indexing?

For this part of your question, the length norms of fields are computed at
indexing time and stored in the index (as field norms). If your modified
Similarity changes the length norm computation, re-indexing is required.
See
http://lucene.apache.org/java/docs/api/org/apache/lucene/search/Similarity.html#formula_norm


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: sorting by per doc hit count

Posted by Mark Miller <ma...@gmail.com>.
Could I use another Similarity that returned 1 for most of the scoring terms
and the actual term frequency (rig the equation)? Could I then alternate the
DefaultSimilarity and HitsPerDocSimilairty per search? LIA mentioned
something about needing to rebuild the index if you change Similarity's.
That does not make sense to me yet. It would seem you could alternate them.
What does scoring have to do with indexing?

Thanks,

Mark

On 12/16/06, Mark Miller <ma...@gmail.com> wrote:
>
> I have not really looked into this yet, but maybe you can save me some
> time -- Is it feasible/simple to sort by the number of hits found per
> document? Would this require changing the scoring system (remove idf etc
> etc) and doing a normal relevancy search? Could it be done with
> functionquery? Any Hints? If it is a lot of work I am not interested in
> doing it, but if it is somewhat simple it would make a few customers feel
> fuzzy.
>
> Thanks,
>
> Mark
>

Re: sorting by per doc hit count

Posted by Erick Erickson <er...@gmail.com>.
Well, if you're not interested in doing much in the way of complex queries,
you could use TermDocs/TermEnum (particularly look at TermDocs) to count the
number of times a term appears in each document. I think you'll be surprised
at how quickly you can get this info.

Making your own scorer seems like a reasonable approach also, you could just
return the term frequency (see TermDocs) although I admit I'm staying away
from messing with the relevancy scorers so I"m not speaking from experience.
Others who know more are going to have to weigh in on functionquery.

Warning: I've just been in some (non Lucene) code that tried to do it's own
arbitrarily complex boolean logic by counting term frequency. Don't go there
if you want to keep your work minimal. In fact, I'd recommend against going
there at all <G>. If you can restrict the allowed syntax to be simple AND
you'd be all set (simple OR would be ok too). I suspect that as soon as you
start even combining the two allowing grouping, the effort increases
dramatically.

Which probably argues for making your own scorer that just deals with
frequency.

Best
Erick

On 12/16/06, Mark Miller < markrmiller@gmail.com> wrote:
>
> I have not really looked into this yet, but maybe you can save me some
> time
> -- Is it feasible/simple to sort by the number of hits found per document?
> Would this require changing the scoring system (remove idf etc etc) and
> doing a normal relevancy search? Could it be done with functionquery? Any
> Hints? If it is a lot of work I am not interested in doing it, but if it
> is
> somewhat simple it would make a few customers feel fuzzy.
>
> Thanks,
>
> Mark
>
>