You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Vagelis Kotsonis <ko...@gmail.com> on 2007/01/18 23:22:24 UTC

custom similarity based on tf but greater than 1.0

Hi all.
I am trying to make some experiments in an algorithm that scores results by
counting how many words of the query submited are in a document.

For example if i enter the query 

A B D A

The similarities I want to get for the documents follows:

A A C F D (2-found A and D)
A B D S S A (3 - found A, B and D)
D D D (1 - only found D)

I built a Similarity that actually sets everything's price as 1.0f except tf

The tf functions returns 1.0f if freq>0 and 0.0f else.

I think that this change does count what I want, but when it comes to show
the score, all are normalized. So, the greater similarity is equal to 1.0f
and the others are lower than 1.0f

How can I "deactivate" the score normalization?

Thank you!

I want to 
-- 
View this message in context: http://www.nabble.com/custom-similarity-based-on-tf-but-greater-than-1.0-tf3037071.html#a8440390
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: custom similarity based on tf but greater than 1.0

Posted by Vagelis Kotsonis <ko...@gmail.com>.
Even if I get what a I want using the coord method, I would still have the
same problem becuase the similarity would return a number > 1 and
afterwards, the scoring mechanisms would normilize these number to something
<1.0

Thank you!
Vagelis


Otis Gospodnetic wrote:
> 
> Jumping in at this point and not having read other responses, I think the
> function that Vangelis is looking for is coord method in Similarity -
> that's for document terms/query terms overlap, I believe.
> 
> Otis
> 
> ----- Original Message ----
> From: Mark Miller <ma...@gmail.com>
> To: java-user@lucene.apache.org
> Sent: Thursday, January 18, 2007 5:36:21 PM
> Subject: Re: custom similarity based on tf but greater than 1.0
> 
> I just did the same thing. If you search the list you'll find the thread 
> where Hoss gave me the info you need. It really comes down to makeing a 
> FakeNormsIndexReader. The problem you are having is a result of the 
> field size normalization.
> 
> - mark
> 
> Vagelis Kotsonis wrote:
>> Hi all.
>> I am trying to make some experiments in an algorithm that scores results
>> by
>> counting how many words of the query submited are in a document.
>>
>> For example if i enter the query 
>>
>> A B D A
>>
>> The similarities I want to get for the documents follows:
>>
>> A A C F D (2-found A and D)
>> A B D S S A (3 - found A, B and D)
>> D D D (1 - only found D)
>>
>> I built a Similarity that actually sets everything's price as 1.0f except
>> tf
>>
>> The tf functions returns 1.0f if freq>0 and 0.0f else.
>>
>> I think that this change does count what I want, but when it comes to
>> show
>> the score, all are normalized. So, the greater similarity is equal to
>> 1.0f
>> and the others are lower than 1.0f
>>
>> How can I "deactivate" the score normalization?
>>
>> Thank you!
>>
>> I want to 
>>   
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/custom-similarity-based-on-tf-but-greater-than-1.0-tf3037071.html#a8550043
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: custom similarity based on tf but greater than 1.0

Posted by Vagelis Kotsonis <ko...@gmail.com>.
It is 4 in the morning here in Greece, so I will try it tomorrow...sometime I
must sleep!
I will come up with the results tomorrow.

Thanks!
Vagelis


markrmiller wrote:
> 
> Ahhhh...I brushed over your example too fast...looked like normal 
> counting to me...I see now what you mean. So OMIT_NORMS probably did 
> work. Are you getting the results through hits? Hits will normalize. Use 
> topdocs or a hitcollector.
> 
> - Mark
> 
> 

-- 
View this message in context: http://www.nabble.com/custom-similarity-based-on-tf-but-greater-than-1.0-tf3037071.html#a8442944
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: custom similarity based on tf but greater than 1.0

Posted by Vagelis Kotsonis <ko...@gmail.com>.
So the normalization was made through Hits. That was something I didn't
understand. 
If I was alone I would search in Scorer and query classes. 

Thank you for this.

Finally I used the following:

final HitQueue hq = new HitQueue(results.length());
            searcher.search(qr, new HitCollector() {
                public void collect(int doc, float score) {
                    hq.insert(new ScoreDoc(doc, score));
                }
            });
            ScoreDoc[] scoreDocs = new ScoreDoc[hq.size()];
            for (int i = hq.size()-1; i >= 0; i--)	  // put docs in array
                scoreDocs[i] = (ScoreDoc)hq.pop();

(the HitQueue extend PriorityQueue of lucene library)

I didn't understand how to use the TopDocs, so I followed the above example
I found in the following link:
http://www.devdaily.com/java/jwarehouse/lucene-1.3-final/src/java/org/apache/lucene/search/IndexSearcher.java.shtml
http://www.devdaily.com/java/jwarehouse/lucene-1.3-final/src/java/org/apache/lucene/search/IndexSearcher.java.shtml 

The only problem I have is the case when the HitsQueue size must be
predefined...I don't know what to do then.
Currently I submit the same query twice, one for getting the size of the
results and one to use with the above code.

Thank you very much for your help!
Vagelis


markrmiller wrote:
> 
> Ahhhh...I brushed over your example too fast...looked like normal 
> counting to me...I see now what you mean. So OMIT_NORMS probably did 
> work. Are you getting the results through hits? Hits will normalize. Use 
> topdocs or a hitcollector.
> 
> - Mark
> 
> Vagelis Kotsonis wrote:
>> But i don't want to get the frequency of each term in the doc.
>>
>> what I want is 1 if the term exists in the doc and 0 if it doesn't. After
>> this, I want all thes 1s and 0s to be summed and give me a number to use
>> as
>> a score.
>>
>> If I set the TF value as 1 or 0, as I described above, I get the right
>> number, but this number is normalized to 1.0 and smaller numbers.
>>
>> It is the normalization that I want to avoid.
>>
>> Thanks again!
>> Vagelis
>>
>>
>> markrmiller wrote:
>>   
>>> Dont return 1 for tf...just return the tf straight with no 
>>> changes...return freq. For everything else return 1. After that 
>>> OMIT_NORMS should work. If you want to try a custom reader:
>>>
>>> public class FakeNormsIndexReader extends FilterIndexReader {
>>>     byte[] ones = SegmentReader.createFakeNorms(maxDoc());
>>>
>>>     public FakeNormsIndexReader(IndexReader in) {
>>>         super(in);
>>>     }
>>>     public synchronized byte[] norms(String field) throws IOException {
>>>           System.out.println("returning fake norms...");
>>>         return ones;
>>>     }
>>>
>>>     public synchronized void norms(String field, byte[] result, int 
>>> offset) {
>>>           System.out.println("writing fake norms...");
>>>         System.arraycopy(ones, 0, result, offset, maxDoc());
>>>     }
>>> }
>>>
>>> The beauty of this reader is that you can flip between it and your 
>>> custom similarity and Lucene's default implementations live on the same 
>>> index.
>>>
>>> - Mark
>>>
>>>
>>>     
>>
>>   
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/custom-similarity-based-on-tf-but-greater-than-1.0-tf3037071.html#a8550786
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: custom similarity based on tf but greater than 1.0

Posted by Mark Miller <ma...@gmail.com>.
Ahhhh...I brushed over your example too fast...looked like normal 
counting to me...I see now what you mean. So OMIT_NORMS probably did 
work. Are you getting the results through hits? Hits will normalize. Use 
topdocs or a hitcollector.

- Mark

Vagelis Kotsonis wrote:
> But i don't want to get the frequency of each term in the doc.
>
> what I want is 1 if the term exists in the doc and 0 if it doesn't. After
> this, I want all thes 1s and 0s to be summed and give me a number to use as
> a score.
>
> If I set the TF value as 1 or 0, as I described above, I get the right
> number, but this number is normalized to 1.0 and smaller numbers.
>
> It is the normalization that I want to avoid.
>
> Thanks again!
> Vagelis
>
>
> markrmiller wrote:
>   
>> Dont return 1 for tf...just return the tf straight with no 
>> changes...return freq. For everything else return 1. After that 
>> OMIT_NORMS should work. If you want to try a custom reader:
>>
>> public class FakeNormsIndexReader extends FilterIndexReader {
>>     byte[] ones = SegmentReader.createFakeNorms(maxDoc());
>>
>>     public FakeNormsIndexReader(IndexReader in) {
>>         super(in);
>>     }
>>     public synchronized byte[] norms(String field) throws IOException {
>>           System.out.println("returning fake norms...");
>>         return ones;
>>     }
>>
>>     public synchronized void norms(String field, byte[] result, int 
>> offset) {
>>           System.out.println("writing fake norms...");
>>         System.arraycopy(ones, 0, result, offset, maxDoc());
>>     }
>> }
>>
>> The beauty of this reader is that you can flip between it and your 
>> custom similarity and Lucene's default implementations live on the same 
>> index.
>>
>> - Mark
>>
>>
>>     
>
>   

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: custom similarity based on tf but greater than 1.0

Posted by Vagelis Kotsonis <ko...@gmail.com>.
But i don't want to get the frequency of each term in the doc.

what I want is 1 if the term exists in the doc and 0 if it doesn't. After
this, I want all thes 1s and 0s to be summed and give me a number to use as
a score.

If I set the TF value as 1 or 0, as I described above, I get the right
number, but this number is normalized to 1.0 and smaller numbers.

It is the normalization that I want to avoid.

Thanks again!
Vagelis


markrmiller wrote:
> 
> Dont return 1 for tf...just return the tf straight with no 
> changes...return freq. For everything else return 1. After that 
> OMIT_NORMS should work. If you want to try a custom reader:
> 
> public class FakeNormsIndexReader extends FilterIndexReader {
>     byte[] ones = SegmentReader.createFakeNorms(maxDoc());
> 
>     public FakeNormsIndexReader(IndexReader in) {
>         super(in);
>     }
>     public synchronized byte[] norms(String field) throws IOException {
>           System.out.println("returning fake norms...");
>         return ones;
>     }
> 
>     public synchronized void norms(String field, byte[] result, int 
> offset) {
>           System.out.println("writing fake norms...");
>         System.arraycopy(ones, 0, result, offset, maxDoc());
>     }
> }
> 
> The beauty of this reader is that you can flip between it and your 
> custom similarity and Lucene's default implementations live on the same 
> index.
> 
> - Mark
> 
> 

-- 
View this message in context: http://www.nabble.com/custom-similarity-based-on-tf-but-greater-than-1.0-tf3037071.html#a8442711
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: custom similarity based on tf but greater than 1.0

Posted by Mark Miller <ma...@gmail.com>.
Dont return 1 for tf...just return the tf straight with no 
changes...return freq. For everything else return 1. After that 
OMIT_NORMS should work. If you want to try a custom reader:

public class FakeNormsIndexReader extends FilterIndexReader {
    byte[] ones = SegmentReader.createFakeNorms(maxDoc());

    public FakeNormsIndexReader(IndexReader in) {
        super(in);
    }
    public synchronized byte[] norms(String field) throws IOException {
          System.out.println("returning fake norms...");
        return ones;
    }

    public synchronized void norms(String field, byte[] result, int 
offset) {
          System.out.println("writing fake norms...");
        System.arraycopy(ones, 0, result, offset, maxDoc());
    }
}

The beauty of this reader is that you can flip between it and your 
custom similarity and Lucene's default implementations live on the same 
index.

- Mark


Vagelis Kotsonis wrote:
> I feel kind of stupid...I don't get what hossman says in his post.
>
> I got the thing abou the OMMIT_NORMS and I tried to do it by calling
> Field.setOmitNorms(true); before adding a field in the index. After that I
> re-indexed my collection but still not making any difference.
>
> Tell me if I got it right. The second solution that you followed is building
> a custom FilteredIndexReader and implement these 2 functions :
>
> byte[] norms(String field)
> void norms(String field, byte[] result, int offset)
>
> Did I  get it right?
>
> Thank you and excuse me for continuously asking the same thing.
> Vagelis
>
>
> markrmiller wrote:
>   
>> Sorry your having trouble find it! Allow me...bingo: 
>> http://www.gossamer-threads.com/lists/lucene/java-user/43251?search_string=sorting%20by%20per%20doc%20hit;#43251
>>
>> Prob doesn't have great keyword for finding it. That should get you 
>> going though. Let me know if you have any questions.
>>
>> - Mark
>>
>>
>>     
>
>   

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: custom similarity based on tf but greater than 1.0

Posted by Vagelis Kotsonis <ko...@gmail.com>.
I feel kind of stupid...I don't get what hossman says in his post.

I got the thing abou the OMMIT_NORMS and I tried to do it by calling
Field.setOmitNorms(true); before adding a field in the index. After that I
re-indexed my collection but still not making any difference.

Tell me if I got it right. The second solution that you followed is building
a custom FilteredIndexReader and implement these 2 functions :

byte[] norms(String field)
void norms(String field, byte[] result, int offset)

Did I  get it right?

Thank you and excuse me for continuously asking the same thing.
Vagelis


markrmiller wrote:
> 
> Sorry your having trouble find it! Allow me...bingo: 
> http://www.gossamer-threads.com/lists/lucene/java-user/43251?search_string=sorting%20by%20per%20doc%20hit;#43251
> 
> Prob doesn't have great keyword for finding it. That should get you 
> going though. Let me know if you have any questions.
> 
> - Mark
> 
> 

-- 
View this message in context: http://www.nabble.com/custom-similarity-based-on-tf-but-greater-than-1.0-tf3037071.html#a8442395
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: custom similarity based on tf but greater than 1.0

Posted by Mark Miller <ma...@gmail.com>.
Sorry your having trouble find it! Allow me...bingo: 
http://www.gossamer-threads.com/lists/lucene/java-user/43251?search_string=sorting%20by%20per%20doc%20hit;#43251

Prob doesn't have great keyword for finding it. That should get you 
going though. Let me know if you have any questions.

- Mark

Vagelis Kotsonis wrote:
> Before I make this questions I have been looking the list for over 2 hours
> and I didn't find something to make me understand how to do what I want.
>
> After you sent the message I made a quick pass through all your messages,
> but I didn't find something. I also searched for FakeNormsIndexReader and
> still didn't find something.
>
> Can you plz tell me, if you remember, how you did it? 
>
> Thank you!
> Vagelis
>
>
> markrmiller wrote:
>   
>> I just did the same thing. If you search the list you'll find the thread 
>> where Hoss gave me the info you need. It really comes down to makeing a 
>> FakeNormsIndexReader. The problem you are having is a result of the 
>> field size normalization.
>>
>> - mark
>>
>>     

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: custom similarity based on tf but greater than 1.0

Posted by Vagelis Kotsonis <ko...@gmail.com>.
Before I make this questions I have been looking the list for over 2 hours
and I didn't find something to make me understand how to do what I want.

After you sent the message I made a quick pass through all your messages,
but I didn't find something. I also searched for FakeNormsIndexReader and
still didn't find something.

Can you plz tell me, if you remember, how you did it? 

Thank you!
Vagelis


markrmiller wrote:
> 
> I just did the same thing. If you search the list you'll find the thread 
> where Hoss gave me the info you need. It really comes down to makeing a 
> FakeNormsIndexReader. The problem you are having is a result of the 
> field size normalization.
> 
> - mark
> 
-- 
View this message in context: http://www.nabble.com/custom-similarity-based-on-tf-but-greater-than-1.0-tf3037071.html#a8441331
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: custom similarity based on tf but greater than 1.0

Posted by Mark Miller <ma...@gmail.com>.
I just did the same thing. If you search the list you'll find the thread 
where Hoss gave me the info you need. It really comes down to makeing a 
FakeNormsIndexReader. The problem you are having is a result of the 
field size normalization.

- mark

Vagelis Kotsonis wrote:
> Hi all.
> I am trying to make some experiments in an algorithm that scores results by
> counting how many words of the query submited are in a document.
>
> For example if i enter the query 
>
> A B D A
>
> The similarities I want to get for the documents follows:
>
> A A C F D (2-found A and D)
> A B D S S A (3 - found A, B and D)
> D D D (1 - only found D)
>
> I built a Similarity that actually sets everything's price as 1.0f except tf
>
> The tf functions returns 1.0f if freq>0 and 0.0f else.
>
> I think that this change does count what I want, but when it comes to show
> the score, all are normalized. So, the greater similarity is equal to 1.0f
> and the others are lower than 1.0f
>
> How can I "deactivate" the score normalization?
>
> Thank you!
>
> I want to 
>   

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org