You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by calin014 <ca...@gmail.com> on 2010/05/26 11:32:11 UTC
Precision-recall curve with /contrib/benchmark/quality
Hi,
I am testing some ranking methods with /contrib/benchmark/quality package,
and i was wondering if there is a simple way of building a precision-recall
graph with the info gathered(maybe QualityStats.RecallPoint).
Thanks
--
View this message in context: http://lucene.472066.n3.nabble.com/Precision-recall-curve-with-contrib-benchmark-quality-tp844442p844442.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Precision-recall curve with /contrib/benchmark/quality
Posted by calin014 <ca...@gmail.com>.
Thanks a lot for your quick answer.
I modified the idf formula as you suggested, and got a small improvement.
Then I added coord() to scoring and the boost was pretty big :)
http://lucene.472066.n3.nabble.com/file/n873240/prcurve_bm25.png
These tests are done on the ohsumed train corpus(i used coord on f2exp also)
http://lucene.472066.n3.nabble.com/file/n873240/prcurve.png
--
View this message in context: http://lucene.472066.n3.nabble.com/Precision-recall-curve-with-contrib-benchmark-quality-tp844442p873240.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Precision-recall curve with /contrib/benchmark/quality
Posted by Robert Muir <rc...@gmail.com>.
I experienced this problem too, when implementing a prototype of this
similarity for LUCENE-2392, i noticed it consistently provided better
relevance than this patch. (you can see the prototype, i uploaded it to
LUCENE-2091).
After lots of discussion and debugging with José Ramón Pérez Agüera, I found
that both implementations are the same as far as the core formula, but there
are some current limitations in the LUCENE-2091 patch that can cause it to
often perform worse than lucene's default ranking formula. Here are some
examples:
1. it has its own BooleanQuery, which doesn't support coordination-level
matching [coord()]. This feature often gives a nice relevance boost for
short queries, such as the topic queries in ohsumed and typical queries
users enter to search engines.
2. its IDF formula has no floor, causing some IDF values for very common
words to have negative values. this can be addressed by using
Math.log(*1 +*((n - dfj + 0.5F)/(dfj + 0.5F))).
3. the length normalization encoding is unchanged from lucene's default:
1/sqrt(n), but then quantized to byte encoding. but this patch then has to
undo the square root which itself already lost precision from the encoding.
I think this too causes some problems.
float fieldNorm =
this.getSimilarity().decodeNormValue(this.norm[this.docID()]);
float length = 1 / (fieldNorm * fieldNorm);
So, I think we can fix all of these issues in LUCENE-2392, where the scoring
system is modified to be more friendly to these types of similarities, and
in my tests so far BM25 performs quite well reliably with that system in
place.
Until some things are improved in lucene itself, it would be hard to adjust
the patch in LUCENE-2091 to avoid problems like #3, which is really just the
patch trying to dance around limitations of lucene's scoring API. This
one is much better with the flexible scoring system since it can run the
calculations incorporating both the raw docLength and avgDocLength before
even going to byte[] at all... or if you prefer, not even use byte[] norms
but float[] or something else :)
On Sat, Jun 5, 2010 at 8:10 AM, calin014 <ca...@gmail.com> wrote:
>
> Hy,
>
> I did some tests and i keep getting low scores for bm25 on ORP collections.
> The implementation is from this patch:
> https://issues.apache.org/jira/browse/LUCENE-2091
> https://issues.apache.org/jira/browse/LUCENE-2091
> Is that normal?
>
> I got the following results using StandardAnalyzer:
>
> http://lucene.472066.n3.nabble.com/file/n872585/prcurve1.png
> http://lucene.472066.n3.nabble.com/file/n872585/prcurve.png
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Precision-recall-curve-with-contrib-benchmark-quality-tp844442p872585.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Robert Muir
rcmuir@gmail.com
Re: Precision-recall curve with /contrib/benchmark/quality
Posted by calin014 <ca...@gmail.com>.
Hy,
I did some tests and i keep getting low scores for bm25 on ORP collections.
The implementation is from this patch:
https://issues.apache.org/jira/browse/LUCENE-2091
https://issues.apache.org/jira/browse/LUCENE-2091
Is that normal?
I got the following results using StandardAnalyzer:
http://lucene.472066.n3.nabble.com/file/n872585/prcurve1.png
http://lucene.472066.n3.nabble.com/file/n872585/prcurve.png
--
View this message in context: http://lucene.472066.n3.nabble.com/Precision-recall-curve-with-contrib-benchmark-quality-tp844442p872585.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Precision-recall curve with /contrib/benchmark/quality
Posted by calin014 <ca...@gmail.com>.
Thanks, that worked:)
--
View this message in context: http://lucene.472066.n3.nabble.com/Precision-recall-curve-with-contrib-benchmark-quality-tp844442p845518.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Precision-recall curve with /contrib/benchmark/quality
Posted by Robert Muir <rc...@gmail.com>.
Personally I use the generated submission.txt, and run it thru
trec_eval to get all numbers.
by default, trec_eval will dump ircl_prn.xxxx values, and you could
plug them in openoffice.
I prefer to use trec_eval as the results from the benchmark summary
often differ with trec_eval.
it would be great if somehow this process could be improved by default.
Nicola Ferro mentioned on the list once that for the CLEF work, they
have written a JNI wrapper around trec_eval to access all its
computations, this might be one thing we could consider, then we could
access the correct calculations and produce fancy output.
On Wed, May 26, 2010 at 5:32 AM, calin014 <ca...@gmail.com> wrote:
>
> Hi,
>
> I am testing some ranking methods with /contrib/benchmark/quality package,
> and i was wondering if there is a simple way of building a precision-recall
> graph with the info gathered(maybe QualityStats.RecallPoint).
>
> Thanks
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Precision-recall-curve-with-contrib-benchmark-quality-tp844442p844442.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Robert Muir
rcmuir@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org