You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Melissa Mifsud <me...@yahoo.com> on 2002/04/04 20:13:19 UTC
Document Scoring
Hi,
I've been going through Lucene's code for days now and I'm puzzled as to how the scoring works.
The scoring is ultimately performed by the Scorer which in turn uses the Similarity class.
The Similiarity class needs the docFreq(Term t) amd maxDoc() methods.
It calls these methods through the Searcher which is passed [Similarity.idf(Term t, Searcher searcher)].
Now, the class IndexSearcher, which extends Searcher, refers back to the IndexReader. Here is the puzzling part:
The method docFreq(Term t) and the method maxDoc() are both declared abstract in IndexReader.
Faced with this question, I was obliged to check out how it was done in the demo that comes with Lucene. But the demo still uses IndexSearcher which in turn uses IndexReader to call these methods.
Can anyone please shed some light here?
Melissa
Re: QueryParser for default AND
Posted by Melissa Mifsud <me...@yahoo.com>.
Can we please verify:
new mode: +a +b +c +d
just mean that you have to pre-append "+" to each term in the query string?
Query String = World Wide Web
AND Query String = +World +Wide +Web
So that the AND query string means that you would like to search all those
documents in which ALL the terms in the query appear?
Melissa.
----- Original Message -----
From: "Halácsy Péter" <ha...@axelero.hu>
To: "Lucene Developer" <lu...@jakarta.apache.org>
Sent: Saturday, April 06, 2002 8:40 PM
Subject: QueryParser for default AND
> 19 Febr. Doug wrote:
>
> >> How could I modify the queryParser to implement
> >> default AND logic?
>
> >I haven't tested this, but it should be as simple as changing line 318 of
> >QueryParser.jj to:
> > int ret = MOD_REQ;
>
> >Unfortunately, I think this would end up disabling OR, so the proper
change
> >is more complex, requiring some changes to the addClause() method. This
> >should also be something that folks can turn on and off.
>
> >Doug
>
> I attached a modified QueryParser.jj. Finally I modified addClause(). I
also
> attached a tester class.
>
> Use:
> QueryParser qp = new QueryParser(defaultField, analyzer);
> qp.setMode(QueryParser.MODE_DEFAULT_END);
>
> Please try & test it.
>
> Some results:
> Query: a b c d
> classic mode: a b c d
> new mode: +a +b +c +d
>
> Query: a AND NOT b
> classic mode: +a -b
> new mode: +a -b
>
> Query: a OR b
> classic mode: a b
> new mode: a b
>
> Query: a b c -d
> classic mode: a b c -d
> new mode: +a +b +c -d
>
> Query: Capital of Hungary
> classic mode: capital of hungary
> new mode: +capital +of +hungary
>
>
> Peter
>
>
>
>
----------------------------------------------------------------------------
----
> --
> To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
> For additional commands, e-mail:
<ma...@jakarta.apache.org>
_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: QueryParser for default AND
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hello P�ter,
Do you think you could resend this with a context diff against the
latest QueryParser.jj?
Brian Goetz made some improvements to QueryParser.jj the other day, so
the diff doesn't apply well, and I don't want to break things.
If you can also send a diff for the tester class that would also be
nice, but if you can't it's not a big deal, I can extract your
additions from there.
[otis@linux2 jakarta-lucene]$ patch
src/java/org/apache/lucene/queryParser/QueryParser.jj
QueryParser.jj.diff
patching file src/java/org/apache/lucene/queryParser/QueryParser.jj
Reversed (or previously applied) patch detected! Assume -R? [n] n
Apply anyway? [n] y
Hunk #1 FAILED at 65.
Hunk #2 FAILED at 78.
Hunk #3 FAILED at 114.
Hunk #4 FAILED at 157.
Hunk #5 FAILED at 166.
Hunk #6 FAILED at 197.
Hunk #7 FAILED at 219.
Hunk #8 FAILED at 271.
Hunk #9 FAILED at 299.
Hunk #10 FAILED at 353.
Hunk #11 FAILED at 389.
Hunk #12 FAILED at 407.
Hunk #13 FAILED at 416.
Hunk #14 FAILED at 466.
14 out of 14 hunks FAILED -- saving rejects to file
src/java/org/apache/lucene/queryParser/QueryParser.jj.rej
Thanks,
Otis
--- Hal�csy_P�ter <ha...@axelero.hu> wrote:
> 19 Febr. Doug wrote:
>
> >> How could I modify the queryParser to implement
> >> default AND logic?
>
> >I haven't tested this, but it should be as simple as changing line
> 318 of
> >QueryParser.jj to:
> > int ret = MOD_REQ;
>
> >Unfortunately, I think this would end up disabling OR, so the proper
> change
> >is more complex, requiring some changes to the addClause() method.
> This
> >should also be something that folks can turn on and off.
>
> >Doug
>
> I attached a modified QueryParser.jj. Finally I modified addClause().
> I also
> attached a tester class.
>
> Use:
> QueryParser qp = new QueryParser(defaultField, analyzer);
> qp.setMode(QueryParser.MODE_DEFAULT_END);
>
> Please try & test it.
>
> Some results:
> Query: a b c d
> classic mode: a b c d
> new mode: +a +b +c +d
>
> Query: a AND NOT b
> classic mode: +a -b
> new mode: +a -b
>
> Query: a OR b
> classic mode: a b
> new mode: a b
>
> Query: a b c -d
> classic mode: a b c -d
> new mode: +a +b +c -d
>
> Query: Capital of Hungary
> classic mode: capital of hungary
> new mode: +capital +of +hungary
>
>
> Peter
>
>
>
>
> ATTACHMENT part 2 application/octet-stream name=QueryParser.jj
> ATTACHMENT part 3 application/octet-stream
name=QueryParserTester.java
__________________________________________________
Do You Yahoo!?
Yahoo! - Official partner of 2002 FIFA World Cup
http://fifaworldcup.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
QueryParser for default AND
Posted by Halácsy Péter <ha...@axelero.hu>.
19 Febr. Doug wrote:
>> How could I modify the queryParser to implement
>> default AND logic?
>I haven't tested this, but it should be as simple as changing line 318 of
>QueryParser.jj to:
> int ret = MOD_REQ;
>Unfortunately, I think this would end up disabling OR, so the proper change
>is more complex, requiring some changes to the addClause() method. This
>should also be something that folks can turn on and off.
>Doug
I attached a modified QueryParser.jj. Finally I modified addClause(). I also
attached a tester class.
Use:
QueryParser qp = new QueryParser(defaultField, analyzer);
qp.setMode(QueryParser.MODE_DEFAULT_END);
Please try & test it.
Some results:
Query: a b c d
classic mode: a b c d
new mode: +a +b +c +d
Query: a AND NOT b
classic mode: +a -b
new mode: +a -b
Query: a OR b
classic mode: a b
new mode: a b
Query: a b c -d
classic mode: a b c -d
new mode: +a +b +c -d
Query: Capital of Hungary
classic mode: capital of hungary
new mode: +capital +of +hungary
Peter
Re: Document Scoring
Posted by Dmitry Serebrennikov <dm...@earthlink.net>.
>
>
>
>Now, the class IndexSearcher, which extends Searcher, refers back to the IndexReader. Here is the puzzling part:
>
> The method docFreq(Term t) and the method maxDoc() are both declared abstract in IndexReader.
>
>Faced with this question, I was obliged to check out how it was done in the demo that comes with Lucene. But the demo still uses IndexSearcher which in turn uses IndexReader to call these methods.
>
There are a couple of kinds of IndexReaders that are supported. One is
called SegmentReader and the other - SegmentsReader (notice the ploral).
The first one is used to access a single index segment, whereas the
second one is used to do a search in unoptimized indexes that have more
then one segment. These are implementation classes that extend
IndexReader in different ways. Other readers could potentially be
created as well that would use implement access into some other kind of
a datastructure.
Not sure if this sheds any light onto your original question about how
the scoring works, but it should explaing the mystery of the abstract
methods. As far as the scoring, you will probably find your questions
answered by one of the FAQs. They give the answer in terms of the
searching theory that might save you time in trying to figure this out
from code. I would include the answer here, but I'm not that strong in
the theory/math of it, so the FAQ is your best bet. I just know it
works, and that happens to be good enough for me today :).
Dmitry.
>
>
>Can anyone please shed some light here?
>
>Melissa
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>