You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Paul Elschot <pa...@xs4all.nl> on 2006/05/21 14:15:45 UTC

Explaining a filter; Scorer extending Matcher; (was: BooleanWeight.normalize(float) doesn't normalize prohibited clauses?)

On Friday 12 May 2006 00:12, Chris Hostetter wrote:
> 
> : >    Boolean match = null;
> :
> : As for the thoughts question below: this java-dev, not c-dev :)
> 
...
> 
> ...i'm not trying to use null for false, i'm using null to indicate that
> wether or not a match occured has not been explicitly specified -- it can
> only be infered from the "value" of the explanation.  "true" means a
> definitive match, and "false" means a definitive non-match.
> 
> : As long as there is no match, there will be no score, and no score could
> : also be represented by NaN, so one might by default initialize the score
> : value to NaN, drop setMatch() and isMatch() above, and have only:
> :
> : public Boolean getMatch() { return ! Float.isNaN(score); }
> 
> I assume by "score" you mean "value" (Explanations don't have a score
> attribute, just a value attribute).  I don't want to go down the road of
> assuming a match based on some special value of of the value -- that's the
> cause of the current problems.  NaN is admitedly a better choice then
> "0.0", but it's still a value that could concievably come up when scoring
> a document for some as yet non-existent Scorer.
> 
> what i really want is for the Explanation class to precisely model the
> same information as a Scorer returns for each doc...
> 
>    1) if scorer.doc() would ever return X, then the Explanation for that X
>       should have a boolean indicating a "match"
>    2) whatever value is returned by scorer.score() when scorer.doc() is
>       returning X should be what the Explanation for X returns when you
>       call getValue().
> 
> : But I'm not yet sure wether that would work in all cases.
> : Is it possible/thinkable for a (sub)query to have a score value for a
> : document, but no match against the same document?
> 
> I'm not sure if it can ever exist, since you currently can't ask a Scorer
> for the score of a document unless the document matches, but the converse
> is certainly true: a document can match on a query but have a score of 0,
> or less then 0, or NaN ... that's what i'm trying to deal with, i want to
> be able to model all of those cases in an Explanation object.

Having a score value without a match not normally possible when searching
a query, but for Filter this is actually the normal case: a Filter may match a 
document, but it does not provide a score value.

> 
> : That would be avoided by having getMatch() only. Once setMatch is called,
> : getMatch would return false, except when setMatch is given a NaN, but
> : that is probably not done in the current Lucene code.
> 
> Right.  currently *most* Explanations get a 0.0 value set if it's a non
> match (some of them don't work at all for non-matches) .. which is why if
> an explicit match boolean isn't specified, I want the fall back assumption
> to be based on wether the value is 0.0 -- because that's the current
> method for determining a match, and it will work with legacy custom Query
> types people may have written which aren't in the lucene code base.
> 
> : > 4) change BooleanWeight.explain to call isMatch on the sub-explanations
> : > when testing prohibited/required clauses.
> :
> : Or call getMatch(), whichever is implemented. This makes explaining the
> 
> i want to impliment both ... getMatch() existing for people that want to
> know the exact state of the match (and will check for null to determine if
> hte exact state is unknown) and isMatch for people who want the "best
> guess" behavior ... which is not encapsulated in a method, instead of it
> just being "convention"

In case Explanation is also to explain what a Filter does, it would need to
have both a match flag and a score value.

At the moment I'm trying to implement filters by refactoring Scorer to have an
abstract superclass Matcher that could also become a superclass for filter
implementations (instead of DocNrSkipper).

This Matcher class has all methods of Scorer that are not using score
values: doc(), next(), skipTo(docNr).
An explain() method is also useful in such a Matcher, but it has
no score value available, it only knows whether or not a document matches.

The implementation also has an interface MatcherProvider (instead of
SkipFilter):
   public Matcher getMatcher(IndexReader reader) throws ioe;
Filter could implement MatcherProvider as an alternative to the
SkipFilter1.patch here:
http://issues.apache.org/jira/browse/LUCENE-328

Any thoughts on whether such a Matcher would be preferable to 
a DocNrSkipper that only has this method:
  int nextDocNr(int docNr)
?

Regards,
Paul Elschot

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Explaining a filter; Scorer extending Matcher;

Posted by Paul Elschot <pa...@xs4all.nl>.
On Tuesday 23 May 2006 20:05, Chris Hostetter wrote:
> 
> : I'll repeat the DocIterator from that thread:
> :
> :      public interface DocIterator {
> :          public int doc();
> :          public boolean next();
> :          public boolean skipTo(int target);
> :      }
> :
> : to come to my question: should that interface should also have an
> : explain(int docNr) method?
> : When so, it does more than iterating, so it might even be called Matcher.
> : Or would both DocIterator and Matcher be useful?
> 
> that's a damn fine question, and the most opinionated answer i can give,
> is "i don't know".
> 
> Setting aside for the moment discussion about refactoring a common
> interface used for both scoring and filtering, Let me raise some seemingly
> unrelated questions:
>    1) Does it make sense from an API perspective for both Weight and Scorer
>       to have explain methods (ignore migration issues that might be
>       involved in making the change naow, imagine if we were starting from
>       scratch) ?
>    2) Should Weight.explain(IndexReader,doc) allways call 
scorer(reader).explain(doc)?
>    3) Should Weight.explain be deprecated in favor of usage in #2
>    4) should Scorer.explain be removed, and Weights that want to delegate
>       to their Scorer can do so if-and-only-if the Scorer class they use
>       specificaly declares the method (as oppose to now where many Scorer
>       implimentations throw UnsupportedOperationException when they expect
>       their corrisponding Weight to take care of it
> 
> ...in my optinion, the answers to those questions should drive any
> question of wether a "Matcher" API which is intended to be
> implimented/superclass of Scorer should contain an explain method.  And
> assuming the idea of being able to "explain" Filter's appeal to people,
> then should the Filter API have an explain method ala Weight, or should
> the Matcher returned by Filter.getMatcher(IndexReader) have an explain
> method.
> 
> 
> I honestly don't know.
> 
> : I don't know whether interfaces or abstract classes would be better for 
this.
> 
> At this point ... i think many interface/abstract-class decissiosn in
> Lucene have to be based more on the migration path of least change
> then by any sense of correct design principles.

As far as I can see the smallest change would be the SkipFilter1.patch
here: http://issues.apache.org/jira/browse/LUCENE-328
This makes Filter implement SkipFilter and it lets IndexSearcher
use the SkipFilter implementation when it is available.
The SkipFilter interface is here:
http://issues.apache.org/jira/browse/LUCENE-330

The next step would then be a global replacement of Filter by
SkipFilter in all API's, together with the addition of a test for
  filter instanceof Filter
in IndexSearcher for backwards compatibility in case no
DocNrSkipper is provided as SkipFilter.

To allow adding things in DocNrSkipper (for example a method
nextDocNr() without argument), DocNrSkipper could be implemented
as an abstract class instead of as an interface.

Explaining a filter might be done by FilteredQuery only.

Regards,
Paul Elschot

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Explaining a filter; Scorer extending Matcher;

Posted by Chris Hostetter <ho...@fucit.org>.
: I'll repeat the DocIterator from that thread:
:
:      public interface DocIterator {
:          public int doc();
:          public boolean next();
:          public boolean skipTo(int target);
:      }
:
: to come to my question: should that interface should also have an
: explain(int docNr) method?
: When so, it does more than iterating, so it might even be called Matcher.
: Or would both DocIterator and Matcher be useful?

that's a damn fine question, and the most opinionated answer i can give,
is "i don't know".

Setting aside for the moment discussion about refactoring a common
interface used for both scoring and filtering, Let me raise some seemingly
unrelated questions:
   1) Does it make sense from an API perspective for both Weight and Scorer
      to have explain methods (ignore migration issues that might be
      involved in making the change naow, imagine if we were starting from
      scratch) ?
   2) Should Weight.explain(IndexReader,doc) allways call scorer(reader).explain(doc)?
   3) Should Weight.explain be deprecated in favor of usage in #2
   4) should Scorer.explain be removed, and Weights that want to delegate
      to their Scorer can do so if-and-only-if the Scorer class they use
      specificaly declares the method (as oppose to now where many Scorer
      implimentations throw UnsupportedOperationException when they expect
      their corrisponding Weight to take care of it

...in my optinion, the answers to those questions should drive any
question of wether a "Matcher" API which is intended to be
implimented/superclass of Scorer should contain an explain method.  And
assuming the idea of being able to "explain" Filter's appeal to people,
then should the Filter API have an explain method ala Weight, or should
the Matcher returned by Filter.getMatcher(IndexReader) have an explain
method.


I honestly don't know.

: I don't know whether interfaces or abstract classes would be better for this.

At this point ... i think many interface/abstract-class decissiosn in
Lucene have to be based more on the migration path of least change
then by any sense of correct design principles.



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Explaining a filter; Scorer extending Matcher;

Posted by Doug Cutting <cu...@apache.org>.
Erik Hatcher wrote:
> Since we're moving forward with API refactoring discussions, on the  
> topic of exceptions, perhaps its time to consider putting in our own  
> exception class or hierarchy to decouple ourselves a bit from "IO".   
> There are many in-memory non-disk-IO uses of these API's and having  
> IOException being caught when it doesn't make sense (like iterating  
> through tokens of a StringReader from an Analyzer, for instance).

Back-compatibility will be much simpler if new exceptions are subclasses 
of IOException.  But adding a Lucene-specific exception hierarchy would 
be good hygiene.  Does someone want to draft a proposal?

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Explaining a filter; Scorer extending Matcher;

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On May 23, 2006, at 4:26 AM, eks dev wrote:

>> "DocIterator", that way even TermDocs could impliment it ...
>
> I played with that idea and what I learned is that DocIterator  
> should throw IOException as TermDocs throws it (It has been  
> allready mentioned on compact sparse filters JIRA).

Since we're moving forward with API refactoring discussions, on the  
topic of exceptions, perhaps its time to consider putting in our own  
exception class or hierarchy to decouple ourselves a bit from "IO".   
There are many in-memory non-disk-IO uses of these API's and having  
IOException being caught when it doesn't make sense (like iterating  
through tokens of a StringReader from an Analyzer, for instance).

	Erik


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Explaining a filter; Scorer extending Matcher;

Posted by eks dev <ek...@yahoo.co.uk>.
>"DocIterator", that way even TermDocs could impliment it ...

I played with that idea and what I learned is that DocIterator should throw IOException as TermDocs throws it (It has been allready mentioned on compact sparse filters JIRA).







---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Explaining a filter; Scorer extending Matcher;

Posted by Paul Elschot <pa...@xs4all.nl>.
On Monday 22 May 2006 23:43, Chris Hostetter wrote:
> 
> : In case Explanation is also to explain what a Filter does, it would need 
to
> : have both a match flag and a score value.
> 
> that's a good point, i hadn't considered hte possibility of "explaining"
> filters much ... but there's no reason why the "valueO 'f an explanation
> couldn't be an optional part of the explanation as well (ie: migrate from
> float to Float when adding the "Boolean match" so that future Explanation
> instances (of Filters) can say they match without claiming a particular
> score (or "faking it" with some magic float value)
> 
> : At the moment I'm trying to implement filters by refactoring Scorer to 
have an
> : abstract superclass Matcher that could also become a superclass for filter
> : implementations (instead of DocNrSkipper).
> :
> : This Matcher class has all methods of Scorer that are not using score
> : values: doc(), next(), skipTo(docNr).
> 
> : Any thoughts on whether such a Matcher would be preferable to
> : a DocNrSkipper that only has this method:
> :   int nextDocNr(int docNr)
> 
> (I'm very ammused by this question, you'll see why in a second)
> 
> I am 100% on board with the "Matcher" API you are describing ... but i
> think it might make more sense as an interface, perhaps called
> "DocIterator", that way even TermDocs could impliment it ...
> 
> http://www.nabble.com/Filter-t1002705.html#a2605686
> 
> 	:)

Well, I was about to add a reference to that thread at the
issue on compact sparse filters. The current patch there is
just the result of me trying to get something in that direction
to actually work, after mostly forgetting some of the "details"
of that thread.

I'll repeat the DocIterator from that thread:

     public interface DocIterator { 
         public int doc(); 
         public boolean next(); 
         public boolean skipTo(int target); 
     } 
 
to come to my question: should that interface should also have an
explain(int docNr) method?
When so, it does more than iterating, so it might even be called Matcher.
Or would both DocIterator and Matcher be useful?

I don't know whether interfaces or abstract classes would be better for this.

Regards,
Paul Elschot


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Explaining a filter; Scorer extending Matcher; (was: BooleanWeight.normalize(float) doesn't normalize prohibited clauses?)

Posted by Chris Hostetter <ho...@fucit.org>.
: In case Explanation is also to explain what a Filter does, it would need to
: have both a match flag and a score value.

that's a good point, i hadn't considered hte possibility of "explaining"
filters much ... but there's no reason why the "valueO 'f an explanation
couldn't be an optional part of the explanation as well (ie: migrate from
float to Float when adding the "Boolean match" so that future Explanation
instances (of Filters) can say they match without claiming a particular
score (or "faking it" with some magic float value)

: At the moment I'm trying to implement filters by refactoring Scorer to have an
: abstract superclass Matcher that could also become a superclass for filter
: implementations (instead of DocNrSkipper).
:
: This Matcher class has all methods of Scorer that are not using score
: values: doc(), next(), skipTo(docNr).

: Any thoughts on whether such a Matcher would be preferable to
: a DocNrSkipper that only has this method:
:   int nextDocNr(int docNr)

(I'm very ammused by this question, you'll see why in a second)

I am 100% on board with the "Matcher" API you are describing ... but i
think it might make more sense as an interface, perhaps called
"DocIterator", that way even TermDocs could impliment it ...

http://www.nabble.com/Filter-t1002705.html#a2605686

	:)


-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Explaining a filter; Scorer extending Matcher; (was: BooleanWeight.normalize(float) doesn't normalize prohibited clauses?)

Posted by Paul Elschot <pa...@xs4all.nl>.
On Sunday 21 May 2006 21:06, eks dev wrote:
> 
> "Any thoughts on whether such a Matcher would be preferable to 
> a DocNrSkipper that only has this method:
>   int nextDocNr(int docNr)
> ?"
> 
> As far as I can comprehend, it makes a lot of sense to decouple Scoring from 
Matching (of course their intermixing as well).  This would practically mean 
that complete Query execution  mechanics could be reused instead of making 
"paralel world" of non scoring queries/ChainedFilters... 

Query execution is always done by a Scorer.

> 
> Please correct me if understanding is wrong:
> - It would enable one Clause in BooleanQuery to be "pure boolean" (aka 
constant scoring)

No, for the reason above. There already is a ConstantScoreQuery that
has a constructor that takes a Filter argument.

> - it would permit Filter-s to be combined without "external helpers" like 
ChainedFilter by using Matcher only, by purely using existing Query

Same situation. However, it might be possible to refactor the Matcher
logic out of the various scorers used by boolean query to be reused for
boolean operations on filter like things, but I wouldn't expect much to
be gained from that.

> 
> 
> DocNrSkipper and such Matcher are semantically the same,  it couldd work 
just the other way round  as well where  Matcher implements  DocNrSkipper. 
DocNrSkipper just skipps docs, Matcher can do u bit more so the other option 
could even be ok? Anyhow, sounds like style question    

A Matcher needs a little bit of dynamic state in the form of the current
document number, and DocNrSkipper does not have that, so they
are not completely equivalent. 

As for the style, I have no preference either way. Doing a filtered search
in IndexSearcher takes only a few lines more code than a non filtered search
in any case.

Regards,
Paul Elschot

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Explaining a filter; Scorer extending Matcher; (was: BooleanWeight.normalize(float) doesn't normalize prohibited clauses?)

Posted by eks dev <ek...@yahoo.co.uk>.
"Any thoughts on whether such a Matcher would be preferable to 
a DocNrSkipper that only has this method:
  int nextDocNr(int docNr)
?"

As far as I can comprehend, it makes a lot of sense to decouple Scoring from Matching (of course their intermixing as well).  This would practically mean that complete Query execution  mechanics could be reused instead of making "paralel world" of non scoring queries/ChainedFilters... 

Please correct me if understanding is wrong:
- It would enable one Clause in BooleanQuery to be "pure boolean" (aka constant scoring)
- it would permit Filter-s to be combined without "external helpers" like ChainedFilter by using Matcher only, by purely using existing Query


DocNrSkipper and such Matcher are semantically the same,  it couldd work just the other way round  as well where  Matcher implements  DocNrSkipper. DocNrSkipper just skipps docs, Matcher can do u bit more so the other option could even be ok? Anyhow, sounds like style question    






---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org