You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Puneeth Bikkumanla <pu...@mongodb.com.INVALID> on 2021/04/09 22:39:00 UTC

Lucene Explanation

Hello,
I am currently working on a project that would like to implement Document
Explain where we can see how a document was scored internally in lucene
given a query.

I see that the IndexSearcher has an explain
<https://lucene.apache.org/core/8_0_0/core/org/apache/lucene/search/IndexSearcher.html#explain-org.apache.lucene.search.Query-int->
method
available that returns an Explanation
<https://lucene.apache.org/core/8_0_0/core/org/apache/lucene/search/Explanation.html>
object. An Explanation object only contains a description field (string)
but there is no way to know what part of a score that Explanation object is
for without parsing the description field itself. We wanted to implement
Document Explain in a more safe way where we could know what part of the
score an Explanation object is associated with and not parse the
description string field to find out. Here are a few of the options I have
thought of:

1. I was thinking about extending the similarity class (BM25Similarity) and
then overriding the particular methods that dealt with the different
subcomponents of explain but saw that the explainTF
<https://github.com/apache/lucene/blob/e510ef11c2a4307dd6ecc8c8974eef2c04e3e4d6/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java#L268>
method
is private. Is there a reason why this is? It would be very useful if it
could be public so that I can override it and store the knowledge that the
returned Explanation is for the TF component of the document score.

2. I also thought about extending the IndexSearcher and overriding the
createWeight method to store the weight structure and then use that to
understand the resulting Explanation structure from the IndexSearcher's
explain method.

Please let me know if any of that didn't make sense. Also, if anyone has
any other ideas on how I could approach this problem suggestions would be
greatly appreciated. Lastly, I would be happy to submit a PR to modify
Lucene's Explanation to be more aware of where it is created.

Re: Lucene Explanation

Posted by Puneeth Bikkumanla <pu...@mongodb.com.INVALID>.
Thank you this was very helpful!

On Mon, Apr 12, 2021 at 9:07 AM Michael Sokolov <ms...@gmail.com> wrote:

> You might want to check out
> https://issues.apache.org/jira/browse/LUCENE-8019 where I tried to
> implement some debugging utilities on top of Explain. It never got
> committed, but it does explore some of the challenges around
> introducing a more structured explain response.
>
> On Fri, Apr 9, 2021 at 6:40 PM Puneeth Bikkumanla
> <pu...@mongodb.com.invalid> wrote:
> >
> > Hello,
> > I am currently working on a project that would like to implement Document
> > Explain where we can see how a document was scored internally in lucene
> > given a query.
> >
> > I see that the IndexSearcher has an explain
> > <
> https://lucene.apache.org/core/8_0_0/core/org/apache/lucene/search/IndexSearcher.html#explain-org.apache.lucene.search.Query-int-
> >
> > method
> > available that returns an Explanation
> > <
> https://lucene.apache.org/core/8_0_0/core/org/apache/lucene/search/Explanation.html
> >
> > object. An Explanation object only contains a description field (string)
> > but there is no way to know what part of a score that Explanation object
> is
> > for without parsing the description field itself. We wanted to implement
> > Document Explain in a more safe way where we could know what part of the
> > score an Explanation object is associated with and not parse the
> > description string field to find out. Here are a few of the options I
> have
> > thought of:
> >
> > 1. I was thinking about extending the similarity class (BM25Similarity)
> and
> > then overriding the particular methods that dealt with the different
> > subcomponents of explain but saw that the explainTF
> > <
> https://github.com/apache/lucene/blob/e510ef11c2a4307dd6ecc8c8974eef2c04e3e4d6/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java#L268
> >
> > method
> > is private. Is there a reason why this is? It would be very useful if it
> > could be public so that I can override it and store the knowledge that
> the
> > returned Explanation is for the TF component of the document score.
> >
> > 2. I also thought about extending the IndexSearcher and overriding the
> > createWeight method to store the weight structure and then use that to
> > understand the resulting Explanation structure from the IndexSearcher's
> > explain method.
> >
> > Please let me know if any of that didn't make sense. Also, if anyone has
> > any other ideas on how I could approach this problem suggestions would be
> > greatly appreciated. Lastly, I would be happy to submit a PR to modify
> > Lucene's Explanation to be more aware of where it is created.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Lucene Explanation

Posted by Michael Sokolov <ms...@gmail.com>.
You might want to check out
https://issues.apache.org/jira/browse/LUCENE-8019 where I tried to
implement some debugging utilities on top of Explain. It never got
committed, but it does explore some of the challenges around
introducing a more structured explain response.

On Fri, Apr 9, 2021 at 6:40 PM Puneeth Bikkumanla
<pu...@mongodb.com.invalid> wrote:
>
> Hello,
> I am currently working on a project that would like to implement Document
> Explain where we can see how a document was scored internally in lucene
> given a query.
>
> I see that the IndexSearcher has an explain
> <https://lucene.apache.org/core/8_0_0/core/org/apache/lucene/search/IndexSearcher.html#explain-org.apache.lucene.search.Query-int->
> method
> available that returns an Explanation
> <https://lucene.apache.org/core/8_0_0/core/org/apache/lucene/search/Explanation.html>
> object. An Explanation object only contains a description field (string)
> but there is no way to know what part of a score that Explanation object is
> for without parsing the description field itself. We wanted to implement
> Document Explain in a more safe way where we could know what part of the
> score an Explanation object is associated with and not parse the
> description string field to find out. Here are a few of the options I have
> thought of:
>
> 1. I was thinking about extending the similarity class (BM25Similarity) and
> then overriding the particular methods that dealt with the different
> subcomponents of explain but saw that the explainTF
> <https://github.com/apache/lucene/blob/e510ef11c2a4307dd6ecc8c8974eef2c04e3e4d6/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java#L268>
> method
> is private. Is there a reason why this is? It would be very useful if it
> could be public so that I can override it and store the knowledge that the
> returned Explanation is for the TF component of the document score.
>
> 2. I also thought about extending the IndexSearcher and overriding the
> createWeight method to store the weight structure and then use that to
> understand the resulting Explanation structure from the IndexSearcher's
> explain method.
>
> Please let me know if any of that didn't make sense. Also, if anyone has
> any other ideas on how I could approach this problem suggestions would be
> greatly appreciated. Lastly, I would be happy to submit a PR to modify
> Lucene's Explanation to be more aware of where it is created.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org