problem with payload retrieval in custom query - lucene 5.3.1

[Bob Price <rj...@gmail.com>]

Via elasticsearch 2.1 we have written a custom Lucene TokenFilter that adds
a payload to a special term during indexing analysis, and we also have
custom Lucene Query/Weight/Scorer classes that compare this payload against
some query data to decide if a document matches a query or not.

I know that the payloads are being written as desired since I can easily
see them via elasticsearch's termvector API.

However, while our Scorer can see all of the documents with this special
term, I can never get access to its payload.

The approach to doing this in the Scorer is simply:

docs = leafReaderContext.reader().postings(specialTerm, PostingsEnum.PAYLOADS);

Then, walking through the docs via nextDoc() and trying to retrieve each
doc's payload via:

BytesRef payloadRef = docs.getPayload();
if (payloadRef != null) {
   byte[] payload = payloadRef.bytes;

}

I have logging in place so that I see that every document I have in a test
index is being looked at, but for every document, the payloadRef comes back
null.

What am I doing wrong?  Is there some setting I need to enable somewhere to
allow the postings method to do what it says it can do?

Is there a better way to do this?

Any advice would be appreciated!

Bob

Re: problem with payload retrieval in custom query - lucene 5.3.1

[Bob Price <rj...@gmail.com>]

Thanks Adrien!  That was the missing piece.  Now I can access the payloads.

On Thu, Dec 24, 2015 at 8:50 AM, Adrien Grand <jp...@gmail.com> wrote:

> Note that payloads are stored per position, not per document. Maybe the
> problem is that you never call docs.nextPosition()?
>
> Le mer. 23 déc. 2015 à 19:30, Bob Price <rj...@gmail.com> a écrit :
>
> > Via elasticsearch 2.1 we have written a custom Lucene TokenFilter that
> adds
> > a payload to a special term during indexing analysis, and we also have
> > custom Lucene Query/Weight/Scorer classes that compare this payload
> against
> > some query data to decide if a document matches a query or not.
> >
> > I know that the payloads are being written as desired since I can easily
> > see them via elasticsearch's termvector API.
> >
> > However, while our Scorer can see all of the documents with this special
> > term, I can never get access to its payload.
> >
> > The approach to doing this in the Scorer is simply:
> >
> > docs = leafReaderContext.reader().postings(specialTerm,
> > PostingsEnum.PAYLOADS);
> >
> > Then, walking through the docs via nextDoc() and trying to retrieve each
> > doc's payload via:
> >
> > BytesRef payloadRef = docs.getPayload();
> > if (payloadRef != null) {
> >    byte[] payload = payloadRef.bytes;
> >
> > }
> >
> > I have logging in place so that I see that every document I have in a
> test
> > index is being looked at, but for every document, the payloadRef comes
> back
> > null.
> >
> > What am I doing wrong?  Is there some setting I need to enable somewhere
> to
> > allow the postings method to do what it says it can do?
> >
> > Is there a better way to do this?
> >
> > Any advice would be appreciated!
> >
> > Bob
> >
>

Re: problem with payload retrieval in custom query - lucene 5.3.1

[Adrien Grand <jp...@gmail.com>]

Note that payloads are stored per position, not per document. Maybe the
problem is that you never call docs.nextPosition()?

Le mer. 23 déc. 2015 à 19:30, Bob Price <rj...@gmail.com> a écrit :

> Via elasticsearch 2.1 we have written a custom Lucene TokenFilter that adds
> a payload to a special term during indexing analysis, and we also have
> custom Lucene Query/Weight/Scorer classes that compare this payload against
> some query data to decide if a document matches a query or not.
>
> I know that the payloads are being written as desired since I can easily
> see them via elasticsearch's termvector API.
>
> However, while our Scorer can see all of the documents with this special
> term, I can never get access to its payload.
>
> The approach to doing this in the Scorer is simply:
>
> docs = leafReaderContext.reader().postings(specialTerm,
> PostingsEnum.PAYLOADS);
>
> Then, walking through the docs via nextDoc() and trying to retrieve each
> doc's payload via:
>
> BytesRef payloadRef = docs.getPayload();
> if (payloadRef != null) {
>    byte[] payload = payloadRef.bytes;
>
> }
>
> I have logging in place so that I see that every document I have in a test
> index is being looked at, but for every document, the payloadRef comes back
> null.
>
> What am I doing wrong?  Is there some setting I need to enable somewhere to
> allow the postings method to do what it says it can do?
>
> Is there a better way to do this?
>
> Any advice would be appreciated!
>
> Bob
>