You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jamie Johnson <je...@gmail.com> on 2015/08/25 18:09:48 UTC
Tokenizers and DelimitedPayloadTokenFilterFactory
I would like to specify a particular payload for all tokens emitted from a
tokenizer, but don't see a clear way to do this. Ideally I could specify
that something like the DelimitedPayloadTokenFilter be run on the entire
field and then standard analysis be done on the rest of the field, so in
the case that I had the following text
this is a test\Foo
I would like to create tokens "this", "is", "a", "test" each with a payload
of Foo. From what I'm seeing though only test get's the payload. Is there
anyway to accomplish this or will I need to implement a custom tokenizer?
Re: Tokenizers and DelimitedPayloadTokenFilterFactory
Posted by Erick Erickson <er...@gmail.com>.
Oh My. What fun!
bq: I need a way to specify the payload on the other field types
Not to my knowledge. The payload mechanism is built on
the capability of having a filter in the analysis chain. And there's
no analysis chain with primitive types (string, numeric and the like).
Hmmm. Totally off the top of my head, but I wonder if you could
use a Binary type and customize all the reading to spoof
whatever primitive types you wanted while respecting your
auth tokens?
Best,
Erick
On Tue, Aug 25, 2015 at 10:37 AM, Jamie Johnson <je...@gmail.com> wrote:
> To be clear, we are using payloads as a way to attach authorizations to
> individual tokens within Solr. The payloads are normal Solr Payloads
> though we are not using floats, we are using the identity payload encoder
> (org.apache.lucene.analysis.payloads.IdentityEncoder) which allows for
> storing a byte[] of our choosing into the payload field.
>
> This works great for text, but now that I'm indexing more than just text I
> need a way to specify the payload on the other field types. Does that make
> more sense?
>
> On Tue, Aug 25, 2015 at 12:52 PM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> This really sounds like an XY problem. Or when you use
>> "payload" it's not the Solr payload.
>>
>> So Solr Payloads are a float value that you can attach to
>> individual terms to influence the scoring. Attaching the
>> _same_ payload to all terms in a field is much the same
>> thing as boosting on any matches in the field at query time
>> or boosting on the field at index time (this latter assuming
>> that different docs would have different boosts).
>>
>> So can you back up a bit and tell us what you're trying to
>> accomplish maybe we can be sure we're both talking about
>> the same thing ;)
>>
>> Best,
>> Erick
>>
>> On Tue, Aug 25, 2015 at 9:09 AM, Jamie Johnson <je...@gmail.com> wrote:
>> > I would like to specify a particular payload for all tokens emitted from
>> a
>> > tokenizer, but don't see a clear way to do this. Ideally I could specify
>> > that something like the DelimitedPayloadTokenFilter be run on the entire
>> > field and then standard analysis be done on the rest of the field, so in
>> > the case that I had the following text
>> >
>> > this is a test\Foo
>> >
>> > I would like to create tokens "this", "is", "a", "test" each with a
>> payload
>> > of Foo. From what I'm seeing though only test get's the payload. Is
>> there
>> > anyway to accomplish this or will I need to implement a custom tokenizer?
>>
Re: Tokenizers and DelimitedPayloadTokenFilterFactory
Posted by Jamie Johnson <je...@gmail.com>.
To be clear, we are using payloads as a way to attach authorizations to
individual tokens within Solr. The payloads are normal Solr Payloads
though we are not using floats, we are using the identity payload encoder
(org.apache.lucene.analysis.payloads.IdentityEncoder) which allows for
storing a byte[] of our choosing into the payload field.
This works great for text, but now that I'm indexing more than just text I
need a way to specify the payload on the other field types. Does that make
more sense?
On Tue, Aug 25, 2015 at 12:52 PM, Erick Erickson <er...@gmail.com>
wrote:
> This really sounds like an XY problem. Or when you use
> "payload" it's not the Solr payload.
>
> So Solr Payloads are a float value that you can attach to
> individual terms to influence the scoring. Attaching the
> _same_ payload to all terms in a field is much the same
> thing as boosting on any matches in the field at query time
> or boosting on the field at index time (this latter assuming
> that different docs would have different boosts).
>
> So can you back up a bit and tell us what you're trying to
> accomplish maybe we can be sure we're both talking about
> the same thing ;)
>
> Best,
> Erick
>
> On Tue, Aug 25, 2015 at 9:09 AM, Jamie Johnson <je...@gmail.com> wrote:
> > I would like to specify a particular payload for all tokens emitted from
> a
> > tokenizer, but don't see a clear way to do this. Ideally I could specify
> > that something like the DelimitedPayloadTokenFilter be run on the entire
> > field and then standard analysis be done on the rest of the field, so in
> > the case that I had the following text
> >
> > this is a test\Foo
> >
> > I would like to create tokens "this", "is", "a", "test" each with a
> payload
> > of Foo. From what I'm seeing though only test get's the payload. Is
> there
> > anyway to accomplish this or will I need to implement a custom tokenizer?
>
Re: Tokenizers and DelimitedPayloadTokenFilterFactory
Posted by Erick Erickson <er...@gmail.com>.
This really sounds like an XY problem. Or when you use
"payload" it's not the Solr payload.
So Solr Payloads are a float value that you can attach to
individual terms to influence the scoring. Attaching the
_same_ payload to all terms in a field is much the same
thing as boosting on any matches in the field at query time
or boosting on the field at index time (this latter assuming
that different docs would have different boosts).
So can you back up a bit and tell us what you're trying to
accomplish maybe we can be sure we're both talking about
the same thing ;)
Best,
Erick
On Tue, Aug 25, 2015 at 9:09 AM, Jamie Johnson <je...@gmail.com> wrote:
> I would like to specify a particular payload for all tokens emitted from a
> tokenizer, but don't see a clear way to do this. Ideally I could specify
> that something like the DelimitedPayloadTokenFilter be run on the entire
> field and then standard analysis be done on the rest of the field, so in
> the case that I had the following text
>
> this is a test\Foo
>
> I would like to create tokens "this", "is", "a", "test" each with a payload
> of Foo. From what I'm seeing though only test get's the payload. Is there
> anyway to accomplish this or will I need to implement a custom tokenizer?