You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Dave Seltzer <ds...@tveyes.com> on 2014/08/18 16:32:08 UTC

Combining a String Tag with a Numeric Value

Hello!

I have some new entity data that I'm indexing which takes the form of:

String: EntityString
Float: Confidence

I want to add these to a generic "Tags" field (for faceting), but I'm not
sure how to hold onto the confidence. Token Payloads seem like one method,
but then I'm not sure how to extract the Payload.

Alternatively I could create two fields: TagIndexed which stores just the
string value and TagStored which contains a delimited String|Float.

What's the right way to do this?

Thanks!

-D

Re: Combining a String Tag with a Numeric Value

Posted by Erick Erickson <er...@gmail.com>.
If you're doing this in a sharded environment, it may be "interesting".

Good Luck!

Erick

On Mon, Aug 18, 2014 at 2:03 PM, Dave Seltzer <ds...@tveyes.com> wrote:
> Thanks Erick,
>
> I'm not sure I need to score the documents based on the numeric value, but
> I am interested in being able to calculate the average (Mean) of all the
> numeric values for a given tag. For example, what is the average confidence
> of Tag1 across all documents.
>
> I'm not sure I can do that without building a FunctionQuery.
>
> -Dave
>
>
> On Mon, Aug 18, 2014 at 12:46 PM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> Hmmm, there's no particular "right way". It'd be simpler
>> to index these as two separate fields _if_ there's only
>> one pair per document. If there are more and you index them
>> as two mutliValued fields, there's no good way at _query_ time
>> to retain the association. The returned multiValued fields are
>> guaranteed to be in the same order of insertion so you can
>> display the correct pairs, but you can't use the association
>> to score docs. Hmmm, somewhat abstract. OK say you want to
>> associate two tag/value pairs, tag1:50 and tag2:100. Say further
>> that you have two multiValued fields, Tags and Values and then
>> index tag1 and tag2 into Tags and 50 and 100 into Values.
>> There's no good way to express "q=tags:tag1 and factor the
>> associated value of 50 into the score"
>>
>> Note that the returned _values_ will be
>> Tags:   tag1 tag2
>> Values  50  100
>>
>> So at that point you can see the associations.
>>
>> that said, if there's only _one_ such tag/value pair per document,
>> it's easy to write a FunctionQuery (
>> http://wiki.apache.org/solr/FunctionQuery)
>> that does this.
>>
>> *******************************
>>
>> If you have many tag/value pairs, payloads are probably what you want.
>> Here's an end-to-end example:
>>
>> http://searchhub.org/2014/06/13/end-to-end-payload-example-in-solr/
>>
>> Best,
>> Erick
>>
>> On Mon, Aug 18, 2014 at 7:32 AM, Dave Seltzer <ds...@tveyes.com> wrote:
>> > Hello!
>> >
>> > I have some new entity data that I'm indexing which takes the form of:
>> >
>> > String: EntityString
>> > Float: Confidence
>> >
>> > I want to add these to a generic "Tags" field (for faceting), but I'm not
>> > sure how to hold onto the confidence. Token Payloads seem like one
>> method,
>> > but then I'm not sure how to extract the Payload.
>> >
>> > Alternatively I could create two fields: TagIndexed which stores just the
>> > string value and TagStored which contains a delimited String|Float.
>> >
>> > What's the right way to do this?
>> >
>> > Thanks!
>> >
>> > -D

Re: Combining a String Tag with a Numeric Value

Posted by Dave Seltzer <ds...@tveyes.com>.
Thanks Erick,

I'm not sure I need to score the documents based on the numeric value, but
I am interested in being able to calculate the average (Mean) of all the
numeric values for a given tag. For example, what is the average confidence
of Tag1 across all documents.

I'm not sure I can do that without building a FunctionQuery.

-Dave


On Mon, Aug 18, 2014 at 12:46 PM, Erick Erickson <er...@gmail.com>
wrote:

> Hmmm, there's no particular "right way". It'd be simpler
> to index these as two separate fields _if_ there's only
> one pair per document. If there are more and you index them
> as two mutliValued fields, there's no good way at _query_ time
> to retain the association. The returned multiValued fields are
> guaranteed to be in the same order of insertion so you can
> display the correct pairs, but you can't use the association
> to score docs. Hmmm, somewhat abstract. OK say you want to
> associate two tag/value pairs, tag1:50 and tag2:100. Say further
> that you have two multiValued fields, Tags and Values and then
> index tag1 and tag2 into Tags and 50 and 100 into Values.
> There's no good way to express "q=tags:tag1 and factor the
> associated value of 50 into the score"
>
> Note that the returned _values_ will be
> Tags:   tag1 tag2
> Values  50  100
>
> So at that point you can see the associations.
>
> that said, if there's only _one_ such tag/value pair per document,
> it's easy to write a FunctionQuery (
> http://wiki.apache.org/solr/FunctionQuery)
> that does this.
>
> *******************************
>
> If you have many tag/value pairs, payloads are probably what you want.
> Here's an end-to-end example:
>
> http://searchhub.org/2014/06/13/end-to-end-payload-example-in-solr/
>
> Best,
> Erick
>
> On Mon, Aug 18, 2014 at 7:32 AM, Dave Seltzer <ds...@tveyes.com> wrote:
> > Hello!
> >
> > I have some new entity data that I'm indexing which takes the form of:
> >
> > String: EntityString
> > Float: Confidence
> >
> > I want to add these to a generic "Tags" field (for faceting), but I'm not
> > sure how to hold onto the confidence. Token Payloads seem like one
> method,
> > but then I'm not sure how to extract the Payload.
> >
> > Alternatively I could create two fields: TagIndexed which stores just the
> > string value and TagStored which contains a delimited String|Float.
> >
> > What's the right way to do this?
> >
> > Thanks!
> >
> > -D

Re: Combining a String Tag with a Numeric Value

Posted by Erick Erickson <er...@gmail.com>.
Hmmm, there's no particular "right way". It'd be simpler
to index these as two separate fields _if_ there's only
one pair per document. If there are more and you index them
as two mutliValued fields, there's no good way at _query_ time
to retain the association. The returned multiValued fields are
guaranteed to be in the same order of insertion so you can
display the correct pairs, but you can't use the association
to score docs. Hmmm, somewhat abstract. OK say you want to
associate two tag/value pairs, tag1:50 and tag2:100. Say further
that you have two multiValued fields, Tags and Values and then
index tag1 and tag2 into Tags and 50 and 100 into Values.
There's no good way to express "q=tags:tag1 and factor the
associated value of 50 into the score"

Note that the returned _values_ will be
Tags:   tag1 tag2
Values  50  100

So at that point you can see the associations.

that said, if there's only _one_ such tag/value pair per document,
it's easy to write a FunctionQuery (http://wiki.apache.org/solr/FunctionQuery)
that does this.

*******************************

If you have many tag/value pairs, payloads are probably what you want.
Here's an end-to-end example:

http://searchhub.org/2014/06/13/end-to-end-payload-example-in-solr/

Best,
Erick

On Mon, Aug 18, 2014 at 7:32 AM, Dave Seltzer <ds...@tveyes.com> wrote:
> Hello!
>
> I have some new entity data that I'm indexing which takes the form of:
>
> String: EntityString
> Float: Confidence
>
> I want to add these to a generic "Tags" field (for faceting), but I'm not
> sure how to hold onto the confidence. Token Payloads seem like one method,
> but then I'm not sure how to extract the Payload.
>
> Alternatively I could create two fields: TagIndexed which stores just the
> string value and TagStored which contains a delimited String|Float.
>
> What's the right way to do this?
>
> Thanks!
>
> -D