Shared Stored Field

[StrW_dev <r....@structweb.nl>]

Hello,

We have a denormalized index where certain documents point in essence to the
same content. 
The relevance of the documents depends on the current context. E.g. document
A has a different boost factor when we apply filter F1 compared to when we
use filter F2 (or F3, etc).
To support this we denormalize document A with a unique boost field, such
that for each filter he can be found in he has a different relevance.

The problem is that the documents have a big stored content that is required
for the highlighting snippets.

This denormalization grows the index size with a factor 100 in worse case.
Storing the same big content field a lot of times times seems really
inefficient. 
Is there a way to point a group of documents to the same stored content
fields? 
Or is there a different way to influence the relevance depending on the
current search context?



--
View this message in context: http://lucene.472066.n3.nabble.com/Shared-Stored-Field-tp4130351.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Shared Stored Field

[StrW_dev <r....@structweb.nl>]

Erick Erickson wrote
> So you're saying that you have B_1 - B_8 in one doc, B_9 - B_16 in
> another doc etc?

Well yes that could work, but this would mean we get a lot of unique dymanic
fields, basically equal to the number of documents in our system and I am
not sure if that is a good practice.



--
View this message in context: http://lucene.472066.n3.nabble.com/Shared-Stored-Field-tp4130351p4130589.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Shared Stored Field

[Erick Erickson <er...@gmail.com>]

So you're saying that you have B_1 - B_8 in one doc, B_9 - B_16 in
another doc etc?

What's so confusing is that in your first e-mail, you said:
bq: This denormalization grows the index size with a factor 100 in worse case.

Which I took to mean you have at most 100 of these fields.

Please look at the function query page I referenced and try a few
things so we can deal with specific questions. You can put the results
of a _query_ in a function query, so you could probably just form a
sub-query that returns a score that you in turn use to boost the doc.

Best,
Erick

On Thu, Apr 10, 2014 at 8:04 AM, StrW_dev <r....@structweb.nl> wrote:
> Erick Erickson wrote
>> Well, you're constructing the URL somewhere, you can choose the right
>> boost there can't you?
>
> Yes of course!
>
> As example:
> We have one filter field called FILTER which can have unlimited values acros
> all documents.
> Each document as on average 8 values set for FILTER (e.g. FILTER
> ["1","2",..,"8"]).
> So we could add boost fields depending on each of these values as B_1:1.0,
> ...  ,B_7:5.0 for example and use that during query time. This is your
> suggestions correct?
>
> So each document has on average 8 of these dynamic fields, while over the
> whole index we have unlimited of these fields. What would this mean for the
> performance?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Shared-Stored-Field-tp4130351p4130411.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Shared Stored Field

[StrW_dev <r....@structweb.nl>]

Erick Erickson wrote
> Well, you're constructing the URL somewhere, you can choose the right 
> boost there can't you? 

Yes of course!

As example:
We have one filter field called FILTER which can have unlimited values acros
all documents.
Each document as on average 8 values set for FILTER (e.g. FILTER
["1","2",..,"8"]).
So we could add boost fields depending on each of these values as B_1:1.0,
...  ,B_7:5.0 for example and use that during query time. This is your
suggestions correct?

So each document has on average 8 of these dynamic fields, while over the
whole index we have unlimited of these fields. What would this mean for the
performance?



--
View this message in context: http://lucene.472066.n3.nabble.com/Shared-Stored-Field-tp4130351p4130411.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Shared Stored Field

[Erick Erickson <er...@gmail.com>]

bq: But it is possible to select a different boost field depending on
the current filter query?

Well, you're constructing the URL somewhere, you can choose the right
boost there can't you?

I don't understand this bit:
Well its basically one multivalued field that can have unlimited
values and has multiple per document (on average like 8)

The _values_ aren't at issue, it's just the name of the field. You can
have lots of dynamic fields defined in your documents and it's not too
expensive. Don't go wild here, when you get up into the hundreds maybe
you should think about it a bit.

I feel I'm missing something, some concrete examples would help a lot.

Best,
Erick

On Thu, Apr 10, 2014 at 7:33 AM, StrW_dev <r....@structweb.nl> wrote:
> Erick Erickson wrote
>>
>> So why not index these boosts in separate fields in the document (e.g.
>> f1_boost, f2_boost etc) and use a function query
>> (https://cwiki.apache.org/confluence/display/solr/Function+Queries) at
>> query time to boost by the correct one?
>
> Well its basically one multivalued field that can have unlimited values and
> has multiple per document (on average like 8). In that case we should add a
> boost field for each of the values in the document, in general we would get
> unlimited amount of dynamic fields in the index.
>
> But it is possible to select a different boost field depending on the
> current filter query?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Shared-Stored-Field-tp4130351p4130399.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Shared Stored Field

[StrW_dev <r....@structweb.nl>]

Erick Erickson wrote
>  
> So why not index these boosts in separate fields in the document (e.g.
> f1_boost, f2_boost etc) and use a function query
> (https://cwiki.apache.org/confluence/display/solr/Function+Queries) at
> query time to boost by the correct one?

Well its basically one multivalued field that can have unlimited values and
has multiple per document (on average like 8). In that case we should add a
boost field for each of the values in the document, in general we would get
unlimited amount of dynamic fields in the index. 

But it is possible to select a different boost field depending on the
current filter query? 



--
View this message in context: http://lucene.472066.n3.nabble.com/Shared-Stored-Field-tp4130351p4130399.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Shared Stored Field

[Erick Erickson <er...@gmail.com>]

Hmmm, I scanned your question, so maybe I missed something. It sounds
like you have a fixed number of filters known at index time, right? So
why not index these boosts in separate fields in the document (e.g.
f1_boost, f2_boost etc) and use a function query
(https://cwiki.apache.org/confluence/display/solr/Function+Queries) at
query time to boost by the correct one?

Of course I may be waaaay off base here, but....

BTW, you could use dynamic fields to not have to pre-define the
maximum number of boost fields, something like this in my example:
<dynamicField name="*_boost"  type="float"  indexed="true"  stored="false"/>

Best
Erick

On Thu, Apr 10, 2014 at 4:30 AM, StrW_dev <r....@structweb.nl> wrote:
> Hello,
>
> We have a denormalized index where certain documents point in essence to the
> same content.
> The relevance of the documents depends on the current context. E.g. document
> A has a different boost factor when we apply filter F1 compared to when we
> use filter F2 (or F3, etc).
> To support this we denormalize document A with a unique boost field, such
> that for each filter he can be found in he has a different relevance.
>
> The problem is that the documents have a big stored content that is required
> for the highlighting snippets.
>
> This denormalization grows the index size with a factor 100 in worse case.
> Storing the same big content field a lot of times times seems really
> inefficient.
> Is there a way to point a group of documents to the same stored content
> fields?
> Or is there a different way to influence the relevance depending on the
> current search context?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Shared-Stored-Field-tp4130351.html
> Sent from the Solr - User mailing list archive at Nabble.com.