You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Edward Turner <ed...@gmail.com> on 2020/09/28 14:56:53 UTC
Solr storage of fields <-> indexed data
Hi all,
We have recently switched to using edismax + qf fields, and no longer use
copyfields to allow us to easily search over values in multiple fields (by
copying multiple fields' values to the copyfield destinations, and then
performing queries over the destination field).
By removing the copyfields, we've found that our index sizes have reduced
by ~40% in some cases, which is great! We're just curious now as to exactly
how this can be ...
My question is, given the following two schemas, if we index some data to
the "description" field, will the index for schema1 be twice as large as
the index of schema2? (I guess this relates to how, internally, Solr stores
field + index data)
Old way -- schema1:
=======
<field name="description type="text_general" indexed="true"
multiValued="false"/>
<field name="default_field" type="text_general" indexed="true"
multiValued="false" />
<copyField source="description" dest="default_field />
New way -- schema2:
=======
<field name="description type="text_general" indexed="true"
multiValued="false"/>
Many thanks and kind regards,
Edd
Re: Solr storage of fields <-> indexed data
Posted by Edward Turner <ed...@gmail.com>.
That's really good and helpful info, thank you. Perfect.
Best wishes,
Edd
On Mon, 28 Sep 2020, 5:53 pm Shawn Heisey, <ap...@elyograg.org> wrote:
> On 9/28/2020 8:56 AM, Edward Turner wrote:
> > By removing the copyfields, we've found that our index sizes have reduced
> > by ~40% in some cases, which is great! We're just curious now as to
> exactly
> > how this can be ...
>
> That's not surprising.
>
> > My question is, given the following two schemas, if we index some data to
> > the "description" field, will the index for schema1 be twice as large as
> > the index of schema2? (I guess this relates to how, internally, Solr
> stores
> > field + index data)
> >
> > Old way -- schema1:
> > =======
> > <field name="description type="text_general" indexed="true"
> > multiValued="false"/>
> > <field name="default_field" type="text_general" indexed="true"
> > multiValued="false" />
> > <copyField source="description" dest="default_field />
> >
> > New way -- schema2:
> > =======
> > <field name="description type="text_general" indexed="true"
> > multiValued="false"/>
>
> If the only field in the indexed documents is "description", the index
> built with schema2 will be half the size of the index built with
> schema1. Both fields referenced by "copyField" are the same type and
> have the same settings, so they would contain exactly the same data at
> the Lucene level.
>
> Having the same type for a source and destination field is normally only
> useful if multiple sources are copied to a destination, which requires
> multiValued="true" on the destination -- NOT the case in your example.
>
> There is one other use case for a copyField -- using the same data
> differently, with different type values. For example you might have one
> type for faceting and one for searching.
>
> Thanks,
> Shawn
>
Re: Solr storage of fields <-> indexed data
Posted by Shawn Heisey <ap...@elyograg.org>.
On 9/28/2020 8:56 AM, Edward Turner wrote:
> By removing the copyfields, we've found that our index sizes have reduced
> by ~40% in some cases, which is great! We're just curious now as to exactly
> how this can be ...
That's not surprising.
> My question is, given the following two schemas, if we index some data to
> the "description" field, will the index for schema1 be twice as large as
> the index of schema2? (I guess this relates to how, internally, Solr stores
> field + index data)
>
> Old way -- schema1:
> =======
> <field name="description type="text_general" indexed="true"
> multiValued="false"/>
> <field name="default_field" type="text_general" indexed="true"
> multiValued="false" />
> <copyField source="description" dest="default_field />
>
> New way -- schema2:
> =======
> <field name="description type="text_general" indexed="true"
> multiValued="false"/>
If the only field in the indexed documents is "description", the index
built with schema2 will be half the size of the index built with
schema1. Both fields referenced by "copyField" are the same type and
have the same settings, so they would contain exactly the same data at
the Lucene level.
Having the same type for a source and destination field is normally only
useful if multiple sources are copied to a destination, which requires
multiValued="true" on the destination -- NOT the case in your example.
There is one other use case for a copyField -- using the same data
differently, with different type values. For example you might have one
type for faceting and one for searching.
Thanks,
Shawn
Re: Solr storage of fields <-> indexed data
Posted by Erick Erickson <er...@gmail.com>.
Fields are placed in the index totally separately from each
other, so it’s no wonder that removing
the copyField results in this kind of savings.
And they have to be separate. Consider what comes out of the end of the
analysis chain. The same input could produce totally different output.
As a trivial example, imagine two fields:
whitespacetokenizer
lowercasefilter
whitespacetokenizer
lowercasefilter
edgengramfilterfactory
and identical input "fleas”. The output of the first would be “fleas”, and the
output of the second would be something like “f”, “fl”, “fle”, “flea”, “fleas”.
Trying to share the tokens between fields would be a nightmare.
And that’s only one of many ways the output of two different analysis
chains could be different…
Best,
Erick
> On Sep 28, 2020, at 10:56 AM, Edward Turner <ed...@gmail.com> wrote:
>
> Hi all,
>
> We have recently switched to using edismax + qf fields, and no longer use
> copyfields to allow us to easily search over values in multiple fields (by
> copying multiple fields' values to the copyfield destinations, and then
> performing queries over the destination field).
>
> By removing the copyfields, we've found that our index sizes have reduced
> by ~40% in some cases, which is great! We're just curious now as to exactly
> how this can be ...
>
> My question is, given the following two schemas, if we index some data to
> the "description" field, will the index for schema1 be twice as large as
> the index of schema2? (I guess this relates to how, internally, Solr stores
> field + index data)
>
> Old way -- schema1:
> =======
> <field name="description type="text_general" indexed="true"
> multiValued="false"/>
> <field name="default_field" type="text_general" indexed="true"
> multiValued="false" />
> <copyField source="description" dest="default_field />
>
> New way -- schema2:
> =======
> <field name="description type="text_general" indexed="true"
> multiValued="false"/>
>
> Many thanks and kind regards,
>
> Edd