You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Petersen, Robert" <ro...@mail.rakuten.com> on 2013/04/24 00:09:49 UTC
Solr 3.6.1: changing a field from stored to not stored
Hi guys,
What would happen if I changed a field definition on an existing field in an existing index from stored to not stored? Would solr just party on ignoring the fact that this field's data is stored in the current index? I noticed I am unnecessarily storing some fields in my index and I'd like to stop storing them without having to 'reindex the world' and let the changes just naturally percolate into my index as updates come in the normal course of things. Do you guys think I could get away with this?
Thanks,
Robert (Robi) Petersen
Senior Software Engineer
& Search Engineer
Re: Solr 3.6.1: changing a field from stored to not stored
Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
(13/04/24 7:09), Petersen, Robert wrote:
> Hi guys,
>
> What would happen if I changed a field definition on an existing field in an existing index from stored to not stored? Would solr just party on ignoring the fact that this field's data is stored in the current index? I noticed I am unnecessarily storing some fields in my index and I'd like to stop storing them without having to 'reindex the world' and let the changes just naturally percolate into my index as updates come in the normal course of things. Do you guys think I could get away with this?
>
> Thanks,
>
> Robert (Robi) Petersen
> Senior Software Engineer
> & Search Engineer
>
I think Solr will just ignore the existing stored data.
But I've never to do it myself. Please try it.
koji
--
http://soleami.com/blog/lucene-4-is-super-convenient-for-developing-nlp-tools.html
Re: Solr 3.6.1: changing a field from stored to not stored
Posted by Majirus FANSI <ma...@gmail.com>.
Good to know I missed something about solr replication.
Thanks Jan
On 24 April 2013 17:42, Jan Høydahl <ja...@cominvent.com> wrote:
> > I would create a new core as slave of the existing configuration without
> > replicating the core schema and configuration. This way I can get the
>
> This won't work, as master/slave replication copies the index files as-is.
>
> You should re-index all your data. You don't need to take down the cluster
> to do that, just re-index on top of what's there already, and your index
> will become smaller and smaller as merging kicks out the old data :)
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
> Solr Training - www.solrtraining.com
>
> 24. apr. 2013 kl. 15:59 skrev Majirus FANSI <ma...@gmail.com>:
>
> > I would create a new core as slave of the existing configuration without
> > replicating the core schema and configuration. This way I can get the
> > information from one index to the other while saving the space as fields
> in
> > the new schema are mainly not stored. After the replication I would swap
> > the cores for the online core to point to the right index dir and conf.
> > i.e. the one with less stored fields.
> >
> > Maj
> >
> >
> > On 24 April 2013 01:48, Petersen, Robert
> > <ro...@mail.rakuten.com>wrote:
> >
> >> Hey I just want to verify one thing before I start doing this: function
> >> queries only require fields to be indexed but don't require them to be
> >> stored right?
> >>
> >> -----Original Message-----
> >> From: Petersen, Robert [mailto:robert.petersen@mail.rakuten.com]
> >> Sent: Tuesday, April 23, 2013 4:39 PM
> >> To: solr-user@lucene.apache.org
> >> Subject: RE: Solr 3.6.1: changing a field from stored to not stored
> >>
> >> Good info, Thanks Hoss! I was going to add a more specific fl=
> parameter
> >> to my queries at the same time. Currently I am doing fl=*,score so that
> >> will have to be changed.
> >>
> >>
> >> -----Original Message-----
> >> From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
> >> Sent: Tuesday, April 23, 2013 4:18 PM
> >> To: solr-user@lucene.apache.org
> >> Subject: Re: Solr 3.6.1: changing a field from stored to not stored
> >>
> >>
> >> : index? I noticed I am unnecessarily storing some fields in my index
> and
> >> : I'd like to stop storing them without having to 'reindex the world'
> and
> >> : let the changes just naturally percolate into my index as updates come
> >> : in the normal course of things. Do you guys think I could get away
> with
> >> : this?
> >>
> >> Yes, you can easily get away with this type of change w/o re-indexing,
> >> however you won't gain any immediate index size savings until each and
> >> every existing doc has been reindexed and the old copies expunged from
> the
> >> index via segment merges.
> >>
> >> the one hicup thta can affect people when doing this is what happens if
> >> you use something like "fl=*" (and likely "hl=*" as well) ... many
> places
> >> in Solr will try to "avoid failure" if a stored field is found in the
> index
> >> which isn't defined in the schema, and treat that stored value as a
> string
> >> (legacy behavior designed to make it easier for people to point Solr at
> old
> >> lucene indexes built w/o using Solr) ... so if these stored values are
> not
> >> strings, you might get some weird data in your response for these
> documents.
> >>
> >>
> >> -Hoss
> >>
> >>
> >>
> >>
> >>
>
>
Re: Solr 3.6.1: changing a field from stored to not stored
Posted by Jan Høydahl <ja...@cominvent.com>.
> I would create a new core as slave of the existing configuration without
> replicating the core schema and configuration. This way I can get the
This won't work, as master/slave replication copies the index files as-is.
You should re-index all your data. You don't need to take down the cluster
to do that, just re-index on top of what's there already, and your index
will become smaller and smaller as merging kicks out the old data :)
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Solr Training - www.solrtraining.com
24. apr. 2013 kl. 15:59 skrev Majirus FANSI <ma...@gmail.com>:
> I would create a new core as slave of the existing configuration without
> replicating the core schema and configuration. This way I can get the
> information from one index to the other while saving the space as fields in
> the new schema are mainly not stored. After the replication I would swap
> the cores for the online core to point to the right index dir and conf.
> i.e. the one with less stored fields.
>
> Maj
>
>
> On 24 April 2013 01:48, Petersen, Robert
> <ro...@mail.rakuten.com>wrote:
>
>> Hey I just want to verify one thing before I start doing this: function
>> queries only require fields to be indexed but don't require them to be
>> stored right?
>>
>> -----Original Message-----
>> From: Petersen, Robert [mailto:robert.petersen@mail.rakuten.com]
>> Sent: Tuesday, April 23, 2013 4:39 PM
>> To: solr-user@lucene.apache.org
>> Subject: RE: Solr 3.6.1: changing a field from stored to not stored
>>
>> Good info, Thanks Hoss! I was going to add a more specific fl= parameter
>> to my queries at the same time. Currently I am doing fl=*,score so that
>> will have to be changed.
>>
>>
>> -----Original Message-----
>> From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
>> Sent: Tuesday, April 23, 2013 4:18 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr 3.6.1: changing a field from stored to not stored
>>
>>
>> : index? I noticed I am unnecessarily storing some fields in my index and
>> : I'd like to stop storing them without having to 'reindex the world' and
>> : let the changes just naturally percolate into my index as updates come
>> : in the normal course of things. Do you guys think I could get away with
>> : this?
>>
>> Yes, you can easily get away with this type of change w/o re-indexing,
>> however you won't gain any immediate index size savings until each and
>> every existing doc has been reindexed and the old copies expunged from the
>> index via segment merges.
>>
>> the one hicup thta can affect people when doing this is what happens if
>> you use something like "fl=*" (and likely "hl=*" as well) ... many places
>> in Solr will try to "avoid failure" if a stored field is found in the index
>> which isn't defined in the schema, and treat that stored value as a string
>> (legacy behavior designed to make it easier for people to point Solr at old
>> lucene indexes built w/o using Solr) ... so if these stored values are not
>> strings, you might get some weird data in your response for these documents.
>>
>>
>> -Hoss
>>
>>
>>
>>
>>
Re: Solr 3.6.1: changing a field from stored to not stored
Posted by Majirus FANSI <ma...@gmail.com>.
I would create a new core as slave of the existing configuration without
replicating the core schema and configuration. This way I can get the
information from one index to the other while saving the space as fields in
the new schema are mainly not stored. After the replication I would swap
the cores for the online core to point to the right index dir and conf.
i.e. the one with less stored fields.
Maj
On 24 April 2013 01:48, Petersen, Robert
<ro...@mail.rakuten.com>wrote:
> Hey I just want to verify one thing before I start doing this: function
> queries only require fields to be indexed but don't require them to be
> stored right?
>
> -----Original Message-----
> From: Petersen, Robert [mailto:robert.petersen@mail.rakuten.com]
> Sent: Tuesday, April 23, 2013 4:39 PM
> To: solr-user@lucene.apache.org
> Subject: RE: Solr 3.6.1: changing a field from stored to not stored
>
> Good info, Thanks Hoss! I was going to add a more specific fl= parameter
> to my queries at the same time. Currently I am doing fl=*,score so that
> will have to be changed.
>
>
> -----Original Message-----
> From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
> Sent: Tuesday, April 23, 2013 4:18 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr 3.6.1: changing a field from stored to not stored
>
>
> : index? I noticed I am unnecessarily storing some fields in my index and
> : I'd like to stop storing them without having to 'reindex the world' and
> : let the changes just naturally percolate into my index as updates come
> : in the normal course of things. Do you guys think I could get away with
> : this?
>
> Yes, you can easily get away with this type of change w/o re-indexing,
> however you won't gain any immediate index size savings until each and
> every existing doc has been reindexed and the old copies expunged from the
> index via segment merges.
>
> the one hicup thta can affect people when doing this is what happens if
> you use something like "fl=*" (and likely "hl=*" as well) ... many places
> in Solr will try to "avoid failure" if a stored field is found in the index
> which isn't defined in the schema, and treat that stored value as a string
> (legacy behavior designed to make it easier for people to point Solr at old
> lucene indexes built w/o using Solr) ... so if these stored values are not
> strings, you might get some weird data in your response for these documents.
>
>
> -Hoss
>
>
>
>
>
RE: Solr 3.6.1: changing a field from stored to not stored
Posted by "Petersen, Robert" <ro...@mail.rakuten.com>.
Hey I just want to verify one thing before I start doing this: function queries only require fields to be indexed but don't require them to be stored right?
-----Original Message-----
From: Petersen, Robert [mailto:robert.petersen@mail.rakuten.com]
Sent: Tuesday, April 23, 2013 4:39 PM
To: solr-user@lucene.apache.org
Subject: RE: Solr 3.6.1: changing a field from stored to not stored
Good info, Thanks Hoss! I was going to add a more specific fl= parameter to my queries at the same time. Currently I am doing fl=*,score so that will have to be changed.
-----Original Message-----
From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
Sent: Tuesday, April 23, 2013 4:18 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr 3.6.1: changing a field from stored to not stored
: index? I noticed I am unnecessarily storing some fields in my index and
: I'd like to stop storing them without having to 'reindex the world' and
: let the changes just naturally percolate into my index as updates come
: in the normal course of things. Do you guys think I could get away with
: this?
Yes, you can easily get away with this type of change w/o re-indexing, however you won't gain any immediate index size savings until each and every existing doc has been reindexed and the old copies expunged from the index via segment merges.
the one hicup thta can affect people when doing this is what happens if you use something like "fl=*" (and likely "hl=*" as well) ... many places in Solr will try to "avoid failure" if a stored field is found in the index which isn't defined in the schema, and treat that stored value as a string (legacy behavior designed to make it easier for people to point Solr at old lucene indexes built w/o using Solr) ... so if these stored values are not strings, you might get some weird data in your response for these documents.
-Hoss
RE: Solr 3.6.1: changing a field from stored to not stored
Posted by "Petersen, Robert" <ro...@mail.rakuten.com>.
Good info, Thanks Hoss! I was going to add a more specific fl= parameter to my queries at the same time. Currently I am doing fl=*,score so that will have to be changed.
-----Original Message-----
From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
Sent: Tuesday, April 23, 2013 4:18 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr 3.6.1: changing a field from stored to not stored
: index? I noticed I am unnecessarily storing some fields in my index and
: I'd like to stop storing them without having to 'reindex the world' and
: let the changes just naturally percolate into my index as updates come
: in the normal course of things. Do you guys think I could get away with
: this?
Yes, you can easily get away with this type of change w/o re-indexing, however you won't gain any immediate index size savings until each and every existing doc has been reindexed and the old copies expunged from the index via segment merges.
the one hicup thta can affect people when doing this is what happens if you use something like "fl=*" (and likely "hl=*" as well) ... many places in Solr will try to "avoid failure" if a stored field is found in the index which isn't defined in the schema, and treat that stored value as a string (legacy behavior designed to make it easier for people to point Solr at old lucene indexes built w/o using Solr) ... so if these stored values are not strings, you might get some weird data in your response for these documents.
-Hoss
Re: Solr 3.6.1: changing a field from stored to not stored
Posted by Chris Hostetter <ho...@fucit.org>.
: index? I noticed I am unnecessarily storing some fields in my index and
: I'd like to stop storing them without having to 'reindex the world' and
: let the changes just naturally percolate into my index as updates come
: in the normal course of things. Do you guys think I could get away with
: this?
Yes, you can easily get away with this type of change w/o re-indexing,
however you won't gain any immediate index size savings until each and
every existing doc has been reindexed and the old copies expunged from the
index via segment merges.
the one hicup thta can affect people when doing this is what happens if
you use something like "fl=*" (and likely "hl=*" as well) ... many places
in Solr will try to "avoid failure" if a stored field is found in the
index which isn't defined in the schema, and treat that stored value as a
string (legacy behavior designed to make it easier for people to point
Solr at old lucene indexes built w/o using Solr) ... so if these stored
values are not strings, you might get some weird data in your response for
these documents.
-Hoss