You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by mganeshs <mg...@live.in> on 2018/02/13 10:03:58 UTC
docvalues set to true, and indexed is false and stored is set to
false
Hi,
If I have set following in the schema
<dynamicField name="Fields_*" type="string" indexed="false"
stored="false" docValues="true"/>
What will be the impact of deleting a single field, "Fields_one" field or
what's the impact of adding a new field "Fields_100" ?
Will the whole document will re-indexed again, or only this field alone will
be deleted and added correspondingly.
Idea here is we are trying to avoid complete re-indexing of document ( as
document would be very huge one and number of documents are also in huge,
and we have a situation, where we may need to add one new dynamic field to
all the documents or to remove a dynamic from all the documents ).
Early responses are really appreciated !
Regards,
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: docvalues set to true, and indexed is false and stored is set to
false
Posted by Emir Arnautović <em...@sematext.com>.
Hi Ganesh,
I cannot confirm for sure, but I would assume that it will not get reindexed, but just segments doc values file rewritten. It is best if you test this and see for yourself.
Regards,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 14 Feb 2018, at 13:09, mganeshs <mg...@live.in> wrote:
>
> Hi Emir,
>
> Thanks for confirming that strField is not considered / available for in
> place updates.
>
> As per documentation, it says...
>
> *An atomic update operation is performed using this approach only when the
> fields to be updated meet these three conditions:
>
> are non-indexed (indexed="false"), non-stored (stored="false"), single
> valued (multiValued="false") numeric docValues (docValues="true") fields;
>
> the _version_ field is also a non-indexed, non-stored single valued
> docValues field; and,
>
> copy targets of updated fields, if any, are also non-indexed, non-stored
> single valued numeric docValues fields.*
>
> Let's consider I have declared following three fields in the schema
>
> <uniqueKey>id</uniqueKey>
> <field name="id" type="string" indexed="true" required="true"/>
> <field name='Field1' type="string" stored="true" indexed="true"
> docValues="false"/>
> <field name='Field2' type="string" stored="true" indexed="true"
> docValues="false"/>
> <field name='Field3' type="int" stored="false" indexed="false"
> docValues="true"/>
>
> With this I am trying to create couple of solr document ( id =1) with only
> Field1 and Field2 and it's also indexed. And I could search the documents
> based on Field1 and Field2
>
> Now after a while, I am adding a new field called Field3 by passing the id
> field ( id=1) and Field3 ( Field3=100 ( which is docvalues field in our case
> ).
>
> What will happen now ? Will the complete document gets re indexed or only
> Field3 get added under docValues ?
>
> Pls confirm.
>
> Regards,
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: docvalues set to true, and indexed is false and stored is set
to false
Posted by mganeshs <mg...@live.in>.
Hi Emir,
Thanks for confirming that strField is not considered / available for in
place updates.
As per documentation, it says...
*An atomic update operation is performed using this approach only when the
fields to be updated meet these three conditions:
are non-indexed (indexed="false"), non-stored (stored="false"), single
valued (multiValued="false") numeric docValues (docValues="true") fields;
the _version_ field is also a non-indexed, non-stored single valued
docValues field; and,
copy targets of updated fields, if any, are also non-indexed, non-stored
single valued numeric docValues fields.*
Let's consider I have declared following three fields in the schema
<uniqueKey>id</uniqueKey>
<field name="id" type="string" indexed="true" required="true"/>
<field name='Field1' type="string" stored="true" indexed="true"
docValues="false"/>
<field name='Field2' type="string" stored="true" indexed="true"
docValues="false"/>
<field name='Field3' type="int" stored="false" indexed="false"
docValues="true"/>
With this I am trying to create couple of solr document ( id =1) with only
Field1 and Field2 and it's also indexed. And I could search the documents
based on Field1 and Field2
Now after a while, I am adding a new field called Field3 by passing the id
field ( id=1) and Field3 ( Field3=100 ( which is docvalues field in our case
).
What will happen now ? Will the complete document gets re indexed or only
Field3 get added under docValues ?
Pls confirm.
Regards,
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: docvalues set to true, and indexed is false and stored is set to
false
Posted by Emir Arnautović <em...@sematext.com>.
Hi Ganesh,
Doc values are enabled for strField and UUID but in place updates are not.
It is not free = according to some discussions on mailing list (did not check the code) in place updates are not update of some value in doc values file but rewrite of doc values file for the segment that it is holding doc that is updated. In case of updating docs that are in larger segment, larger doc values file will be rewritten.
Regards,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 14 Feb 2018, at 04:38, mganeshs <mg...@live.in> wrote:
>
> Hi,
>
> Thanks for clearing.
>
> But as per this link
> <https://lucene.apache.org/solr/guide/6_6/docvalues.html#DocValues-EnablingDocValues>
> (Enabling DocValues) it says that it supports strField and UUID field also.
>
> Again, what you mean by it's not free for large segments. Can you point me
> to some documentation on that ?
>
> Regards,
> Ganesh
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: docvalues set to true, and indexed is false and stored is set
to false
Posted by mganeshs <mg...@live.in>.
Hi,
Thanks for clearing.
But as per this link
<https://lucene.apache.org/solr/guide/6_6/docvalues.html#DocValues-EnablingDocValues>
(Enabling DocValues) it says that it supports strField and UUID field also.
Again, what you mean by it's not free for large segments. Can you point me
to some documentation on that ?
Regards,
Ganesh
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: docvalues set to true, and indexed is false and stored is set to
false
Posted by Emir Arnautović <em...@sematext.com>.
Hi,
It is clearer now, but you mentioned strings in your first mail and in place updates only work for numeric fields. If you meet all conditions, document will not be reindexed, but only doc values rewritten for the segment where in place update happened. Note that this is not free for large segments.
HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 13 Feb 2018, at 13:43, mganeshs <mg...@live.in> wrote:
>
> Hi,
>
> I guess my point is not conceived correctly.
>
> Here I am talking about the field "In Place Updates
> <https://lucene.apache.org/solr/guide/6_6/updating-parts-of-documents.html#UpdatingPartsofDocuments-In-PlaceUpdates>
> "
>
> As per above link, it says that complete document will not be re-indexed
> during updates, if the field is set as docValues="true" and indexed and
> stored is set as false.
>
> But I want to know whether complete document will re-index, when I delete a
> field of type "docvalue" is set as true, but indexed and stored is set as
> false. Also when I add new field of type "docvalue"is set as true, but
> indexed and stored is set as false.
>
> Hope my question is clear now.
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: docvalues set to true, and indexed is false and stored is set
to false
Posted by mganeshs <mg...@live.in>.
Hi,
I guess my point is not conceived correctly.
Here I am talking about the field "In Place Updates
<https://lucene.apache.org/solr/guide/6_6/updating-parts-of-documents.html#UpdatingPartsofDocuments-In-PlaceUpdates>
"
As per above link, it says that complete document will not be re-indexed
during updates, if the field is set as docValues="true" and indexed and
stored is set as false.
But I want to know whether complete document will re-index, when I delete a
field of type "docvalue" is set as true, but indexed and stored is set as
false. Also when I add new field of type "docvalue"is set as true, but
indexed and stored is set as false.
Hope my question is clear now.
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: docvalues set to true, and indexed is false and stored is set to
false
Posted by Emir Arnautović <em...@sematext.com>.
Whenever you send doc to indexing, it is indexed completely and old document with the same id (if one exists) is just flagged as deleted and will be removed from index when segment that it is stored is merged. In case of large segments, it might be never.
The safest option is to do full reindexing on a fresh collection once you change schema.
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 13 Feb 2018, at 11:34, mganeshs <mg...@live.in> wrote:
>
> Hi,
>
> Thanks for quick response.
>
> I forgot to mention that after adding it, I have re-indexed all the data
> with dynamic fields Field_one, Field_two etc.
>
> In that case, by adding new field ( docvalue field ) or removing existing
> docvalue field, Will the whole document will re-indexed again, or only this
> field alone will be deleted and added correspondingly.
>
> Regards,
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: docvalues set to true, and indexed is false and stored is set
to false
Posted by mganeshs <mg...@live.in>.
Hi,
Thanks for quick response.
I forgot to mention that after adding it, I have re-indexed all the data
with dynamic fields Field_one, Field_two etc.
In that case, by adding new field ( docvalue field ) or removing existing
docvalue field, Will the whole document will re-indexed again, or only this
field alone will be deleted and added correspondingly.
Regards,
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: docvalues set to true, and indexed is false and stored is set to
false
Posted by Emir Arnautović <em...@sematext.com>.
Hi,
Changing schema will not do anything by itself. After changes are applied (core reloaded if not used API to update schema) it will use new schema to index new documents. What matters is what you had in index before schema updates. So if you had defined Field_one as string or you had it as number but never used that field, deleting it and letting dynamic field handle from that moment should be fine. Similar goes with adding new field - if you are adding new definition for Field_100 before you used it, it will be ok.
HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 13 Feb 2018, at 11:03, mganeshs <mg...@live.in> wrote:
>
> Hi,
>
> If I have set following in the schema
>
> <dynamicField name="Fields_*" type="string" indexed="false"
> stored="false" docValues="true"/>
>
> What will be the impact of deleting a single field, "Fields_one" field or
> what's the impact of adding a new field "Fields_100" ?
>
> Will the whole document will re-indexed again, or only this field alone will
> be deleted and added correspondingly.
>
> Idea here is we are trying to avoid complete re-indexing of document ( as
> document would be very huge one and number of documents are also in huge,
> and we have a situation, where we may need to add one new dynamic field to
> all the documents or to remove a dynamic from all the documents ).
>
> Early responses are really appreciated !
>
> Regards,
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html