Frequency of Full reindex on SolrCloud

[bhavin v <bh...@gmail.com>]

Hi Guys,

How often do I need to run full reindex on SolrCloud? It takes more than 12
hours for full reindex to run and we run it every night but is it really
necessary to do it as delta runs correctly.

New data comes in at the rate of 2000 documents on every delta per 30
seconds.

Total index size : 20GB Solr: 6.5.2
ᐧ

Re: Frequency of Full reindex on SolrCloud

[Joe Obernberger <jo...@gmail.com>]

Almost never.  I would only run a re-index for newer versions (such as 
6.5.2 to 7.2) that have a required feature or schema changes such as 
changing the type of an existing field (int to string for example).  Not 
sure what you mean by 'every delta', but I would assume you just mean 
new data?  If so, then re-indexing is not necessary.

-Joe


On 1/2/2018 10:41 AM, bhavin v wrote:
> Hi Guys,
>
> How often do I need to run full reindex on SolrCloud? It takes more than 12
> hours for full reindex to run and we run it every night but is it really
> necessary to do it as delta runs correctly.
>
> New data comes in at the rate of 2000 documents on every delta per 30
> seconds.
>
> Total index size : 20GB Solr: 6.5.2
> ᐧ
>
>
> ---
> This email has been checked for viruses by AVG.
> http://www.avg.com
>

Re: Frequency of Full reindex on SolrCloud

[Shawn Heisey <ap...@elyograg.org>]

On 1/2/2018 8:41 AM, bhavin v wrote:
> How often do I need to run full reindex on SolrCloud? It takes more than 12
> hours for full reindex to run and we run it every night but is it really
> necessary to do it as delta runs correctly.
>
> New data comes in at the rate of 2000 documents on every delta per 30
> seconds.
>
> Total index size : 20GB Solr: 6.5.2

Only you can answer this question.  We know nothing about your dataset.

If each data record in your source system does not change once it has
been created, then you should only need to do a full reindex when you
change something about Solr's index configuration that requires a
reindex.  Most changes to the schema do require a reindex.  Other config
changes might require a reindex.

If the data for an existing record in your source system can change,
then to avoid a full reindex, you would need some way for the delta
updates to notice that an existing record has been updated so it can be
indexed again.  A proper uniqueKey field in the schema is required for
this to work without a full reindex.

Thanks,
Shawn