Auto-optimization for Solr indexes

[Zheng Lin Edwin Yeo <ed...@gmail.com>]

Hi,

I would like to find out, will it be good to do write a script to do an
auto-opitmization of the indexes at a certain time every day? Is there any
advantage to do so?

I found that optimization can reduce the index size by quite a
signification amount, and allow the searching of the index to run faster.
But will there be advantage if we do the optimization every day?

I'm using Solr 5.3.0

Regards,
Edwin

Re: Auto-optimization for Solr indexes

[Zheng Lin Edwin Yeo <ed...@gmail.com>]

Thanks Erick!

Yes, most likely I'll have to do sharing soon before the index size grows
too big.

Regards,
Edwin

On 21 December 2015 at 13:18, Rahul Ramesh <rr...@gmail.com> wrote:

> Thanks Erick!
>
> Rahul
>
> On Mon, Dec 21, 2015 at 10:07 AM, Erick Erickson <er...@gmail.com>
> wrote:
>
> > Rahul:
> >
> > bq:  we dont want the index sizes to grow too large and auto optimzie to
> > kick in
> >
> > Not what quite what's going on. There is no "auto optimize". What
> > there is is background merging that will take _some_ segments and
> > merge them together. Very occasionally this will be the same as a full
> > optimize if it just happens that "some" means all the segments.
> >
> > bq: recovery takes a bit more time when it is not optimized
> >
> > I'd be interested in formal measurements here. A recovery that copied
> > the _entire_ index down from the leader shouldn't really have that
> > much be different between an optimized and non-optimized index, but
> > all things are possible. If the recovery is a "peer sync" it shouldn't
> > matter at all.
> >
> > If you're continually adding documents that _replace_ older documents,
> > optimizing will recover any "holes" left by the old updated docs. An
> > update is really a mark-as-deleted for the old version and a re-index
> > of the new. Since segments are write-once, the old data is left there
> > until the segment is merged. Now, one of the bits of information that
> > goes into deciding whether to merge a segment or not is the size.
> > Another is the percentage of deleted docs. When you optimize, you get
> > one huge segment. Now you have to update a lot of docs for that
> > segment to have a large percentage of deleted documents and be merged,
> > thus wasting space and memory.
> >
> > So it's a tradeoff. But if you're getting satisfactory performance
> > from what you have now, there's no reason to change.
> >
> > Here's a wonderful video about the process. you want the third one
> > down (TieredMergePolicy) as that's the default.
> >
> >
> >
> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
> >
> > Best,
> > Erick
> >
> > On Sun, Dec 20, 2015 at 8:26 PM, Rahul Ramesh <rr...@gmail.com>
> wrote:
> > > Hi Erick,
> > > We index around several million documents/ day and we optimize everyday
> > > when the relative load is low. The reason we optimize is, we dont want
> > the
> > > index sizes to grow too large and auto optimzie to kick in. When auto
> > > optimize kicks in, it results in unpredictable performance as it is CPU
> > and
> > > IO intensive.
> > >
> > > In older solr (4.2), when the segment size grows too large, insertion
> > used
> > > to fail .  Have we seen this problem in solr cloud?
> > >
> > > Also, we have observed, recovery takes a bit more time when it is not
> > > optimized. We dont have any quantitative measurement for the same. Its
> > just
> > > an observation. Is this correct observation?
> > >
> > > If we optimize it every day, the indexes will not be skewed right?
> > >
> > > Please let me know if my understanding is correct.
> > >
> > > Regards,
> > > Rahul
> > >
> > > On Mon, Dec 21, 2015 at 9:54 AM, Erick Erickson <
> erickerickson@gmail.com
> > >
> > > wrote:
> > >
> > >> You'll probably have to shard before you get to the TB range. At that
> > >> point, all the optimization is done individually on each shard so it
> > >> really doesn't matter how many shards you have.
> > >>
> > >> Just issuing
> > >> http://solr:port/solr/collection/update?optimize=true
> > >>
> > >> is sufficient, that'll forward the optimize command to all the shards
> > >> in the collection.
> > >>
> > >> Best,
> > >> Erick
> > >>
> > >> On Sun, Dec 20, 2015 at 8:19 PM, Zheng Lin Edwin Yeo
> > >> <ed...@gmail.com> wrote:
> > >> > Thanks for your information Erick.
> > >> >
> > >> > We have yet to decide how often we will update the index to include
> > new
> > >> > documents that came in. Let's say we update the index once a day,
> then
> > >> when
> > >> > the indexed is updated, we do the optimization (this will be done at
> > >> night
> > >> > when there are not many users using the system).
> > >> > But my index size will probably grow quite big (potentially can go
> up
> > to
> > >> > more than 1TB in the future), so does that have to be taken into
> > >> > consideration too?
> > >> >
> > >> > Regards,
> > >> > Edwin
> > >> >
> > >> >
> > >> > On 21 December 2015 at 12:12, Erick Erickson <
> erickerickson@gmail.com
> > >
> > >> > wrote:
> > >> >
> > >> >> Much depends on how often the index is updated. If your index only
> > >> >> changes, say, once a day then it's probably a good idea. If you're
> > >> >> constantly updating your index, then I'd recommend that you do
> _not_
> > >> >> optimize.
> > >> >>
> > >> >> Optimizing will create one large segment. That segment will be
> > >> >> unlikely to be merged since it is so large relative to other
> segments
> > >> >> for quite a while, resulting in significant wasted space. So if
> > you're
> > >> >> regularly indexing documents that _replace_ existing documents,
> this
> > >> >> will skew your index.
> > >> >>
> > >> >> Bottom line:
> > >> >> If you have a relatively static index the you can build and then
> use
> > >> >> for an extended time (as in 12 hours plus) it can be worth the time
> > to
> > >> >> optimize. Otherwise I wouldn't bother.
> > >> >>
> > >> >> Best,
> > >> >> Erick
> > >> >>
> > >> >> On Sun, Dec 20, 2015 at 7:57 PM, Zheng Lin Edwin Yeo
> > >> >> <ed...@gmail.com> wrote:
> > >> >> > Hi,
> > >> >> >
> > >> >> > I would like to find out, will it be good to do write a script to
> > do
> > >> an
> > >> >> > auto-opitmization of the indexes at a certain time every day? Is
> > there
> > >> >> any
> > >> >> > advantage to do so?
> > >> >> >
> > >> >> > I found that optimization can reduce the index size by quite a
> > >> >> > signification amount, and allow the searching of the index to run
> > >> faster.
> > >> >> > But will there be advantage if we do the optimization every day?
> > >> >> >
> > >> >> > I'm using Solr 5.3.0
> > >> >> >
> > >> >> > Regards,
> > >> >> > Edwin
> > >> >>
> > >>
> >
>

Re: Auto-optimization for Solr indexes

[Rahul Ramesh <rr...@gmail.com>]

Thanks Erick!

Rahul

On Mon, Dec 21, 2015 at 10:07 AM, Erick Erickson <er...@gmail.com>
wrote:

> Rahul:
>
> bq:  we dont want the index sizes to grow too large and auto optimzie to
> kick in
>
> Not what quite what's going on. There is no "auto optimize". What
> there is is background merging that will take _some_ segments and
> merge them together. Very occasionally this will be the same as a full
> optimize if it just happens that "some" means all the segments.
>
> bq: recovery takes a bit more time when it is not optimized
>
> I'd be interested in formal measurements here. A recovery that copied
> the _entire_ index down from the leader shouldn't really have that
> much be different between an optimized and non-optimized index, but
> all things are possible. If the recovery is a "peer sync" it shouldn't
> matter at all.
>
> If you're continually adding documents that _replace_ older documents,
> optimizing will recover any "holes" left by the old updated docs. An
> update is really a mark-as-deleted for the old version and a re-index
> of the new. Since segments are write-once, the old data is left there
> until the segment is merged. Now, one of the bits of information that
> goes into deciding whether to merge a segment or not is the size.
> Another is the percentage of deleted docs. When you optimize, you get
> one huge segment. Now you have to update a lot of docs for that
> segment to have a large percentage of deleted documents and be merged,
> thus wasting space and memory.
>
> So it's a tradeoff. But if you're getting satisfactory performance
> from what you have now, there's no reason to change.
>
> Here's a wonderful video about the process. you want the third one
> down (TieredMergePolicy) as that's the default.
>
>
> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
>
> Best,
> Erick
>
> On Sun, Dec 20, 2015 at 8:26 PM, Rahul Ramesh <rr...@gmail.com> wrote:
> > Hi Erick,
> > We index around several million documents/ day and we optimize everyday
> > when the relative load is low. The reason we optimize is, we dont want
> the
> > index sizes to grow too large and auto optimzie to kick in. When auto
> > optimize kicks in, it results in unpredictable performance as it is CPU
> and
> > IO intensive.
> >
> > In older solr (4.2), when the segment size grows too large, insertion
> used
> > to fail .  Have we seen this problem in solr cloud?
> >
> > Also, we have observed, recovery takes a bit more time when it is not
> > optimized. We dont have any quantitative measurement for the same. Its
> just
> > an observation. Is this correct observation?
> >
> > If we optimize it every day, the indexes will not be skewed right?
> >
> > Please let me know if my understanding is correct.
> >
> > Regards,
> > Rahul
> >
> > On Mon, Dec 21, 2015 at 9:54 AM, Erick Erickson <erickerickson@gmail.com
> >
> > wrote:
> >
> >> You'll probably have to shard before you get to the TB range. At that
> >> point, all the optimization is done individually on each shard so it
> >> really doesn't matter how many shards you have.
> >>
> >> Just issuing
> >> http://solr:port/solr/collection/update?optimize=true
> >>
> >> is sufficient, that'll forward the optimize command to all the shards
> >> in the collection.
> >>
> >> Best,
> >> Erick
> >>
> >> On Sun, Dec 20, 2015 at 8:19 PM, Zheng Lin Edwin Yeo
> >> <ed...@gmail.com> wrote:
> >> > Thanks for your information Erick.
> >> >
> >> > We have yet to decide how often we will update the index to include
> new
> >> > documents that came in. Let's say we update the index once a day, then
> >> when
> >> > the indexed is updated, we do the optimization (this will be done at
> >> night
> >> > when there are not many users using the system).
> >> > But my index size will probably grow quite big (potentially can go up
> to
> >> > more than 1TB in the future), so does that have to be taken into
> >> > consideration too?
> >> >
> >> > Regards,
> >> > Edwin
> >> >
> >> >
> >> > On 21 December 2015 at 12:12, Erick Erickson <erickerickson@gmail.com
> >
> >> > wrote:
> >> >
> >> >> Much depends on how often the index is updated. If your index only
> >> >> changes, say, once a day then it's probably a good idea. If you're
> >> >> constantly updating your index, then I'd recommend that you do _not_
> >> >> optimize.
> >> >>
> >> >> Optimizing will create one large segment. That segment will be
> >> >> unlikely to be merged since it is so large relative to other segments
> >> >> for quite a while, resulting in significant wasted space. So if
> you're
> >> >> regularly indexing documents that _replace_ existing documents, this
> >> >> will skew your index.
> >> >>
> >> >> Bottom line:
> >> >> If you have a relatively static index the you can build and then use
> >> >> for an extended time (as in 12 hours plus) it can be worth the time
> to
> >> >> optimize. Otherwise I wouldn't bother.
> >> >>
> >> >> Best,
> >> >> Erick
> >> >>
> >> >> On Sun, Dec 20, 2015 at 7:57 PM, Zheng Lin Edwin Yeo
> >> >> <ed...@gmail.com> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > I would like to find out, will it be good to do write a script to
> do
> >> an
> >> >> > auto-opitmization of the indexes at a certain time every day? Is
> there
> >> >> any
> >> >> > advantage to do so?
> >> >> >
> >> >> > I found that optimization can reduce the index size by quite a
> >> >> > signification amount, and allow the searching of the index to run
> >> faster.
> >> >> > But will there be advantage if we do the optimization every day?
> >> >> >
> >> >> > I'm using Solr 5.3.0
> >> >> >
> >> >> > Regards,
> >> >> > Edwin
> >> >>
> >>
>

Re: Auto-optimization for Solr indexes

[Erick Erickson <er...@gmail.com>]

Rahul:

bq:  we dont want the index sizes to grow too large and auto optimzie to kick in

Not what quite what's going on. There is no "auto optimize". What
there is is background merging that will take _some_ segments and
merge them together. Very occasionally this will be the same as a full
optimize if it just happens that "some" means all the segments.

bq: recovery takes a bit more time when it is not optimized

I'd be interested in formal measurements here. A recovery that copied
the _entire_ index down from the leader shouldn't really have that
much be different between an optimized and non-optimized index, but
all things are possible. If the recovery is a "peer sync" it shouldn't
matter at all.

If you're continually adding documents that _replace_ older documents,
optimizing will recover any "holes" left by the old updated docs. An
update is really a mark-as-deleted for the old version and a re-index
of the new. Since segments are write-once, the old data is left there
until the segment is merged. Now, one of the bits of information that
goes into deciding whether to merge a segment or not is the size.
Another is the percentage of deleted docs. When you optimize, you get
one huge segment. Now you have to update a lot of docs for that
segment to have a large percentage of deleted documents and be merged,
thus wasting space and memory.

So it's a tradeoff. But if you're getting satisfactory performance
from what you have now, there's no reason to change.

Here's a wonderful video about the process. you want the third one
down (TieredMergePolicy) as that's the default.

http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html

Best,
Erick

On Sun, Dec 20, 2015 at 8:26 PM, Rahul Ramesh <rr...@gmail.com> wrote:
> Hi Erick,
> We index around several million documents/ day and we optimize everyday
> when the relative load is low. The reason we optimize is, we dont want the
> index sizes to grow too large and auto optimzie to kick in. When auto
> optimize kicks in, it results in unpredictable performance as it is CPU and
> IO intensive.
>
> In older solr (4.2), when the segment size grows too large, insertion used
> to fail .  Have we seen this problem in solr cloud?
>
> Also, we have observed, recovery takes a bit more time when it is not
> optimized. We dont have any quantitative measurement for the same. Its just
> an observation. Is this correct observation?
>
> If we optimize it every day, the indexes will not be skewed right?
>
> Please let me know if my understanding is correct.
>
> Regards,
> Rahul
>
> On Mon, Dec 21, 2015 at 9:54 AM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> You'll probably have to shard before you get to the TB range. At that
>> point, all the optimization is done individually on each shard so it
>> really doesn't matter how many shards you have.
>>
>> Just issuing
>> http://solr:port/solr/collection/update?optimize=true
>>
>> is sufficient, that'll forward the optimize command to all the shards
>> in the collection.
>>
>> Best,
>> Erick
>>
>> On Sun, Dec 20, 2015 at 8:19 PM, Zheng Lin Edwin Yeo
>> <ed...@gmail.com> wrote:
>> > Thanks for your information Erick.
>> >
>> > We have yet to decide how often we will update the index to include new
>> > documents that came in. Let's say we update the index once a day, then
>> when
>> > the indexed is updated, we do the optimization (this will be done at
>> night
>> > when there are not many users using the system).
>> > But my index size will probably grow quite big (potentially can go up to
>> > more than 1TB in the future), so does that have to be taken into
>> > consideration too?
>> >
>> > Regards,
>> > Edwin
>> >
>> >
>> > On 21 December 2015 at 12:12, Erick Erickson <er...@gmail.com>
>> > wrote:
>> >
>> >> Much depends on how often the index is updated. If your index only
>> >> changes, say, once a day then it's probably a good idea. If you're
>> >> constantly updating your index, then I'd recommend that you do _not_
>> >> optimize.
>> >>
>> >> Optimizing will create one large segment. That segment will be
>> >> unlikely to be merged since it is so large relative to other segments
>> >> for quite a while, resulting in significant wasted space. So if you're
>> >> regularly indexing documents that _replace_ existing documents, this
>> >> will skew your index.
>> >>
>> >> Bottom line:
>> >> If you have a relatively static index the you can build and then use
>> >> for an extended time (as in 12 hours plus) it can be worth the time to
>> >> optimize. Otherwise I wouldn't bother.
>> >>
>> >> Best,
>> >> Erick
>> >>
>> >> On Sun, Dec 20, 2015 at 7:57 PM, Zheng Lin Edwin Yeo
>> >> <ed...@gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > I would like to find out, will it be good to do write a script to do
>> an
>> >> > auto-opitmization of the indexes at a certain time every day? Is there
>> >> any
>> >> > advantage to do so?
>> >> >
>> >> > I found that optimization can reduce the index size by quite a
>> >> > signification amount, and allow the searching of the index to run
>> faster.
>> >> > But will there be advantage if we do the optimization every day?
>> >> >
>> >> > I'm using Solr 5.3.0
>> >> >
>> >> > Regards,
>> >> > Edwin
>> >>
>>

Re: Auto-optimization for Solr indexes

[Rahul Ramesh <rr...@gmail.com>]

Hi Erick,
We index around several million documents/ day and we optimize everyday
when the relative load is low. The reason we optimize is, we dont want the
index sizes to grow too large and auto optimzie to kick in. When auto
optimize kicks in, it results in unpredictable performance as it is CPU and
IO intensive.

In older solr (4.2), when the segment size grows too large, insertion used
to fail .  Have we seen this problem in solr cloud?

Also, we have observed, recovery takes a bit more time when it is not
optimized. We dont have any quantitative measurement for the same. Its just
an observation. Is this correct observation?

If we optimize it every day, the indexes will not be skewed right?

Please let me know if my understanding is correct.

Regards,
Rahul

On Mon, Dec 21, 2015 at 9:54 AM, Erick Erickson <er...@gmail.com>
wrote:

> You'll probably have to shard before you get to the TB range. At that
> point, all the optimization is done individually on each shard so it
> really doesn't matter how many shards you have.
>
> Just issuing
> http://solr:port/solr/collection/update?optimize=true
>
> is sufficient, that'll forward the optimize command to all the shards
> in the collection.
>
> Best,
> Erick
>
> On Sun, Dec 20, 2015 at 8:19 PM, Zheng Lin Edwin Yeo
> <ed...@gmail.com> wrote:
> > Thanks for your information Erick.
> >
> > We have yet to decide how often we will update the index to include new
> > documents that came in. Let's say we update the index once a day, then
> when
> > the indexed is updated, we do the optimization (this will be done at
> night
> > when there are not many users using the system).
> > But my index size will probably grow quite big (potentially can go up to
> > more than 1TB in the future), so does that have to be taken into
> > consideration too?
> >
> > Regards,
> > Edwin
> >
> >
> > On 21 December 2015 at 12:12, Erick Erickson <er...@gmail.com>
> > wrote:
> >
> >> Much depends on how often the index is updated. If your index only
> >> changes, say, once a day then it's probably a good idea. If you're
> >> constantly updating your index, then I'd recommend that you do _not_
> >> optimize.
> >>
> >> Optimizing will create one large segment. That segment will be
> >> unlikely to be merged since it is so large relative to other segments
> >> for quite a while, resulting in significant wasted space. So if you're
> >> regularly indexing documents that _replace_ existing documents, this
> >> will skew your index.
> >>
> >> Bottom line:
> >> If you have a relatively static index the you can build and then use
> >> for an extended time (as in 12 hours plus) it can be worth the time to
> >> optimize. Otherwise I wouldn't bother.
> >>
> >> Best,
> >> Erick
> >>
> >> On Sun, Dec 20, 2015 at 7:57 PM, Zheng Lin Edwin Yeo
> >> <ed...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I would like to find out, will it be good to do write a script to do
> an
> >> > auto-opitmization of the indexes at a certain time every day? Is there
> >> any
> >> > advantage to do so?
> >> >
> >> > I found that optimization can reduce the index size by quite a
> >> > signification amount, and allow the searching of the index to run
> faster.
> >> > But will there be advantage if we do the optimization every day?
> >> >
> >> > I'm using Solr 5.3.0
> >> >
> >> > Regards,
> >> > Edwin
> >>
>

Re: Auto-optimization for Solr indexes

[Erick Erickson <er...@gmail.com>]

You'll probably have to shard before you get to the TB range. At that
point, all the optimization is done individually on each shard so it
really doesn't matter how many shards you have.

Just issuing
http://solr:port/solr/collection/update?optimize=true

is sufficient, that'll forward the optimize command to all the shards
in the collection.

Best,
Erick

On Sun, Dec 20, 2015 at 8:19 PM, Zheng Lin Edwin Yeo
<ed...@gmail.com> wrote:
> Thanks for your information Erick.
>
> We have yet to decide how often we will update the index to include new
> documents that came in. Let's say we update the index once a day, then when
> the indexed is updated, we do the optimization (this will be done at night
> when there are not many users using the system).
> But my index size will probably grow quite big (potentially can go up to
> more than 1TB in the future), so does that have to be taken into
> consideration too?
>
> Regards,
> Edwin
>
>
> On 21 December 2015 at 12:12, Erick Erickson <er...@gmail.com>
> wrote:
>
>> Much depends on how often the index is updated. If your index only
>> changes, say, once a day then it's probably a good idea. If you're
>> constantly updating your index, then I'd recommend that you do _not_
>> optimize.
>>
>> Optimizing will create one large segment. That segment will be
>> unlikely to be merged since it is so large relative to other segments
>> for quite a while, resulting in significant wasted space. So if you're
>> regularly indexing documents that _replace_ existing documents, this
>> will skew your index.
>>
>> Bottom line:
>> If you have a relatively static index the you can build and then use
>> for an extended time (as in 12 hours plus) it can be worth the time to
>> optimize. Otherwise I wouldn't bother.
>>
>> Best,
>> Erick
>>
>> On Sun, Dec 20, 2015 at 7:57 PM, Zheng Lin Edwin Yeo
>> <ed...@gmail.com> wrote:
>> > Hi,
>> >
>> > I would like to find out, will it be good to do write a script to do an
>> > auto-opitmization of the indexes at a certain time every day? Is there
>> any
>> > advantage to do so?
>> >
>> > I found that optimization can reduce the index size by quite a
>> > signification amount, and allow the searching of the index to run faster.
>> > But will there be advantage if we do the optimization every day?
>> >
>> > I'm using Solr 5.3.0
>> >
>> > Regards,
>> > Edwin
>>

Re: Auto-optimization for Solr indexes

[Zheng Lin Edwin Yeo <ed...@gmail.com>]

Thanks for your information Erick.

We have yet to decide how often we will update the index to include new
documents that came in. Let's say we update the index once a day, then when
the indexed is updated, we do the optimization (this will be done at night
when there are not many users using the system).
But my index size will probably grow quite big (potentially can go up to
more than 1TB in the future), so does that have to be taken into
consideration too?

Regards,
Edwin


On 21 December 2015 at 12:12, Erick Erickson <er...@gmail.com>
wrote:

> Much depends on how often the index is updated. If your index only
> changes, say, once a day then it's probably a good idea. If you're
> constantly updating your index, then I'd recommend that you do _not_
> optimize.
>
> Optimizing will create one large segment. That segment will be
> unlikely to be merged since it is so large relative to other segments
> for quite a while, resulting in significant wasted space. So if you're
> regularly indexing documents that _replace_ existing documents, this
> will skew your index.
>
> Bottom line:
> If you have a relatively static index the you can build and then use
> for an extended time (as in 12 hours plus) it can be worth the time to
> optimize. Otherwise I wouldn't bother.
>
> Best,
> Erick
>
> On Sun, Dec 20, 2015 at 7:57 PM, Zheng Lin Edwin Yeo
> <ed...@gmail.com> wrote:
> > Hi,
> >
> > I would like to find out, will it be good to do write a script to do an
> > auto-opitmization of the indexes at a certain time every day? Is there
> any
> > advantage to do so?
> >
> > I found that optimization can reduce the index size by quite a
> > signification amount, and allow the searching of the index to run faster.
> > But will there be advantage if we do the optimization every day?
> >
> > I'm using Solr 5.3.0
> >
> > Regards,
> > Edwin
>

Re: Auto-optimization for Solr indexes

[Erick Erickson <er...@gmail.com>]

Much depends on how often the index is updated. If your index only
changes, say, once a day then it's probably a good idea. If you're
constantly updating your index, then I'd recommend that you do _not_
optimize.

Optimizing will create one large segment. That segment will be
unlikely to be merged since it is so large relative to other segments
for quite a while, resulting in significant wasted space. So if you're
regularly indexing documents that _replace_ existing documents, this
will skew your index.

Bottom line:
If you have a relatively static index the you can build and then use
for an extended time (as in 12 hours plus) it can be worth the time to
optimize. Otherwise I wouldn't bother.

Best,
Erick

On Sun, Dec 20, 2015 at 7:57 PM, Zheng Lin Edwin Yeo
<ed...@gmail.com> wrote:
> Hi,
>
> I would like to find out, will it be good to do write a script to do an
> auto-opitmization of the indexes at a certain time every day? Is there any
> advantage to do so?
>
> I found that optimization can reduce the index size by quite a
> signification amount, and allow the searching of the index to run faster.
> But will there be advantage if we do the optimization every day?
>
> I'm using Solr 5.3.0
>
> Regards,
> Edwin