You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Midas A <te...@gmail.com> on 2019/06/04 06:18:02 UTC
not able to optimize
Hi ,
Index size is 400GB. we used master slave architecture .
commit is taking time while not able to perform optimize .
what should i do .
Re: not able to optimize
Posted by Walter Underwood <wu...@wunderwood.org>.
Correct, do not optimize.
“Optimize” was a bad choice for this action. It is a forced merge.
With master/slave, it means the slaves must always copy the entire
400 GB index. Without optimize, they would only need to copy the
changed segments.
Solr automatically merges segments for you.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Jun 4, 2019, at 2:34 AM, Midas A <te...@gmail.com> wrote:
>
> So we should not optimize our index ?
>
> On Tue, Jun 4, 2019 at 2:37 PM Toke Eskildsen <to...@kb.dk> wrote:
>
>> On Tue, 2019-06-04 at 11:48 +0530, Midas A wrote:
>>> Index size is 400GB. we used master slave architecture .
>>>
>>> commit is taking time while not able to perform optimize .
>>
>> Why do you want to optimize in the first place? What are you hoping to
>> achieve?
>>
>> There should be an error message in your Solr log from the failed
>> optimize, so see if you can find that. At least for some Solr versions,
>> optimize can be quite memory hungry so a very quick guess is that you
>> have hit an OutOfMemory. Not enough disk space is also a guess: Make
>> sure you have 2*index size free space before optimizing as worst case
>> for storage usage during optimize is a total of 3*index size.
>>
>> - Toke Eskildsen, Royal Danish Library
>>
>>
>>
Re: not able to optimize
Posted by Midas A <te...@gmail.com>.
400GB index is good ?
Are we should shard it .?
When we should start caring about inex size .?
On Tue, Jun 4, 2019 at 3:04 PM Midas A <te...@gmail.com> wrote:
> So we should not optimize our index ?
>
> On Tue, Jun 4, 2019 at 2:37 PM Toke Eskildsen <to...@kb.dk> wrote:
>
>> On Tue, 2019-06-04 at 11:48 +0530, Midas A wrote:
>> > Index size is 400GB. we used master slave architecture .
>> >
>> > commit is taking time while not able to perform optimize .
>>
>> Why do you want to optimize in the first place? What are you hoping to
>> achieve?
>>
>> There should be an error message in your Solr log from the failed
>> optimize, so see if you can find that. At least for some Solr versions,
>> optimize can be quite memory hungry so a very quick guess is that you
>> have hit an OutOfMemory. Not enough disk space is also a guess: Make
>> sure you have 2*index size free space before optimizing as worst case
>> for storage usage during optimize is a total of 3*index size.
>>
>> - Toke Eskildsen, Royal Danish Library
>>
>>
>>
Re: not able to optimize
Posted by Midas A <te...@gmail.com>.
So we should not optimize our index ?
On Tue, Jun 4, 2019 at 2:37 PM Toke Eskildsen <to...@kb.dk> wrote:
> On Tue, 2019-06-04 at 11:48 +0530, Midas A wrote:
> > Index size is 400GB. we used master slave architecture .
> >
> > commit is taking time while not able to perform optimize .
>
> Why do you want to optimize in the first place? What are you hoping to
> achieve?
>
> There should be an error message in your Solr log from the failed
> optimize, so see if you can find that. At least for some Solr versions,
> optimize can be quite memory hungry so a very quick guess is that you
> have hit an OutOfMemory. Not enough disk space is also a guess: Make
> sure you have 2*index size free space before optimizing as worst case
> for storage usage during optimize is a total of 3*index size.
>
> - Toke Eskildsen, Royal Danish Library
>
>
>
Re: not able to optimize
Posted by Toke Eskildsen <to...@kb.dk>.
On Tue, 2019-06-04 at 11:48 +0530, Midas A wrote:
> Index size is 400GB. we used master slave architecture .
>
> commit is taking time while not able to perform optimize .
Why do you want to optimize in the first place? What are you hoping to
achieve?
There should be an error message in your Solr log from the failed
optimize, so see if you can find that. At least for some Solr versions,
optimize can be quite memory hungry so a very quick guess is that you
have hit an OutOfMemory. Not enough disk space is also a guess: Make
sure you have 2*index size free space before optimizing as worst case
for storage usage during optimize is a total of 3*index size.
- Toke Eskildsen, Royal Danish Library
Re: not able to optimize
Posted by Gus Heck <gu...@gmail.com>.
Hi Midas,
Your question will probably attract more useful answers if you provide
better details. What version of solr, How many nodes, and any associated
error messages from the logs. I see you asking questions that nobody can
answer because we don't know the details of your system, or why you are
asking. It's important to remember, that people on the list don't know
anything about your system unless you tell them. For example one reason
version matters is that, optimize is sometimes useful, but in some older
versions of solr it can cause also cause issues (depending on use case)
https://lucidworks.com/2017/10/13/segment-merging-deleted-documents-optimize-may-bad/
)
This may be useful reading for you:
http://opensourcestrategies.blogspot.com/2005/09/how-to-get-support-from-open-source.html
-Gus
On Tue, Jun 4, 2019 at 2:18 AM Midas A <te...@gmail.com> wrote:
> Hi ,
> Index size is 400GB. we used master slave architecture .
>
> commit is taking time while not able to perform optimize .
>
> what should i do .
>
--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)