You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by heaven <ah...@gmail.com> on 2013/05/25 13:56:35 UTC
Re: Overlapping onDeckSearchers=2
Hi, I am getting this warning with auto commits. My application does not send
commits. I am optimizing the index once per day and seems like this warning
appear every time the optimization process is launched.
Best,
Alex
--
View this message in context: http://lucene.472066.n3.nabble.com/Overlapping-onDeckSearchers-2-tp772556p4066050.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Overlapping onDeckSearchers=2
Posted by Markus Jelsma <ma...@openindex.io>.
forceMerge is very useful if you delete a significant portion of an index. It can take a very long time before any merge policy decides to finally merge them all away, especially for a static or infrequently changing index. Also, having a lot of deleted docs in the index can be an issue if your similarity uses maxDoc for IDF.
The cost is also much lower when using SSD's, forceMerging a 1GB core is a matter of seconds.
-----Original message-----
> From:Yonik Seeley <yo...@lucidworks.com>
> Sent: Mon 27-May-2013 15:47
> To: solr-user@lucene.apache.org
> Subject: Re: Overlapping onDeckSearchers=2
>
> On Mon, May 27, 2013 at 7:11 AM, Jack Krupansky <ja...@basetechnology.com> wrote:
> > The intent is that optimize is obsolete and should no longer be used
>
> That's incorrect.
>
> People need to understand the cost of optimize, and that it's use is optional.
> It's up to the developer to figure out of the benefits of calling
> optimize outweigh the costs in their particular situations.
>
> The wiki currently says:
> """
> An optimize is like a hard commit except that it forces all of the
> index segments to be merged into a single segment first. Depending on
> the use cases, this operation should be performed infrequently (like
> nightly), if at all, since it is very expensive and involves reading
> and re-writing the entire index. Segments are normally merged over
> time anyway (as determined by the merge policy), and optimize just
> forces these merges to occur immediately.
> """
>
> -Yonik
> http://lucidworks.com
>
Re: Overlapping onDeckSearchers=2
Posted by Jack Krupansky <ja...@basetechnology.com>.
As the wiki does say: "if at all ... Segments are normally merged over time
anyway (as determined by the merge policy), and optimize just forces these
merges to occur immediately."
So, the only real question here is if the optimize really does lie outside
the "if at all" category and whether "Segments are normally merged over time
anyway" is in fact not good enough.
This is why I referred to "the intent" - whether the actual reality of a
specific Solr application does align with the expectation that "Segments are
normally merged over time anyway". But, start with the presumption that
merge policy does eliminate the need for optimize.
As a general proposition:
1. Try to avoid using optimize "if at all" possible. Let merge policy do its
thing.
2. Try to take a server out offline while optimizing if an optimize is
really absolutely needed.
3. Try to understand why #1 is not sufficient and resolve the cause(s), so
that optimize is no longer needed.
-- Jack Krupansky
-----Original Message-----
From: Yonik Seeley
Sent: Monday, May 27, 2013 9:46 AM
To: solr-user@lucene.apache.org
Subject: Re: Overlapping onDeckSearchers=2
On Mon, May 27, 2013 at 7:11 AM, Jack Krupansky <ja...@basetechnology.com>
wrote:
> The intent is that optimize is obsolete and should no longer be used
That's incorrect.
People need to understand the cost of optimize, and that it's use is
optional.
It's up to the developer to figure out of the benefits of calling
optimize outweigh the costs in their particular situations.
The wiki currently says:
"""
An optimize is like a hard commit except that it forces all of the
index segments to be merged into a single segment first. Depending on
the use cases, this operation should be performed infrequently (like
nightly), if at all, since it is very expensive and involves reading
and re-writing the entire index. Segments are normally merged over
time anyway (as determined by the merge policy), and optimize just
forces these merges to occur immediately.
"""
-Yonik
http://lucidworks.com
Re: Overlapping onDeckSearchers=2
Posted by Yonik Seeley <yo...@lucidworks.com>.
On Mon, May 27, 2013 at 7:11 AM, Jack Krupansky <ja...@basetechnology.com> wrote:
> The intent is that optimize is obsolete and should no longer be used
That's incorrect.
People need to understand the cost of optimize, and that it's use is optional.
It's up to the developer to figure out of the benefits of calling
optimize outweigh the costs in their particular situations.
The wiki currently says:
"""
An optimize is like a hard commit except that it forces all of the
index segments to be merged into a single segment first. Depending on
the use cases, this operation should be performed infrequently (like
nightly), if at all, since it is very expensive and involves reading
and re-writing the entire index. Segments are normally merged over
time anyway (as determined by the merge policy), and optimize just
forces these merges to occur immediately.
"""
-Yonik
http://lucidworks.com
Re: Overlapping onDeckSearchers=2
Posted by heaven <ah...@gmail.com>.
I am on 4.2.1
@Yonik Seeley I do understand the cost and run it once per 24 hours and
perhaps later this interval will be increased up to a few days.
In general I am optimizing not to merge the fragments but to remove deleted
docs. My index refreshes quickly and number of deleted docs could reach a
few millions per week.
The questions is: if optimize does the same what the hard commit does + some
other optimizations why does Solr schedule a new commit when optimize is in
the process? That's the problem.
Since I am optimizing once per day and all commits are scheduled by Solr
itself:
<autoCommit>
<maxDocs>25000</maxDocs>
<maxTime>300000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>5000</maxTime>
</autoSoftCommit>
If Solr see that an optimization procedure is running it could delay all
scheduled hard commits until optimization is complete. Additionally it could
perform a soft commit (for cases when application need to see the updated
docs in index and fires commits) and run the delayed hard commit when
optimization is complete. It will warm a new searcher but at least will
prevent from multiple searchers warming simultaneously.
Best,
Alex
--
View this message in context: http://lucene.472066.n3.nabble.com/Overlapping-onDeckSearchers-2-tp772556p4066267.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Overlapping onDeckSearchers=2
Posted by Jack Krupansky <ja...@basetechnology.com>.
The intent is that optimize is obsolete and should no longer be used,
especially with tiered merge policy running. In other words, merging should
be occurring on the fly in Lucene now. What release of Solr are you running?
-- Jack Krupansky
-----Original Message-----
From: heaven
Sent: Monday, May 27, 2013 3:51 AM
To: solr-user@lucene.apache.org
Subject: Re: Overlapping onDeckSearchers=2
Hi, thanks for the response. Seems like this is the case because there are
no
any other applications that could fire commit/optimize calls. All commits
are triggered by Solr and the optimize is triggered by a cron task.
Because of all that it looks like a bug in Solr. It probably should not run
commits when optimize is in the process or should do commit before the
optimize. Or should not run optimize when a commit is in the process. Not
sure which exactly scenario happens.
Is there anything I can do to fix this? I don't have too much memory on this
server and since this could lead to double the RAM used it is a serious
issue for me.
Best,
Alex
--
View this message in context:
http://lucene.472066.n3.nabble.com/Overlapping-onDeckSearchers-2-tp772556p4066205.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Overlapping onDeckSearchers=2
Posted by heaven <ah...@gmail.com>.
Hi, thanks for the response. Seems like this is the case because there are no
any other applications that could fire commit/optimize calls. All commits
are triggered by Solr and the optimize is triggered by a cron task.
Because of all that it looks like a bug in Solr. It probably should not run
commits when optimize is in the process or should do commit before the
optimize. Or should not run optimize when a commit is in the process. Not
sure which exactly scenario happens.
Is there anything I can do to fix this? I don't have too much memory on this
server and since this could lead to double the RAM used it is a serious
issue for me.
Best,
Alex
--
View this message in context: http://lucene.472066.n3.nabble.com/Overlapping-onDeckSearchers-2-tp772556p4066205.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Overlapping onDeckSearchers=2
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
This can happen if an auto commit overlaps with the optimize command too.
On Sat, May 25, 2013 at 5:26 PM, heaven <ah...@gmail.com> wrote:
> Hi, I am getting this warning with auto commits. My application does not
> send
> commits. I am optimizing the index once per day and seems like this warning
> appear every time the optimization process is launched.
>
> Best,
> Alex
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Overlapping-onDeckSearchers-2-tp772556p4066050.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Regards,
Shalin Shekhar Mangar.