You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Ramsey Haddad (BLOOMBERG/ LONDON)" <rh...@bloomberg.net> on 2020/09/16 11:42:25 UTC
Solr waitForMerges() causing leaderless shard during shutdown
Hi Solr community,
We have been investigating an issue in our solr (7.5.0) setup where the shutdown of our solr node takes quite some time (3-4 minutes) during which we are effectively leaderless.
After investigating and digging deeper we were able to track it down to segment merges which happen before a solr core is closed.
************************************ stack trace when killing the node ************************************
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode):
"Attach Listener" #150736 daemon prio=9 os_prio=0 tid=0x00007f6da4002000 nid=0x13292 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"coreCloseExecutor-22-thread-1" #150733 prio=5 os_prio=0 tid=0x00007f6d54020800 nid=0x11b61 in Object.wait() [0x00007f6c98564000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
~at java.lang.Object.wait(Native Method)
~at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4672)
~- locked <0x00000005499908c0> (a org.apache.solr.update.SolrIndexWriter)
~at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWriter.java:2559)
~- locked <0x00000005499908c0> (a org.apache.solr.update.SolrIndexWriter)
~at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1036)
~at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1078)
~at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:286)
~at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:892)
~at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:105)
~at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:399)
~- locked <0x000000054e150cc0> (a org.apache.solr.update.DefaultSolrCoreState)
~at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:83)
~at org.apache.solr.core.SolrCore.close(SolrCore.java:1574)
~at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:141)
~at org.apache.solr.core.SolrCores$$Lambda$443/1058423472.call(Unknown Source)
~at java.util.concurrent.FutureTask.run(FutureTask.java:266)
************************************************************************************************************
The situation is as follows -
1. The first thing that happens is the request handlers being closed at -
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/core/SolrCore.java#L1588
2. Then it tries to close the index writer via -
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/core/SolrCore.java#L1610
3. When closing the index writer, it waits for any pending merges to finish at -
https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java#L1236
Now, if this waitForMerges() takes a long time (3-4 minutes), the instance won't shut down for the whole of that time, but because of *Step 1* it will stop
accepting any requests.
This becomes a problem when this node has a leader replica and it is stuck on waitForMerges() after closing its reqHandlers. We are in a situation where
the leader is not accepting requests but has not given away the leadership, so we are in a leaderless phase.
This issue triggers when we turnaround our nodes which causes a brief period of leaderless shards which leads to potential data losses.
My question is -
1. How to avoid this situation given that we have big segment sizes and the merging the largest segments is going to take some time.
We do not want to reduce the segment size as it will impact our search performance which is crucial.
2. Should Solr ideally not do the waitForMerges() step before closing the request handlers?
Merge Policy config and segment size -
<mergePolicyFactory class="org.apache.solr.index.SortingMergePolicyFactory">
<str name="sort">time_of_arrival desc</str>
<str name="wrapped.prefix">inner</str>
<str name="inner.class">org.apache.solr.index.TieredMergePolicyFactory</str>
<!-- Override default Solr7 values with the values
we have been using in Solr 4, which allow
more segments to be merged at once and larger
segments to be created.
-->
<int name="inner.maxMergeAtOnce">16</int>
<int name="inner.maxMergedSegmentMB">20480</int>
</mergePolicyFactory>
Re: Solr waitForMerges() causing leaderless shard during shutdown
Posted by Andrzej Białecki <ab...@getopt.org>.
Hi Ramsey,
This is an interesting scenario, I vaguely remember someone (Cao Manh Dat?) on a similar issue - I’m not sure if newer versions of Solr already fixed that but it would be helpful to create a Jira issue to investigate it and verify that it’s indeed fixed in a more recent Solr release.
> On 16 Sep 2020, at 13:42, Ramsey Haddad (BLOOMBERG/ LONDON) <rh...@bloomberg.net> wrote:
>
> Hi Solr community,
>
> We have been investigating an issue in our solr (7.5.0) setup where the shutdown of our solr node takes quite some time (3-4 minutes) during which we are effectively leaderless.
> After investigating and digging deeper we were able to track it down to segment merges which happen before a solr core is closed.
>
> ************************************ stack trace when killing the node ************************************
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode):
>
> "Attach Listener" #150736 daemon prio=9 os_prio=0 tid=0x00007f6da4002000 nid=0x13292 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "coreCloseExecutor-22-thread-1" #150733 prio=5 os_prio=0 tid=0x00007f6d54020800 nid=0x11b61 in Object.wait() [0x00007f6c98564000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> ~at java.lang.Object.wait(Native Method)
> ~at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4672)
> ~- locked <0x00000005499908c0> (a org.apache.solr.update.SolrIndexWriter)
> ~at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWriter.java:2559)
> ~- locked <0x00000005499908c0> (a org.apache.solr.update.SolrIndexWriter)
> ~at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1036)
> ~at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1078)
> ~at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:286)
> ~at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:892)
> ~at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:105)
> ~at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:399)
> ~- locked <0x000000054e150cc0> (a org.apache.solr.update.DefaultSolrCoreState)
> ~at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:83)
> ~at org.apache.solr.core.SolrCore.close(SolrCore.java:1574)
> ~at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:141)
> ~at org.apache.solr.core.SolrCores$$Lambda$443/1058423472.call(Unknown Source)
> ~at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> ************************************************************************************************************
>
>
> The situation is as follows -
>
> 1. The first thing that happens is the request handlers being closed at -
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/core/SolrCore.java#L1588
>
> 2. Then it tries to close the index writer via -
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/core/SolrCore.java#L1610
>
> 3. When closing the index writer, it waits for any pending merges to finish at -
> https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java#L1236
>
> Now, if this waitForMerges() takes a long time (3-4 minutes), the instance won't shut down for the whole of that time, but because of *Step 1* it will stop
> accepting any requests.
>
> This becomes a problem when this node has a leader replica and it is stuck on waitForMerges() after closing its reqHandlers. We are in a situation where
> the leader is not accepting requests but has not given away the leadership, so we are in a leaderless phase.
>
>
> This issue triggers when we turnaround our nodes which causes a brief period of leaderless shards which leads to potential data losses.
>
> My question is -
> 1. How to avoid this situation given that we have big segment sizes and the merging the largest segments is going to take some time.
> We do not want to reduce the segment size as it will impact our search performance which is crucial.
> 2. Should Solr ideally not do the waitForMerges() step before closing the request handlers?
>
>
> Merge Policy config and segment size -
>
> <mergePolicyFactory class="org.apache.solr.index.SortingMergePolicyFactory">
> <str name="sort">time_of_arrival desc</str>
> <str name="wrapped.prefix">inner</str>
> <str name="inner.class">org.apache.solr.index.TieredMergePolicyFactory</str>
> <!-- Override default Solr7 values with the values
> we have been using in Solr 4, which allow
> more segments to be merged at once and larger
> segments to be created.
> -->
> <int name="inner.maxMergeAtOnce">16</int>
> <int name="inner.maxMergedSegmentMB">20480</int>
> </mergePolicyFactory>
>
>