You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Joe Obernberger <jo...@gmail.com> on 2017/07/12 13:14:59 UTC
Auto commit Error - Solr Cloud 6.6.0 with HDFS
Started up a 6.6.0 solr cloud instance running on 45 machines yesterday
using HDFS (managed schema in zookeeper) and began indexing. This error
occurred on several of the nodes:
auto commit error...:org.apache.solr.common.SolrException: Error opening
new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2069)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2189)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:667)
at org.apache.solr.update.CommitTracker.run(CommitTracker.java:217)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: openNewSearcher called
on closed core
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2056)
... 10 more
7/12/2017, 8:53:53 AM
Immediately following this error, the log shows another:
auto commit error...:org.apache.solr.common.SolrException:
openNewSearcher called on closed core
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1943)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:678)
at org.apache.solr.update.CommitTracker.run(CommitTracker.java:217)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Any ideas on what the problem could be? Thank you!
-Joe
Re: Auto commit Error - Solr Cloud 6.6.0 with HDFS
Posted by Joe Obernberger <jo...@gmail.com>.
Hi Shawn - had a shard go down (appears to have just dropped out of the
cluster) with a similar error:
2017-07-14 20:43:04.238 ERROR (commitScheduler-65-thread-1) [c:UNCLASS
s:shard87 r:core_node132 x:UNCLASS_shard87_replica1]
o.a.s.u.CommitTracker auto commit
error...:org.apache.solr.common.SolrException: openNewSearcher called on
closed core
at
org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1943)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:678)
at org.apache.solr.update.CommitTracker.run(CommitTracker.java:217)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
The whole log can be found here:
http://lovehorsepower.com/solr.log
the GC log is here:
http://lovehorsepower.com/solr_gc.log.3.current
-Joe
On 7/12/2017 9:25 AM, Shawn Heisey wrote:
> On 7/12/2017 7:14 AM, Joe Obernberger wrote:
>> Started up a 6.6.0 solr cloud instance running on 45 machines
>> yesterday using HDFS (managed schema in zookeeper) and began
>> indexing. This error occurred on several of the nodes:
> <snip>
>> Caused by: org.apache.solr.common.SolrException: openNewSearcher
>> called on closed core
> There's the important part of the error.
>
> For some reason, which is not immediately clear from the information
> provided, the core is closed. In that situation, Solr is not able to
> open a new searcher, so this error happens. Do you have any other WARN
> or ERROR messages in solr.log before this error? You might want to find
> a way to share an entire logfile and provide a URL for accessing it.
>
> Thanks,
> Shawn
>
>
> ---
> This email has been checked for viruses by AVG.
> http://www.avg.com
>
Re: Auto commit Error - Solr Cloud 6.6.0 with HDFS
Posted by Joe Obernberger <jo...@gmail.com>.
Thank you Shawn - we have some WARN messages like :
DFSClient
Slow waitForAckedSeqno took 31066ms (threshold=30000ms)
and
DFSClient
Slow ReadProcessor read fields took 30737ms (threshold=30000ms); ack:
seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS
downstreamAckTimeNanos: 30735948057,​ targets:
[DatanodeInfoWithStorage[172.16.100.223:50010,​DS-fc197ece-4c43-46a4-964e-9ecc8adcdcb1,​DISK],​
DatanodeInfoWithStorage[172.16.100.228:50010,​DS-fd9fe2ea-780b-490a-b9a7-ad54c994e307,​DISK],​
DatanodeInfoWithStorage[172.16.20.17:50010,​DS-63b38ca5-b674-4cdc-8102-f07a22709dc1,​DISK]]
I was getting the log ready for you, but it was overwritten in the
interim. If it happens again, I'll get the log file ready.
-Joe
On 7/12/2017 9:25 AM, Shawn Heisey wrote:
> On 7/12/2017 7:14 AM, Joe Obernberger wrote:
>> Started up a 6.6.0 solr cloud instance running on 45 machines
>> yesterday using HDFS (managed schema in zookeeper) and began
>> indexing. This error occurred on several of the nodes:
> <snip>
>> Caused by: org.apache.solr.common.SolrException: openNewSearcher
>> called on closed core
> There's the important part of the error.
>
> For some reason, which is not immediately clear from the information
> provided, the core is closed. In that situation, Solr is not able to
> open a new searcher, so this error happens. Do you have any other WARN
> or ERROR messages in solr.log before this error? You might want to find
> a way to share an entire logfile and provide a URL for accessing it.
>
> Thanks,
> Shawn
>
>
> ---
> This email has been checked for viruses by AVG.
> http://www.avg.com
>
Re: Auto commit Error - Solr Cloud 6.6.0 with HDFS
Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/12/2017 7:14 AM, Joe Obernberger wrote:
> Started up a 6.6.0 solr cloud instance running on 45 machines
> yesterday using HDFS (managed schema in zookeeper) and began
> indexing. This error occurred on several of the nodes:
<snip>
> Caused by: org.apache.solr.common.SolrException: openNewSearcher
> called on closed core
There's the important part of the error.
For some reason, which is not immediately clear from the information
provided, the core is closed. In that situation, Solr is not able to
open a new searcher, so this error happens. Do you have any other WARN
or ERROR messages in solr.log before this error? You might want to find
a way to share an entire logfile and provide a URL for accessing it.
Thanks,
Shawn