You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Eric Bus <er...@websight.nl> on 2013/11/07 10:09:55 UTC

SolrCloud keeps repeating exception 'SolrCoreState already closed'

Hi,

I'm having a problem with one of my shards. Since yesterday, SOLR keeps repeating the same exception over and over for this shard.
The webinterface for this SOLR instance is also not working (it hangs on the Loading indicator).

Nov 7, 2013 9:08:12 AM org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: [website1_shard1_replica3] webapp=/solr path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {} 0 0
Nov 7, 2013 9:08:12 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.RuntimeException: SolrCoreState already closed
        at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:79)
        at org.apache.solr.update.DirectUpdateHandler2.delete(DirectUpdateHandler2.java:276)
        at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:77)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:460)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDelete(DistributedUpdateProcessor.java:1036)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:721)
        at org.apache.solr.update.processor.LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:121)
        at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:346)
        at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:277)
        at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:662)

I have about 3GB of logfiles for this single message. Reloading the collection does not work. Reloading the specific shard core returns the same exception. The only option seems to be to restart the server. But because it's the leader for a lot of collections, I want to know why this is happening. I've seen this problem before, and I haven't figured out what is causing it.

I've reported a different problem a few days ago with 'hanging' deleted logfiles. Could this be related? Could the hanging logfiles prevent a new Searcher from opening? I've updated two of my three hosts to 4.5.1 but after only 2 days uptime, I'm still seeing about 11.000 deleted logfiles in the lsof output.

Best regards,
Eric Bus

Re: SolrCloud keeps repeating exception 'SolrCoreState already closed'

Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
No, I am running on the example jetty. I am re-running the import and
haven't hit the problem yet. Still running.

On Tue, Dec 3, 2013 at 5:45 PM, Eric Bus <er...@websight.nl> wrote:
> Are you currently running SOLR under Tomcat or standalone with Jetty? I switched from Tomcat to Jetty and the problems went away.
>
> - Eric
>
>
> -----Oorspronkelijk bericht-----
> Van: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com]
> Verzonden: dinsdag 3 december 2013 12:44
> Aan: solr-user@lucene.apache.org
> Onderwerp: Re: SolrCloud keeps repeating exception 'SolrCoreState already closed'
>
> I just ran into this issue on solr 4.6 on an EC2 machine while indexing wikipedia dump with DIH. I'm trying to isolate exceptions before the SolrCoreState already closed exception.
>
> On Sun, Nov 10, 2013 at 11:58 PM, Mark Miller <ma...@gmail.com> wrote:
>> Can you isolate any exceptions that happened just before that exception. started repeating?
>>
>> - Mark
>>
>>> On Nov 7, 2013, at 9:09 AM, Eric Bus <er...@websight.nl> wrote:
>>>
>>> Hi,
>>>
>>> I'm having a problem with one of my shards. Since yesterday, SOLR keeps repeating the same exception over and over for this shard.
>>> The webinterface for this SOLR instance is also not working (it hangs on the Loading indicator).
>>>
>>> Nov 7, 2013 9:08:12 AM
>>> org.apache.solr.update.processor.LogUpdateProcessor finish
>>> INFO: [website1_shard1_replica3] webapp=/solr path=/update
>>> params={update.distrib=TOLEADER&wt=javabin&version=2} {} 0 0 Nov 7,
>>> 2013 9:08:12 AM org.apache.solr.common.SolrException log
>>> SEVERE: java.lang.RuntimeException: SolrCoreState already closed
>>>        at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:79)
>>>        at org.apache.solr.update.DirectUpdateHandler2.delete(DirectUpdateHandler2.java:276)
>>>        at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:77)
>>>        at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
>>>        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:460)
>>>        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDelete(DistributedUpdateProcessor.java:1036)
>>>        at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:721)
>>>        at org.apache.solr.update.processor.LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:121)
>>>        at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:346)
>>>        at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:277)
>>>        at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
>>>        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
>>>        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>>>        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>>>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
>>>        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448)
>>>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269)
>>>        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>>        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>>        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>>        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>>        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>>        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>>>        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
>>>        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
>>>        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>>>        at java.lang.Thread.run(Thread.java:662)
>>>
>>> I have about 3GB of logfiles for this single message. Reloading the collection does not work. Reloading the specific shard core returns the same exception. The only option seems to be to restart the server. But because it's the leader for a lot of collections, I want to know why this is happening. I've seen this problem before, and I haven't figured out what is causing it.
>>>
>>> I've reported a different problem a few days ago with 'hanging' deleted logfiles. Could this be related? Could the hanging logfiles prevent a new Searcher from opening? I've updated two of my three hosts to 4.5.1 but after only 2 days uptime, I'm still seeing about 11.000 deleted logfiles in the lsof output.
>>>
>>> Best regards,
>>> Eric Bus
>>>
>>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.



-- 
Regards,
Shalin Shekhar Mangar.

RE: SolrCloud keeps repeating exception 'SolrCoreState already closed'

Posted by Eric Bus <er...@websight.nl>.
Are you currently running SOLR under Tomcat or standalone with Jetty? I switched from Tomcat to Jetty and the problems went away.

- Eric


-----Oorspronkelijk bericht-----
Van: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com] 
Verzonden: dinsdag 3 december 2013 12:44
Aan: solr-user@lucene.apache.org
Onderwerp: Re: SolrCloud keeps repeating exception 'SolrCoreState already closed'

I just ran into this issue on solr 4.6 on an EC2 machine while indexing wikipedia dump with DIH. I'm trying to isolate exceptions before the SolrCoreState already closed exception.

On Sun, Nov 10, 2013 at 11:58 PM, Mark Miller <ma...@gmail.com> wrote:
> Can you isolate any exceptions that happened just before that exception. started repeating?
>
> - Mark
>
>> On Nov 7, 2013, at 9:09 AM, Eric Bus <er...@websight.nl> wrote:
>>
>> Hi,
>>
>> I'm having a problem with one of my shards. Since yesterday, SOLR keeps repeating the same exception over and over for this shard.
>> The webinterface for this SOLR instance is also not working (it hangs on the Loading indicator).
>>
>> Nov 7, 2013 9:08:12 AM 
>> org.apache.solr.update.processor.LogUpdateProcessor finish
>> INFO: [website1_shard1_replica3] webapp=/solr path=/update 
>> params={update.distrib=TOLEADER&wt=javabin&version=2} {} 0 0 Nov 7, 
>> 2013 9:08:12 AM org.apache.solr.common.SolrException log
>> SEVERE: java.lang.RuntimeException: SolrCoreState already closed
>>        at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:79)
>>        at org.apache.solr.update.DirectUpdateHandler2.delete(DirectUpdateHandler2.java:276)
>>        at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:77)
>>        at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
>>        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:460)
>>        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDelete(DistributedUpdateProcessor.java:1036)
>>        at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:721)
>>        at org.apache.solr.update.processor.LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:121)
>>        at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:346)
>>        at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:277)
>>        at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
>>        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
>>        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>>        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
>>        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448)
>>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269)
>>        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>>        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
>>        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
>>        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>>        at java.lang.Thread.run(Thread.java:662)
>>
>> I have about 3GB of logfiles for this single message. Reloading the collection does not work. Reloading the specific shard core returns the same exception. The only option seems to be to restart the server. But because it's the leader for a lot of collections, I want to know why this is happening. I've seen this problem before, and I haven't figured out what is causing it.
>>
>> I've reported a different problem a few days ago with 'hanging' deleted logfiles. Could this be related? Could the hanging logfiles prevent a new Searcher from opening? I've updated two of my three hosts to 4.5.1 but after only 2 days uptime, I'm still seeing about 11.000 deleted logfiles in the lsof output.
>>
>> Best regards,
>> Eric Bus
>>
>>



--
Regards,
Shalin Shekhar Mangar.

Re: SolrCloud keeps repeating exception 'SolrCoreState already closed'

Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
I just ran into this issue on solr 4.6 on an EC2 machine while
indexing wikipedia dump with DIH. I'm trying to isolate exceptions
before the SolrCoreState already closed exception.

On Sun, Nov 10, 2013 at 11:58 PM, Mark Miller <ma...@gmail.com> wrote:
> Can you isolate any exceptions that happened just before that exception. started repeating?
>
> - Mark
>
>> On Nov 7, 2013, at 9:09 AM, Eric Bus <er...@websight.nl> wrote:
>>
>> Hi,
>>
>> I'm having a problem with one of my shards. Since yesterday, SOLR keeps repeating the same exception over and over for this shard.
>> The webinterface for this SOLR instance is also not working (it hangs on the Loading indicator).
>>
>> Nov 7, 2013 9:08:12 AM org.apache.solr.update.processor.LogUpdateProcessor finish
>> INFO: [website1_shard1_replica3] webapp=/solr path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {} 0 0
>> Nov 7, 2013 9:08:12 AM org.apache.solr.common.SolrException log
>> SEVERE: java.lang.RuntimeException: SolrCoreState already closed
>>        at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:79)
>>        at org.apache.solr.update.DirectUpdateHandler2.delete(DirectUpdateHandler2.java:276)
>>        at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:77)
>>        at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
>>        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:460)
>>        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDelete(DistributedUpdateProcessor.java:1036)
>>        at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:721)
>>        at org.apache.solr.update.processor.LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:121)
>>        at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:346)
>>        at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:277)
>>        at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
>>        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
>>        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>>        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
>>        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448)
>>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269)
>>        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>>        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
>>        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
>>        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>>        at java.lang.Thread.run(Thread.java:662)
>>
>> I have about 3GB of logfiles for this single message. Reloading the collection does not work. Reloading the specific shard core returns the same exception. The only option seems to be to restart the server. But because it's the leader for a lot of collections, I want to know why this is happening. I've seen this problem before, and I haven't figured out what is causing it.
>>
>> I've reported a different problem a few days ago with 'hanging' deleted logfiles. Could this be related? Could the hanging logfiles prevent a new Searcher from opening? I've updated two of my three hosts to 4.5.1 but after only 2 days uptime, I'm still seeing about 11.000 deleted logfiles in the lsof output.
>>
>> Best regards,
>> Eric Bus
>>
>>



-- 
Regards,
Shalin Shekhar Mangar.

Re: SolrCloud keeps repeating exception 'SolrCoreState already closed'

Posted by Mark Miller <ma...@gmail.com>.
Can you isolate any exceptions that happened just before that exception. started repeating?

- Mark

> On Nov 7, 2013, at 9:09 AM, Eric Bus <er...@websight.nl> wrote:
> 
> Hi,
> 
> I'm having a problem with one of my shards. Since yesterday, SOLR keeps repeating the same exception over and over for this shard.
> The webinterface for this SOLR instance is also not working (it hangs on the Loading indicator).
> 
> Nov 7, 2013 9:08:12 AM org.apache.solr.update.processor.LogUpdateProcessor finish
> INFO: [website1_shard1_replica3] webapp=/solr path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {} 0 0
> Nov 7, 2013 9:08:12 AM org.apache.solr.common.SolrException log
> SEVERE: java.lang.RuntimeException: SolrCoreState already closed
>        at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:79)
>        at org.apache.solr.update.DirectUpdateHandler2.delete(DirectUpdateHandler2.java:276)
>        at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:77)
>        at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
>        at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:460)
>        at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDelete(DistributedUpdateProcessor.java:1036)
>        at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:721)
>        at org.apache.solr.update.processor.LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:121)
>        at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:346)
>        at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:277)
>        at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
>        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
>        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
>        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448)
>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269)
>        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
>        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
>        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>        at java.lang.Thread.run(Thread.java:662)
> 
> I have about 3GB of logfiles for this single message. Reloading the collection does not work. Reloading the specific shard core returns the same exception. The only option seems to be to restart the server. But because it's the leader for a lot of collections, I want to know why this is happening. I've seen this problem before, and I haven't figured out what is causing it.
> 
> I've reported a different problem a few days ago with 'hanging' deleted logfiles. Could this be related? Could the hanging logfiles prevent a new Searcher from opening? I've updated two of my three hosts to 4.5.1 but after only 2 days uptime, I'm still seeing about 11.000 deleted logfiles in the lsof output.
> 
> Best regards,
> Eric Bus
> 
>