You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov.INVALID> on 2022/03/25 14:57:48 UTC
IllegalArgumentException: Unknown directory
I have a core which ceased responding either to select or to admin/core: restarting the Solr instance resolved the problem, but I am wondering whether there is some configuration which may need to be tweaked. Below is a portion of solr.log from the time when the problem began. Please note that the directory /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata does indeed exist
2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.S.Request [run_sel_index_shard1_replica_n3] webapp=/solr path=/select params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on} hits=94 status=0 QTime=2610
2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.SolrCore [run_sel_cache_shard1_replica_n3] CLOSING SolrCore org.apache.solr.core.SolrCore@59c22833<ma...@59c22833>
2022-03-25 08:47:58.527 INFO (qtp1847637306-38437) [c:run_sel_index s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
2022-03-25 08:47:58.531 INFO (qtp1847637306-38437) [c:run_sel_index s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2b7f5d5e: rootName = null, domain = solr.core.run_sel_cache.shard1.replica_n3, service url = null, agent id = null] for registry solr.core.run_sel_cache.shard1.replica_n3 / com.codahale.metrics.MetricRegistry@5211d667<ma...@5211d667>
2022-03-25 08:47:58.568 INFO (qtp1847637306-38090) [c:run_sel_index s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.S.Request [run_sel_index_shard1_replica_n3] webapp=/solr path=/select params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on} hits=94 status=0 QTime=593
2022-03-25 08:47:58.593 INFO (qtp1847637306-38437) [c:run_sel_index s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.run_sel_cache.shard1.leader, tag=SolrCore@59c22833
2022-03-25 08:47:58.596 INFO (qtp1847637306-38437) [c:run_sel_index s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary).
2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ] o.a.s.c.SolrCore [run_sel_cache_shard1_replica_n3] CLOSING SolrCore org.apache.solr.core.SolrCore@59c22833<ma...@59c22833>
2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.run_sel_cache.shard1.leader, tag=SolrCore@59c22833
2022-03-25 08:47:59.258 WARN (qtp1847637306-38404) [ ] o.a.s.c.ZkController /configs/run_sel_cache has no more registered listeners, but a live one attempted to unregister!
2022-03-25 08:47:59.259 ERROR (qtp1847637306-38404) [ ] o.a.s.c.SolrCore java.lang.IllegalArgumentException: Unknown directory: NRTCachingDirectory(MMapDirectory@/data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92<ma...@2bebbe92>; maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
at org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
at org.apache.solr.core.SolrCore.close(SolrCore.java:1674)
at org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:353)
at org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:46)
at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
2022-03-25 08:47:59.260 INFO (qtp1847637306-38404) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={action=STATUS&indexInfo=false&wt=json} status=0 QTime=3
2022-03-25 08:47:59.262 INFO (qtp1847637306-37771) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
2022-03-25 08:47:59.275 INFO (qtp1847637306-37771) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=13
2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ] o.a.s.c.SolrCore [run_sel_cache_shard1_replica_n3] CLOSING SolrCore org.apache.solr.core.SolrCore@59c22833<ma...@59c22833>
2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.run_sel_cache.shard1.leader, tag=SolrCore@59c22833
2022-03-25 08:48:09.579 WARN (qtp1847637306-38348) [ ] o.a.s.c.ZkController /configs/run_sel_cache has no more registered listeners, but a live one attempted to unregister!
Re: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
Posted by matthew sporleder <ms...@gmail.com>.
My only other guess (and I mean wild guess) is that because the error is
for a *lock* that it is actually a timeout or similar locking error with a
bad error message.
On Fri, Mar 25, 2022 at 11:11 AM Oakley, Craig (NIH/NLM/NCBI) [C]
<cr...@nih.gov.invalid> wrote:
> Thanks for the quick reply
>
> grep -i -c OutOfMemory solr.log.20220* shows zero
>
> Nothing new in /var/log/dmesg since reboot of the host a couple weeks ago
>
> Let me know if you have any other suggestions
>
> Thanks again
>
> -----Original Message-----
> From: matthew sporleder <ms...@gmail.com>
> Sent: Friday, March 25, 2022 11:00 AM
> To: users@solr.apache.org
> Subject: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and are
> confident the content is safe.
>
>
> Is there an OOM anywhere in that log? I've definitely seen java lose track
> of things during a slow-moving oom.
>
> Also definitely check dmesg for anything in that same timeframe if you
> still have the logs.
>
> On Fri, Mar 25, 2022 at 10:58 AM Oakley, Craig (NIH/NLM/NCBI) [C]
> <cr...@nih.gov.invalid> wrote:
>
> > I have a core which ceased responding either to select or to admin/core:
> > restarting the Solr instance resolved the problem, but I am wondering
> > whether there is some configuration which may need to be tweaked. Below
> is
> > a portion of solr.log from the time when the problem began. Please note
> > that the directory
> >
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > does indeed exist
> >
> > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.c.S.Request
> > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > hits=94 status=0 QTime=2610
> > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:47:58.527 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:47:58.531 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.r.SolrJmxReporter Closing reporter
> > [org.apache.solr.metrics.reporters.SolrJmxReporter@2b7f5d5e: rootName =
> > null, domain = solr.core.run_sel_cache.shard1.replica_n3, service url =
> > null, agent id = null] for registry
> > solr.core.run_sel_cache.shard1.replica_n3 /
> > com.codahale.metrics.MetricRegistry@5211d667<mailto:
> > com.codahale.metrics.MetricRegistry@5211d667>
> > 2022-03-25 08:47:58.568 INFO (qtp1847637306-38090) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.c.S.Request
> > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > hits=94 status=0 QTime=593
> > 2022-03-25 08:47:58.593 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:47:58.596 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ...
> SKIPPED
> > (unnecessary).
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:47:59.258 WARN (qtp1847637306-38404) [ ]
> > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > listeners, but a live one attempted to unregister!
> > 2022-03-25 08:47:59.259 ERROR (qtp1847637306-38404) [ ]
> o.a.s.c.SolrCore
> > java.lang.IllegalArgumentException: Unknown directory:
> > NRTCachingDirectory(MMapDirectory@
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92<mailto:
> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92>;
> > maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
> > at
> >
> org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
> > at org.apache.solr.core.SolrCore.close(SolrCore.java:1674)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:353)
> > at
> org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:46)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> > at
> > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
> > at
> >
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
> > at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> > at
> >
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at
> >
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at org.eclipse.jetty.server.Server.handle(Server.java:500)
> > at
> >
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> > at
> > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> > at
> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> > at
> >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
> > at org.eclipse.jetty.io
> > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> > at org.eclipse.jetty.io
> > .FillInterest.fillable(FillInterest.java:103)
> > at org.eclipse.jetty.io
> > .ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> > at
> >
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2022-03-25 08:47:59.260 INFO (qtp1847637306-38404) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> > params={action=STATUS&indexInfo=false&wt=json} status=0 QTime=3
> > 2022-03-25 08:47:59.262 INFO (qtp1847637306-37771) [ ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus
> with
> > params action=clusterstatus&wt=json and sendToOCPQueue=true
> > 2022-03-25 08:47:59.275 INFO (qtp1847637306-37771) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> > params={action=clusterstatus&wt=json} status=0 QTime=13
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:48:09.579 WARN (qtp1847637306-38348) [ ]
> > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > listeners, but a live one attempted to unregister!
> >
>
RE: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov.INVALID>.
Thanks for the quick reply
grep -i -c OutOfMemory solr.log.20220* shows zero
Nothing new in /var/log/dmesg since reboot of the host a couple weeks ago
Let me know if you have any other suggestions
Thanks again
-----Original Message-----
From: matthew sporleder <ms...@gmail.com>
Sent: Friday, March 25, 2022 11:00 AM
To: users@solr.apache.org
Subject: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.
Is there an OOM anywhere in that log? I've definitely seen java lose track
of things during a slow-moving oom.
Also definitely check dmesg for anything in that same timeframe if you
still have the logs.
On Fri, Mar 25, 2022 at 10:58 AM Oakley, Craig (NIH/NLM/NCBI) [C]
<cr...@nih.gov.invalid> wrote:
> I have a core which ceased responding either to select or to admin/core:
> restarting the Solr instance resolved the problem, but I am wondering
> whether there is some configuration which may need to be tweaked. Below is
> a portion of solr.log from the time when the problem began. Please note
> that the directory
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> does indeed exist
>
> 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.S.Request
> [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> hits=94 status=0 QTime=2610
> 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.SolrCore
> [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> org.apache.solr.core.SolrCore@59c22833<mailto:
> org.apache.solr.core.SolrCore@59c22833>
> 2022-03-25 08:47:58.527 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> 2022-03-25 08:47:58.531 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.m.r.SolrJmxReporter Closing reporter
> [org.apache.solr.metrics.reporters.SolrJmxReporter@2b7f5d5e: rootName =
> null, domain = solr.core.run_sel_cache.shard1.replica_n3, service url =
> null, agent id = null] for registry
> solr.core.run_sel_cache.shard1.replica_n3 /
> com.codahale.metrics.MetricRegistry@5211d667<mailto:
> com.codahale.metrics.MetricRegistry@5211d667>
> 2022-03-25 08:47:58.568 INFO (qtp1847637306-38090) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.S.Request
> [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> hits=94 status=0 QTime=593
> 2022-03-25 08:47:58.593 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.collection.run_sel_cache.shard1.leader, tag=SolrCore@59c22833
> 2022-03-25 08:47:58.596 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED
> (unnecessary).
> 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ] o.a.s.c.SolrCore
> [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> org.apache.solr.core.SolrCore@59c22833<mailto:
> org.apache.solr.core.SolrCore@59c22833>
> 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.collection.run_sel_cache.shard1.leader, tag=SolrCore@59c22833
> 2022-03-25 08:47:59.258 WARN (qtp1847637306-38404) [ ]
> o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> listeners, but a live one attempted to unregister!
> 2022-03-25 08:47:59.259 ERROR (qtp1847637306-38404) [ ] o.a.s.c.SolrCore
> java.lang.IllegalArgumentException: Unknown directory:
> NRTCachingDirectory(MMapDirectory@/data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92<mailto:
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92>;
> maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
> at
> org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
> at org.apache.solr.core.SolrCore.close(SolrCore.java:1674)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:353)
> at org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:46)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
> at
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:500)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> at
> org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
> at org.eclipse.jetty.io
> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io
> .FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io
> .ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> at java.lang.Thread.run(Thread.java:748)
>
> 2022-03-25 08:47:59.260 INFO (qtp1847637306-38404) [ ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> params={action=STATUS&indexInfo=false&wt=json} status=0 QTime=3
> 2022-03-25 08:47:59.262 INFO (qtp1847637306-37771) [ ]
> o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with
> params action=clusterstatus&wt=json and sendToOCPQueue=true
> 2022-03-25 08:47:59.275 INFO (qtp1847637306-37771) [ ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> params={action=clusterstatus&wt=json} status=0 QTime=13
> 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ] o.a.s.c.SolrCore
> [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> org.apache.solr.core.SolrCore@59c22833<mailto:
> org.apache.solr.core.SolrCore@59c22833>
> 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.collection.run_sel_cache.shard1.leader, tag=SolrCore@59c22833
> 2022-03-25 08:48:09.579 WARN (qtp1847637306-38348) [ ]
> o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> listeners, but a live one attempted to unregister!
>
Re: IllegalArgumentException: Unknown directory
Posted by matthew sporleder <ms...@gmail.com>.
Is there an OOM anywhere in that log? I've definitely seen java lose track
of things during a slow-moving oom.
Also definitely check dmesg for anything in that same timeframe if you
still have the logs.
On Fri, Mar 25, 2022 at 10:58 AM Oakley, Craig (NIH/NLM/NCBI) [C]
<cr...@nih.gov.invalid> wrote:
> I have a core which ceased responding either to select or to admin/core:
> restarting the Solr instance resolved the problem, but I am wondering
> whether there is some configuration which may need to be tweaked. Below is
> a portion of solr.log from the time when the problem began. Please note
> that the directory
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> does indeed exist
>
> 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.S.Request
> [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> hits=94 status=0 QTime=2610
> 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.SolrCore
> [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> org.apache.solr.core.SolrCore@59c22833<mailto:
> org.apache.solr.core.SolrCore@59c22833>
> 2022-03-25 08:47:58.527 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> 2022-03-25 08:47:58.531 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.m.r.SolrJmxReporter Closing reporter
> [org.apache.solr.metrics.reporters.SolrJmxReporter@2b7f5d5e: rootName =
> null, domain = solr.core.run_sel_cache.shard1.replica_n3, service url =
> null, agent id = null] for registry
> solr.core.run_sel_cache.shard1.replica_n3 /
> com.codahale.metrics.MetricRegistry@5211d667<mailto:
> com.codahale.metrics.MetricRegistry@5211d667>
> 2022-03-25 08:47:58.568 INFO (qtp1847637306-38090) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.S.Request
> [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> hits=94 status=0 QTime=593
> 2022-03-25 08:47:58.593 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.collection.run_sel_cache.shard1.leader, tag=SolrCore@59c22833
> 2022-03-25 08:47:58.596 INFO (qtp1847637306-38437) [c:run_sel_index
> s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED
> (unnecessary).
> 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ] o.a.s.c.SolrCore
> [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> org.apache.solr.core.SolrCore@59c22833<mailto:
> org.apache.solr.core.SolrCore@59c22833>
> 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.collection.run_sel_cache.shard1.leader, tag=SolrCore@59c22833
> 2022-03-25 08:47:59.258 WARN (qtp1847637306-38404) [ ]
> o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> listeners, but a live one attempted to unregister!
> 2022-03-25 08:47:59.259 ERROR (qtp1847637306-38404) [ ] o.a.s.c.SolrCore
> java.lang.IllegalArgumentException: Unknown directory:
> NRTCachingDirectory(MMapDirectory@/data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92<mailto:
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92>;
> maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
> at
> org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
> at org.apache.solr.core.SolrCore.close(SolrCore.java:1674)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:353)
> at org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:46)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
> at
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:500)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> at
> org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
> at org.eclipse.jetty.io
> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io
> .FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io
> .ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> at java.lang.Thread.run(Thread.java:748)
>
> 2022-03-25 08:47:59.260 INFO (qtp1847637306-38404) [ ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> params={action=STATUS&indexInfo=false&wt=json} status=0 QTime=3
> 2022-03-25 08:47:59.262 INFO (qtp1847637306-37771) [ ]
> o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with
> params action=clusterstatus&wt=json and sendToOCPQueue=true
> 2022-03-25 08:47:59.275 INFO (qtp1847637306-37771) [ ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> params={action=clusterstatus&wt=json} status=0 QTime=13
> 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ] o.a.s.c.SolrCore
> [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> org.apache.solr.core.SolrCore@59c22833<mailto:
> org.apache.solr.core.SolrCore@59c22833>
> 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.collection.run_sel_cache.shard1.leader, tag=SolrCore@59c22833
> 2022-03-25 08:48:09.579 WARN (qtp1847637306-38348) [ ]
> o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> listeners, but a live one attempted to unregister!
>