You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov.INVALID> on 2022/08/06 02:47:05 UTC
RE: IllegalArgumentException: Unknown directory
This error has happened again. Does anyone yet have any explanation or suggestion?
-----Original Message-----
From: Oakley, Craig (NIH/NLM/NCBI) [C]
Sent: Monday, May 02, 2022 2:29 PM
To: users@solr.apache.org
Subject: Re: IllegalArgumentException: Unknown directory
This has happened several more times, and I have notice something else which might be a clue:
The problem happened over the weekend; and the first erroneous complaint about /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata (as though it did not exist) occurred 17 minutes after /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/replication.properties was updated. Last month, the problem occurred just 40 seconds after replication.properties was updated.
What known connections might there be between replication.properties and snapshot_metadata?
-----Original Message-----
From: matthew sporleder <ms...@gmail.com>
Sent: Monday, March 28, 2022 9:41 AM
To: users@solr.apache.org
Subject: Re: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.
My only other guess (and I mean wild guess) is that because the error is
for a *lock* that it is actually a timeout or similar locking error with a
bad error message.
On Fri, Mar 25, 2022 at 11:11 AM Oakley, Craig (NIH/NLM/NCBI) [C]
<cr...@nih.gov.invalid> wrote:
> Thanks for the quick reply
>
> grep -i -c OutOfMemory solr.log.20220* shows zero
>
> Nothing new in /var/log/dmesg since reboot of the host a couple weeks ago
>
> Let me know if you have any other suggestions
>
> Thanks again
>
> -----Original Message-----
> From: matthew sporleder <ms...@gmail.com>
> Sent: Friday, March 25, 2022 11:00 AM
> To: users@solr.apache.org
> Subject: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and are
> confident the content is safe.
>
>
> Is there an OOM anywhere in that log? I've definitely seen java lose track
> of things during a slow-moving oom.
>
> Also definitely check dmesg for anything in that same timeframe if you
> still have the logs.
>
> On Fri, Mar 25, 2022 at 10:58 AM Oakley, Craig (NIH/NLM/NCBI) [C]
> <cr...@nih.gov.invalid> wrote:
>
> > I have a core which ceased responding either to select or to admin/core:
> > restarting the Solr instance resolved the problem, but I am wondering
> > whether there is some configuration which may need to be tweaked. Below
> is
> > a portion of solr.log from the time when the problem began. Please note
> > that the directory
> >
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > does indeed exist
> >
> > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.c.S.Request
> > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > hits=94 status=0 QTime=2610
> > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:47:58.527 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:47:58.531 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.r.SolrJmxReporter Closing reporter
> > [org.apache.solr.metrics.reporters.SolrJmxReporter@2b7f5d5e: rootName =
> > null, domain = solr.core.run_sel_cache.shard1.replica_n3, service url =
> > null, agent id = null] for registry
> > solr.core.run_sel_cache.shard1.replica_n3 /
> > com.codahale.metrics.MetricRegistry@5211d667<mailto:
> > com.codahale.metrics.MetricRegistry@5211d667>
> > 2022-03-25 08:47:58.568 INFO (qtp1847637306-38090) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.c.S.Request
> > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > hits=94 status=0 QTime=593
> > 2022-03-25 08:47:58.593 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:47:58.596 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ...
> SKIPPED
> > (unnecessary).
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:47:59.258 WARN (qtp1847637306-38404) [ ]
> > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > listeners, but a live one attempted to unregister!
> > 2022-03-25 08:47:59.259 ERROR (qtp1847637306-38404) [ ]
> o.a.s.c.SolrCore
> > java.lang.IllegalArgumentException: Unknown directory:
> > NRTCachingDirectory(MMapDirectory@
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92<mailto:
> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92>;
> > maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
> > at
> >
> org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
> > at org.apache.solr.core.SolrCore.close(SolrCore.java:1674)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:353)
> > at
> org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:46)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> > at
> > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
> > at
> >
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
> > at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> > at
> >
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at
> >
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at org.eclipse.jetty.server.Server.handle(Server.java:500)
> > at
> >
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> > at
> > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> > at
> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> > at
> >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
> > at org.eclipse.jetty.io
> > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> > at org.eclipse.jetty.io
> > .FillInterest.fillable(FillInterest.java:103)
> > at org.eclipse.jetty.io
> > .ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> > at
> >
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2022-03-25 08:47:59.260 INFO (qtp1847637306-38404) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> > params={action=STATUS&indexInfo=false&wt=json} status=0 QTime=3
> > 2022-03-25 08:47:59.262 INFO (qtp1847637306-37771) [ ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus
> with
> > params action=clusterstatus&wt=json and sendToOCPQueue=true
> > 2022-03-25 08:47:59.275 INFO (qtp1847637306-37771) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> > params={action=clusterstatus&wt=json} status=0 QTime=13
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:48:09.579 WARN (qtp1847637306-38348) [ ]
> > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > listeners, but a live one attempted to unregister!
> >
>
Re: IllegalArgumentException: Unknown directory
Posted by Kevin Risden <kr...@apache.org>.
Assuming this relates to the original thread
https://lists.apache.org/thread/58wjltm7h304g76xfnm640n37qo5bjhq
java.lang.IllegalArgumentException: Unknown directory:
> > > NRTCachingDirectory(MMapDirectory@
> >
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92
> <mailto:
> > > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92>;
> > > maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
>
>
> org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
> > > at org.apache.solr.core.SolrCore.close(SolrCore.java:1674)
> > > at
> > >
> >
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:353)
> > > at
> > org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:46)
This looks like close might be called multiple times? At least this looks
like getCoreStatus is looking at CachingDirectoryFactory.release which then
fails.... I don't have an idea of why this might be, but could be an area
to look into.
Kevin Risden
On Mon, Feb 6, 2023 at 11:30 AM Oakley, Craig (NIH/NLM/NCBI) [C]
<cr...@nih.gov.invalid> wrote:
> This error has happened yet again. Does anyone yet have any explanation or
> suggestion?
>
> -----Original Message-----
> From: Oakley, Craig (NIH/NLM/NCBI) [C]
> Sent: Thursday, October 13, 2022 6:09 PM
> To: users@solr.apache.org
> Subject: RE: IllegalArgumentException: Unknown directory
>
> This error has happened again. Does anyone yet have any explanation or
> suggestion?
>
> -----Original Message-----
> From: Oakley, Craig (NIH/NLM/NCBI) [C] <cr...@nih.gov.INVALID>
> Sent: Friday, August 05, 2022 10:47 PM
> To: users@solr.apache.org
> Subject: RE: IllegalArgumentException: Unknown directory
>
> This error has happened again. Does anyone yet have any explanation or
> suggestion?
>
> -----Original Message-----
> From: Oakley, Craig (NIH/NLM/NCBI) [C]
> Sent: Monday, May 02, 2022 2:29 PM
> To: users@solr.apache.org
> Subject: Re: IllegalArgumentException: Unknown directory
>
> This has happened several more times, and I have notice something else
> which might be a clue:
>
> The problem happened over the weekend; and the first erroneous complaint
> about
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> (as though it did not exist) occurred 17 minutes after
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/replication.properties
> was updated. Last month, the problem occurred just 40 seconds after
> replication.properties was updated.
>
> What known connections might there be between replication.properties and
> snapshot_metadata?
>
> -----Original Message-----
> From: matthew sporleder <ms...@gmail.com>
> Sent: Monday, March 28, 2022 9:41 AM
> To: users@solr.apache.org
> Subject: Re: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and are
> confident the content is safe.
>
>
> My only other guess (and I mean wild guess) is that because the error is
> for a *lock* that it is actually a timeout or similar locking error with a
> bad error message.
>
> On Fri, Mar 25, 2022 at 11:11 AM Oakley, Craig (NIH/NLM/NCBI) [C]
> <cr...@nih.gov.invalid> wrote:
>
> > Thanks for the quick reply
> >
> > grep -i -c OutOfMemory solr.log.20220* shows zero
> >
> > Nothing new in /var/log/dmesg since reboot of the host a couple weeks ago
> >
> > Let me know if you have any other suggestions
> >
> > Thanks again
> >
> > -----Original Message-----
> > From: matthew sporleder <ms...@gmail.com>
> > Sent: Friday, March 25, 2022 11:00 AM
> > To: users@solr.apache.org
> > Subject: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you recognize the sender and are
> > confident the content is safe.
> >
> >
> > Is there an OOM anywhere in that log? I've definitely seen java lose
> track
> > of things during a slow-moving oom.
> >
> > Also definitely check dmesg for anything in that same timeframe if you
> > still have the logs.
> >
> > On Fri, Mar 25, 2022 at 10:58 AM Oakley, Craig (NIH/NLM/NCBI) [C]
> > <cr...@nih.gov.invalid> wrote:
> >
> > > I have a core which ceased responding either to select or to
> admin/core:
> > > restarting the Solr instance resolved the problem, but I am wondering
> > > whether there is some configuration which may need to be tweaked. Below
> > is
> > > a portion of solr.log from the time when the problem began. Please note
> > > that the directory
> > >
> >
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > > does indeed exist
> > >
> > > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.c.S.Request
> > > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> > >
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > > hits=94 status=0 QTime=2610
> > > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.c.SolrCore
> > > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > > org.apache.solr.core.SolrCore@59c22833<mailto:
> > > org.apache.solr.core.SolrCore@59c22833>
> > > 2022-03-25 08:47:58.527 INFO (qtp1847637306-38437) [c:run_sel_index
> > > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > > o.a.s.m.SolrMetricManager Closing metric reporters for
> > > registry=solr.core.run_sel_cache.shard1.replica_n3,
> tag=SolrCore@59c22833
> > > 2022-03-25 08:47:58.531 INFO (qtp1847637306-38437) [c:run_sel_index
> > > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > > o.a.s.m.r.SolrJmxReporter Closing reporter
> > > [org.apache.solr.metrics.reporters.SolrJmxReporter@2b7f5d5e: rootName
> =
> > > null, domain = solr.core.run_sel_cache.shard1.replica_n3, service url =
> > > null, agent id = null] for registry
> > > solr.core.run_sel_cache.shard1.replica_n3 /
> > > com.codahale.metrics.MetricRegistry@5211d667<mailto:
> > > com.codahale.metrics.MetricRegistry@5211d667>
> > > 2022-03-25 08:47:58.568 INFO (qtp1847637306-38090) [c:run_sel_index
> > > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.c.S.Request
> > > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> > >
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > > hits=94 status=0 QTime=593
> > > 2022-03-25 08:47:58.593 INFO (qtp1847637306-38437) [c:run_sel_index
> > > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > > o.a.s.m.SolrMetricManager Closing metric reporters for
> > > registry=solr.collection.run_sel_cache.shard1.leader,
> > tag=SolrCore@59c22833
> > > 2022-03-25 08:47:58.596 INFO (qtp1847637306-38437) [c:run_sel_index
> > > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > > o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ...
> > SKIPPED
> > > (unnecessary).
> > > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > o.a.s.c.SolrCore
> > > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > > org.apache.solr.core.SolrCore@59c22833<mailto:
> > > org.apache.solr.core.SolrCore@59c22833>
> > > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > > o.a.s.m.SolrMetricManager Closing metric reporters for
> > > registry=solr.core.run_sel_cache.shard1.replica_n3,
> tag=SolrCore@59c22833
> > > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > > o.a.s.m.SolrMetricManager Closing metric reporters for
> > > registry=solr.collection.run_sel_cache.shard1.leader,
> > tag=SolrCore@59c22833
> > > 2022-03-25 08:47:59.258 WARN (qtp1847637306-38404) [ ]
> > > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > > listeners, but a live one attempted to unregister!
> > > 2022-03-25 08:47:59.259 ERROR (qtp1847637306-38404) [ ]
> > o.a.s.c.SolrCore
> > > java.lang.IllegalArgumentException: Unknown directory:
> > > NRTCachingDirectory(MMapDirectory@
> >
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92
> <mailto:
> > > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92>;
> > > maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
> > > at
> > >
> >
> org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
> > > at org.apache.solr.core.SolrCore.close(SolrCore.java:1674)
> > > at
> > >
> >
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:353)
> > > at
> > org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:46)
> > > at
> > >
> >
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
> > > at
> > >
> >
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
> > > at
> > >
> >
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
> > > at
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> > > at
> > > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
> > > at
> > >
> >
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
> > > at
> > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
> > > at
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> > > at
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > > at
> > >
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> > > at
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> > > at
> > >
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> > > at
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > > at
> > >
> >
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > > at org.eclipse.jetty.server.Server.handle(Server.java:500)
> > > at
> > >
> >
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> > > at
> > > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> > > at
> > > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> > > at
> > >
> >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
> > > at org.eclipse.jetty.io
> > > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> > > at org.eclipse.jetty.io
> > > .FillInterest.fillable(FillInterest.java:103)
> > > at org.eclipse.jetty.io
> > > .ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> > > at
> > >
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> > > at
> > >
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> > > at
> > >
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> > > at
> > >
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> > > at
> > >
> >
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
> > > at
> > >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
> > > at
> > >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> > > at java.lang.Thread.run(Thread.java:748)
> > >
> > > 2022-03-25 08:47:59.260 INFO (qtp1847637306-38404) [ ]
> > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> > > params={action=STATUS&indexInfo=false&wt=json} status=0 QTime=3
> > > 2022-03-25 08:47:59.262 INFO (qtp1847637306-37771) [ ]
> > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus
> > with
> > > params action=clusterstatus&wt=json and sendToOCPQueue=true
> > > 2022-03-25 08:47:59.275 INFO (qtp1847637306-37771) [ ]
> > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> > > params={action=clusterstatus&wt=json} status=0 QTime=13
> > > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > o.a.s.c.SolrCore
> > > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > > org.apache.solr.core.SolrCore@59c22833<mailto:
> > > org.apache.solr.core.SolrCore@59c22833>
> > > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > > o.a.s.m.SolrMetricManager Closing metric reporters for
> > > registry=solr.core.run_sel_cache.shard1.replica_n3,
> tag=SolrCore@59c22833
> > > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > > o.a.s.m.SolrMetricManager Closing metric reporters for
> > > registry=solr.collection.run_sel_cache.shard1.leader,
> > tag=SolrCore@59c22833
> > > 2022-03-25 08:48:09.579 WARN (qtp1847637306-38348) [ ]
> > > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > > listeners, but a live one attempted to unregister!
> > >
> >
>
RE: IllegalArgumentException: Unknown directory
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov.INVALID>.
This error has happened yet again. Does anyone yet have any explanation or suggestion?
-----Original Message-----
From: Oakley, Craig (NIH/NLM/NCBI) [C]
Sent: Thursday, October 13, 2022 6:09 PM
To: users@solr.apache.org
Subject: RE: IllegalArgumentException: Unknown directory
This error has happened again. Does anyone yet have any explanation or suggestion?
-----Original Message-----
From: Oakley, Craig (NIH/NLM/NCBI) [C] <cr...@nih.gov.INVALID>
Sent: Friday, August 05, 2022 10:47 PM
To: users@solr.apache.org
Subject: RE: IllegalArgumentException: Unknown directory
This error has happened again. Does anyone yet have any explanation or suggestion?
-----Original Message-----
From: Oakley, Craig (NIH/NLM/NCBI) [C]
Sent: Monday, May 02, 2022 2:29 PM
To: users@solr.apache.org
Subject: Re: IllegalArgumentException: Unknown directory
This has happened several more times, and I have notice something else which might be a clue:
The problem happened over the weekend; and the first erroneous complaint about /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata (as though it did not exist) occurred 17 minutes after /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/replication.properties was updated. Last month, the problem occurred just 40 seconds after replication.properties was updated.
What known connections might there be between replication.properties and snapshot_metadata?
-----Original Message-----
From: matthew sporleder <ms...@gmail.com>
Sent: Monday, March 28, 2022 9:41 AM
To: users@solr.apache.org
Subject: Re: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.
My only other guess (and I mean wild guess) is that because the error is
for a *lock* that it is actually a timeout or similar locking error with a
bad error message.
On Fri, Mar 25, 2022 at 11:11 AM Oakley, Craig (NIH/NLM/NCBI) [C]
<cr...@nih.gov.invalid> wrote:
> Thanks for the quick reply
>
> grep -i -c OutOfMemory solr.log.20220* shows zero
>
> Nothing new in /var/log/dmesg since reboot of the host a couple weeks ago
>
> Let me know if you have any other suggestions
>
> Thanks again
>
> -----Original Message-----
> From: matthew sporleder <ms...@gmail.com>
> Sent: Friday, March 25, 2022 11:00 AM
> To: users@solr.apache.org
> Subject: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and are
> confident the content is safe.
>
>
> Is there an OOM anywhere in that log? I've definitely seen java lose track
> of things during a slow-moving oom.
>
> Also definitely check dmesg for anything in that same timeframe if you
> still have the logs.
>
> On Fri, Mar 25, 2022 at 10:58 AM Oakley, Craig (NIH/NLM/NCBI) [C]
> <cr...@nih.gov.invalid> wrote:
>
> > I have a core which ceased responding either to select or to admin/core:
> > restarting the Solr instance resolved the problem, but I am wondering
> > whether there is some configuration which may need to be tweaked. Below
> is
> > a portion of solr.log from the time when the problem began. Please note
> > that the directory
> >
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > does indeed exist
> >
> > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.c.S.Request
> > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > hits=94 status=0 QTime=2610
> > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:47:58.527 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:47:58.531 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.r.SolrJmxReporter Closing reporter
> > [org.apache.solr.metrics.reporters.SolrJmxReporter@2b7f5d5e: rootName =
> > null, domain = solr.core.run_sel_cache.shard1.replica_n3, service url =
> > null, agent id = null] for registry
> > solr.core.run_sel_cache.shard1.replica_n3 /
> > com.codahale.metrics.MetricRegistry@5211d667<mailto:
> > com.codahale.metrics.MetricRegistry@5211d667>
> > 2022-03-25 08:47:58.568 INFO (qtp1847637306-38090) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.c.S.Request
> > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > hits=94 status=0 QTime=593
> > 2022-03-25 08:47:58.593 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:47:58.596 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ...
> SKIPPED
> > (unnecessary).
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:47:59.258 WARN (qtp1847637306-38404) [ ]
> > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > listeners, but a live one attempted to unregister!
> > 2022-03-25 08:47:59.259 ERROR (qtp1847637306-38404) [ ]
> o.a.s.c.SolrCore
> > java.lang.IllegalArgumentException: Unknown directory:
> > NRTCachingDirectory(MMapDirectory@
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92<mailto:
> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92>;
> > maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
> > at
> >
> org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
> > at org.apache.solr.core.SolrCore.close(SolrCore.java:1674)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:353)
> > at
> org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:46)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> > at
> > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
> > at
> >
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
> > at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> > at
> >
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at
> >
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at org.eclipse.jetty.server.Server.handle(Server.java:500)
> > at
> >
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> > at
> > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> > at
> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> > at
> >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
> > at org.eclipse.jetty.io
> > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> > at org.eclipse.jetty.io
> > .FillInterest.fillable(FillInterest.java:103)
> > at org.eclipse.jetty.io
> > .ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> > at
> >
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2022-03-25 08:47:59.260 INFO (qtp1847637306-38404) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> > params={action=STATUS&indexInfo=false&wt=json} status=0 QTime=3
> > 2022-03-25 08:47:59.262 INFO (qtp1847637306-37771) [ ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus
> with
> > params action=clusterstatus&wt=json and sendToOCPQueue=true
> > 2022-03-25 08:47:59.275 INFO (qtp1847637306-37771) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> > params={action=clusterstatus&wt=json} status=0 QTime=13
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:48:09.579 WARN (qtp1847637306-38348) [ ]
> > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > listeners, but a live one attempted to unregister!
> >
>
RE: IllegalArgumentException: Unknown directory
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov.INVALID>.
This error has happened again. Does anyone yet have any explanation or suggestion?
-----Original Message-----
From: Oakley, Craig (NIH/NLM/NCBI) [C] <cr...@nih.gov.INVALID>
Sent: Friday, August 05, 2022 10:47 PM
To: users@solr.apache.org
Subject: RE: IllegalArgumentException: Unknown directory
This error has happened again. Does anyone yet have any explanation or suggestion?
-----Original Message-----
From: Oakley, Craig (NIH/NLM/NCBI) [C]
Sent: Monday, May 02, 2022 2:29 PM
To: users@solr.apache.org
Subject: Re: IllegalArgumentException: Unknown directory
This has happened several more times, and I have notice something else which might be a clue:
The problem happened over the weekend; and the first erroneous complaint about /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata (as though it did not exist) occurred 17 minutes after /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/replication.properties was updated. Last month, the problem occurred just 40 seconds after replication.properties was updated.
What known connections might there be between replication.properties and snapshot_metadata?
-----Original Message-----
From: matthew sporleder <ms...@gmail.com>
Sent: Monday, March 28, 2022 9:41 AM
To: users@solr.apache.org
Subject: Re: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.
My only other guess (and I mean wild guess) is that because the error is
for a *lock* that it is actually a timeout or similar locking error with a
bad error message.
On Fri, Mar 25, 2022 at 11:11 AM Oakley, Craig (NIH/NLM/NCBI) [C]
<cr...@nih.gov.invalid> wrote:
> Thanks for the quick reply
>
> grep -i -c OutOfMemory solr.log.20220* shows zero
>
> Nothing new in /var/log/dmesg since reboot of the host a couple weeks ago
>
> Let me know if you have any other suggestions
>
> Thanks again
>
> -----Original Message-----
> From: matthew sporleder <ms...@gmail.com>
> Sent: Friday, March 25, 2022 11:00 AM
> To: users@solr.apache.org
> Subject: [EXTERNAL] Re: IllegalArgumentException: Unknown directory
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and are
> confident the content is safe.
>
>
> Is there an OOM anywhere in that log? I've definitely seen java lose track
> of things during a slow-moving oom.
>
> Also definitely check dmesg for anything in that same timeframe if you
> still have the logs.
>
> On Fri, Mar 25, 2022 at 10:58 AM Oakley, Craig (NIH/NLM/NCBI) [C]
> <cr...@nih.gov.invalid> wrote:
>
> > I have a core which ceased responding either to select or to admin/core:
> > restarting the Solr instance resolved the problem, but I am wondering
> > whether there is some configuration which may need to be tweaked. Below
> is
> > a portion of solr.log from the time when the problem began. Please note
> > that the directory
> >
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > does indeed exist
> >
> > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.c.S.Request
> > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > hits=94 status=0 QTime=2610
> > 2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:47:58.527 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:47:58.531 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.r.SolrJmxReporter Closing reporter
> > [org.apache.solr.metrics.reporters.SolrJmxReporter@2b7f5d5e: rootName =
> > null, domain = solr.core.run_sel_cache.shard1.replica_n3, service url =
> > null, agent id = null] for registry
> > solr.core.run_sel_cache.shard1.replica_n3 /
> > com.codahale.metrics.MetricRegistry@5211d667<mailto:
> > com.codahale.metrics.MetricRegistry@5211d667>
> > 2022-03-25 08:47:58.568 INFO (qtp1847637306-38090) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> o.a.s.c.S.Request
> > [run_sel_index_shard1_replica_n3] webapp=/solr path=/select
> >
> params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
> > hits=94 status=0 QTime=593
> > 2022-03-25 08:47:58.593 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:47:58.596 INFO (qtp1847637306-38437) [c:run_sel_index
> > s:shard1 r:core_node4 x:run_sel_index_shard1_replica_n3]
> > o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ...
> SKIPPED
> > (unnecessary).
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:47:59.258 WARN (qtp1847637306-38404) [ ]
> > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > listeners, but a live one attempted to unregister!
> > 2022-03-25 08:47:59.259 ERROR (qtp1847637306-38404) [ ]
> o.a.s.c.SolrCore
> > java.lang.IllegalArgumentException: Unknown directory:
> > NRTCachingDirectory(MMapDirectory@
> /data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92<mailto:
> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92>;
> > maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
> > at
> >
> org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
> > at org.apache.solr.core.SolrCore.close(SolrCore.java:1674)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:353)
> > at
> org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:46)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> > at
> > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
> > at
> >
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
> > at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> > at
> >
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at
> >
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> > at org.eclipse.jetty.server.Server.handle(Server.java:500)
> > at
> >
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> > at
> > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> > at
> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> > at
> >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
> > at org.eclipse.jetty.io
> > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> > at org.eclipse.jetty.io
> > .FillInterest.fillable(FillInterest.java:103)
> > at org.eclipse.jetty.io
> > .ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> > at
> >
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > 2022-03-25 08:47:59.260 INFO (qtp1847637306-38404) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> > params={action=STATUS&indexInfo=false&wt=json} status=0 QTime=3
> > 2022-03-25 08:47:59.262 INFO (qtp1847637306-37771) [ ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus
> with
> > params action=clusterstatus&wt=json and sendToOCPQueue=true
> > 2022-03-25 08:47:59.275 INFO (qtp1847637306-37771) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> > params={action=clusterstatus&wt=json} status=0 QTime=13
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> o.a.s.c.SolrCore
> > [run_sel_cache_shard1_replica_n3] CLOSING SolrCore
> > org.apache.solr.core.SolrCore@59c22833<mailto:
> > org.apache.solr.core.SolrCore@59c22833>
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
> > 2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
> > o.a.s.m.SolrMetricManager Closing metric reporters for
> > registry=solr.collection.run_sel_cache.shard1.leader,
> tag=SolrCore@59c22833
> > 2022-03-25 08:48:09.579 WARN (qtp1847637306-38348) [ ]
> > o.a.s.c.ZkController /configs/run_sel_cache has no more registered
> > listeners, but a live one attempted to unregister!
> >
>