You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Karl Stoney <Ka...@autotrader.co.uk.INVALID> on 2021/03/29 19:07:39 UTC
Failure to create collection: Lock Held
Hi,
I’ve got an odd problem where I’m unable to create a collection as it’s claiming a lock is held at /solr/solr-data/data/tech-data-001/index/write.lock, but that location (/solr/solr-data/data/tech-data-001) doesn’t exist on any of the nodes on the cluster.
Does anyone know what might be causing this?
Thanks
Karl
18:52:17.366 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Error from shard solr-0.search-solr.svc.cluster.local:80_solr: {responseHeader={status=0,QTime=0},STATUS=failed,Response=Error CREATEing SolrCore 'tech-data-001_shard1_replica_n6': Unable to create core [tech-data-001_shard1_replica_n6] Caused by: Lock held by this virtual machine: /solr/solr-data/data/tech-data-001/index/write.lock}
18:52:18.381 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Cleaning up collection [tech-data-001].
18:52:18.520 [Finalizer] ERROR org.apache.solr.update.SolrIndexWriter - SolrIndexWriter was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
18:52:18.520 [Finalizer] ERROR org.apache.solr.update.SolrIndexWriter - Error closing IndexWriter
java.lang.NullPointerException: null
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3927) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3887) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1235) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1278) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:298) [solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:357) [solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at java.lang.System$2.invokeFinalize(System.java:2125) [?:?]
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:87) [?:?]
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171) [?:?]
Suppressed: java.lang.NullPointerException
at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2435) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2410) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1241) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1278) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:298) [solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:357) [solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at java.lang.System$2.invokeFinalize(System.java:2125) [?:?]
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:87) [?:?]
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171) [?:?]
18:52:18.561 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Collection: tech-data-001 operation: create failed:org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: tech-data-001
at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:371)
at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:266)
at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:524)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
18:56:03.371 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Error from shard: http://solr-0.search-solr.svc.cluster.local:80/solr
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://solr-0.search-solr.svc.cluster.local:80/solr: Error CREATEing SolrCore 'tech-data-001_shard1_replica_n1': Unable to create core [tech-data-001_shard1_replica_n1] Caused by: Lock held by this virtual machine: /solr/solr-data/data/tech-data-001/index/write.lock
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:681) ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:04]
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266) ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:04]
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248) ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:04]
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:04]
at org.apache.solr.handler.component.HttpShardHandlerFactory$1.request(HttpShardHandlerFactory.java:169) ~[solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:130) ~[solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:41) ~[solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) ~[metrics-core-4.1.5.jar:4.1.5]
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218) ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:04]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
18:56:03.372 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Cleaning up collection [tech-data-001].
18:56:03.499 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Collection: tech-data-001 operation: create failed:org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: tech-data-001
at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:371)
at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:266)
at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:524)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Unless expressly stated otherwise in this email, this e-mail is sent on behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited is part of the Auto Trader Group Plc group. This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
Re: Failure to create collection: Lock Held
Posted by Karl Stoney <Ka...@autotrader.co.uk.INVALID>.
Looking at the stack trace the error is being thrown from here: https://github.com/apache/lucene-solr/blob/branch_8_8/lucene/core/src/java/org/apache/lucene/store/NativeFSLockFactory.java#L139
Which is an in memory set.
Feels like that set is (incorrectly) populated with the relevant path on two of the nodes, which is odd.
Feels like a restart may well solve it, but I’m hesitant to do so in case this is an opportunity to collect more information for a bug report?
Thanks
From: Karl Stoney <Ka...@autotrader.co.uk>
Date: Monday, 29 March 2021 at 20:22
To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
Subject: Re: Failure to create collection: Lock Held
To add to this, the request succeeds on 7/9 nodes, but 2/9 fail with this error
From: Karl Stoney <Ka...@autotrader.co.uk>
Date: Monday, 29 March 2021 at 20:07
To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
Subject: Failure to create collection: Lock Held
Hi,
I’ve got an odd problem where I’m unable to create a collection as it’s claiming a lock is held at /solr/solr-data/data/tech-data-001/index/write.lock, but that location (/solr/solr-data/data/tech-data-001) doesn’t exist on any of the nodes on the cluster.
Does anyone know what might be causing this?
Thanks
Karl
18:52:17.366 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Error from shard solr-0.search-solr.svc.cluster.local:80_solr: {responseHeader={status=0,QTime=0},STATUS=failed,Response=Error CREATEing SolrCore 'tech-data-001_shard1_replica_n6': Unable to create core [tech-data-001_shard1_replica_n6] Caused by: Lock held by this virtual machine: /solr/solr-data/data/tech-data-001/index/write.lock}
18:52:18.381 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Cleaning up collection [tech-data-001].
18:52:18.520 [Finalizer] ERROR org.apache.solr.update.SolrIndexWriter - SolrIndexWriter was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
18:52:18.520 [Finalizer] ERROR org.apache.solr.update.SolrIndexWriter - Error closing IndexWriter
java.lang.NullPointerException: null
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3927) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3887) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1235) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1278) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:298) [solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:357) [solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at java.lang.System$2.invokeFinalize(System.java:2125) [?:?]
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:87) [?:?]
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171) [?:?]
Suppressed: java.lang.NullPointerException
at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2435) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2410) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1241) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1278) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:298) [solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:357) [solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at java.lang.System$2.invokeFinalize(System.java:2125) [?:?]
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:87) [?:?]
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171) [?:?]
18:52:18.561 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Collection: tech-data-001 operation: create failed:org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: tech-data-001
at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:371)
at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:266)
at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:524)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
18:56:03.371 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Error from shard: http://solr-0.search-solr.svc.cluster.local:80/solr
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://solr-0.search-solr.svc.cluster.local:80/solr: Error CREATEing SolrCore 'tech-data-001_shard1_replica_n1': Unable to create core [tech-data-001_shard1_replica_n1] Caused by: Lock held by this virtual machine: /solr/solr-data/data/tech-data-001/index/write.lock
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:681) ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:04]
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266) ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:04]
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248) ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:04]
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:04]
at org.apache.solr.handler.component.HttpShardHandlerFactory$1.request(HttpShardHandlerFactory.java:169) ~[solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:130) ~[solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:41) ~[solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) ~[metrics-core-4.1.5.jar:4.1.5]
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218) ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:04]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
18:56:03.372 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Cleaning up collection [tech-data-001].
18:56:03.499 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Collection: tech-data-001 operation: create failed:org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: tech-data-001
at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:371)
at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:266)
at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:524)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Unless expressly stated otherwise in this email, this e-mail is sent on behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited is part of the Auto Trader Group Plc group. This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
Re: Failure to create collection: Lock Held
Posted by Karl Stoney <Ka...@autotrader.co.uk.INVALID>.
To add to this, the request succeeds on 7/9 nodes, but 2/9 fail with this error
From: Karl Stoney <Ka...@autotrader.co.uk>
Date: Monday, 29 March 2021 at 20:07
To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
Subject: Failure to create collection: Lock Held
Hi,
I’ve got an odd problem where I’m unable to create a collection as it’s claiming a lock is held at /solr/solr-data/data/tech-data-001/index/write.lock, but that location (/solr/solr-data/data/tech-data-001) doesn’t exist on any of the nodes on the cluster.
Does anyone know what might be causing this?
Thanks
Karl
18:52:17.366 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Error from shard solr-0.search-solr.svc.cluster.local:80_solr: {responseHeader={status=0,QTime=0},STATUS=failed,Response=Error CREATEing SolrCore 'tech-data-001_shard1_replica_n6': Unable to create core [tech-data-001_shard1_replica_n6] Caused by: Lock held by this virtual machine: /solr/solr-data/data/tech-data-001/index/write.lock}
18:52:18.381 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Cleaning up collection [tech-data-001].
18:52:18.520 [Finalizer] ERROR org.apache.solr.update.SolrIndexWriter - SolrIndexWriter was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
18:52:18.520 [Finalizer] ERROR org.apache.solr.update.SolrIndexWriter - Error closing IndexWriter
java.lang.NullPointerException: null
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3927) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3887) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1235) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1278) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:298) [solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:357) [solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at java.lang.System$2.invokeFinalize(System.java:2125) [?:?]
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:87) [?:?]
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171) [?:?]
Suppressed: java.lang.NullPointerException
at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2435) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2410) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1241) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1278) ~[lucene-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:47:53]
at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:298) [solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:357) [solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at java.lang.System$2.invokeFinalize(System.java:2125) [?:?]
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:87) [?:?]
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171) [?:?]
18:52:18.561 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Collection: tech-data-001 operation: create failed:org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: tech-data-001
at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:371)
at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:266)
at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:524)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
18:56:03.371 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Error from shard: http://solr-0.search-solr.svc.cluster.local:80/solr
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://solr-0.search-solr.svc.cluster.local:80/solr: Error CREATEing SolrCore 'tech-data-001_shard1_replica_n1': Unable to create core [tech-data-001_shard1_replica_n1] Caused by: Lock held by this virtual machine: /solr/solr-data/data/tech-data-001/index/write.lock
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:681) ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:04]
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266) ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:04]
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248) ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:04]
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:04]
at org.apache.solr.handler.component.HttpShardHandlerFactory$1.request(HttpShardHandlerFactory.java:169) ~[solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:130) ~[solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at org.apache.solr.handler.component.ShardRequestor.call(ShardRequestor.java:41) ~[solr-core-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:02]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) ~[metrics-core-4.1.5.jar:4.1.5]
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218) ~[solr-solrj-8.8.1.jar:8.8.1 64f3b496bfee762a9d2dbff40700f457f4464dfe - tjp - 2021-02-16 15:52:04]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
18:56:03.372 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Cleaning up collection [tech-data-001].
18:56:03.499 [OverseerThreadFactory-80-thread-5-processing-n:solr-8.search-solr.svc.cluster.local:80_solr] ERROR org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler - Collection: tech-data-001 operation: create failed:org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: tech-data-001
at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:371)
at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:266)
at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:524)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Unless expressly stated otherwise in this email, this e-mail is sent on behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited is part of the Auto Trader Group Plc group. This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.