You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Eugene Livis <el...@basistech.com> on 2020/04/08 21:17:33 UTC
Unable to RESTORE collections via Collections API
Hello,
I have been unsuccessfully trying to find a way to restore a collection
using the Collections API for the last several days. *I would
greatly appreciate any help as I am now stuck.* I am using Solr 8.2 in
cloud mode. To simplify things, I have only a single Solr node on my local
machine (though I have also tried multi-node setups). I am able to
successfully create collections, index documents, and run searches against
the index. General gist of what I am trying to do is to be able to backup
an existing collection, delete that collection, and then restore the
collection from backup when needed. Our use case requires creating
thousands of collections, which I believe is highly not recommended in Solr
Cloud (is that correct?), so this backup-delete-restore mechanism is our
way of reducing the number of collections in Solr at any given time. Just
FYI, in our use case it is ok for searches to take a couple of minutes to
produce results.
So far I have not been able to restore a single collection using the
collection RESTORE API. I am able to create backups by running the
following HTTP command in Solr admin console:
http://localhost:8983/solr/admin/collections?action=BACKUP&name=test1_backup2&collection=test1_20200407_170438_20200407_170441&location=C:\TEST\DELETE\BACKUPS
The backup appears to be successfully created. It contains a snapshot from
"shard1", which is the only shard in the cluster. It also contains the
"zk_backup" directory, which contains my Solr config and
"collection_state.json" file. And it contains a "backup.properties" file.
Everything looks good to me.
However, when I attempt to restore the collection using the following HTTP
command:
http://localhost:8983/solr/admin/collections?action=RESTORE&name=test1_backup2&location=C:\TEST\DELETE\BACKUPS&collection=test1_backup_NEW
I get the same extremely vague error messages in Solr logs:
*RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing
SolrCore 'test1_backup_NEW_shard1_replica_n1': Unable to create core
[test1_backup_NEW_shard1_replica_n1] Caused by: nullERROR
(qtp1099855928-24) [c:test1_backup_NEW ] o.a.s.s.HttpSolrCall
null:org.apache.solr.common.SolrException: ADDREPLICA failed to create
replica *
Below is the full Solr log of the restore command:
2020-04-08 21:10:56.753 INFO (qtp1099855928-24) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :restore with params
name=test1_backup2&action=RESTORE&location=C:\TEST\DELETE\BACKUPS&collection=test1_backup_NEW
and sendToOCPQueue=true
2020-04-08 21:10:56.783 INFO
(OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
[c:test1_backup_NEW ] o.a.s.c.a.c.RestoreCmd Using existing config
AutopsyConfig
2020-04-08 21:10:56.783 INFO
(OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
[c:test1_backup_NEW ] o.a.s.c.a.c.RestoreCmd Starting restore into
collection=test1_backup_NEW with backup_name=test1_backup2 at
location=file:///C:/TEST/DELETE/BACKUPS/
2020-04-08 21:10:56.784 INFO
(OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
[c:test1_backup_NEW ] o.a.s.c.a.c.CreateCollectionCmd Create collection
test1_backup_NEW
2020-04-08 21:10:56.900 WARN
(OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
[c:test1_backup_NEW ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to
create a collection (test1_backup_NEW) without cores.
2020-04-08 21:10:56.908 INFO
(OverseerStateUpdate-72057883336638464-localhost:8983_solr-n_0000000003) [
] o.a.s.c.o.SliceMutator Update shard state invoked for collection:
test1_backup_NEW with message: {
"shard1":"construction",
"collection":"test1_backup_NEW",
"operation":"updateshardstate"}
2020-04-08 21:10:56.908 INFO
(OverseerStateUpdate-72057883336638464-localhost:8983_solr-n_0000000003) [
] o.a.s.c.o.SliceMutator Update shard state shard1 to construction
2020-04-08 21:10:56.925 INFO
(OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
[c:test1_backup_NEW ] o.a.s.c.a.c.RestoreCmd Adding replica for
shard=shard1
collection=DocCollection(test1_backup_NEW//collections/test1_backup_NEW/state.json/0)={
"pullReplicas":0,
"replicationFactor":1,
"shards":{"shard1":{
"range":"80000000-7fffffff",
"state":"active",
"replicas":{}}},
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"false",
"nrtReplicas":1,
"tlogReplicas":0}
2020-04-08 21:10:56.928 INFO
(OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
[c:test1_backup_NEW ] o.a.s.c.a.c.AddReplicaCmd Node Identified
localhost:8983_solr for creating new replica of shard shard1 for collection
test1_backup_NEW
2020-04-08 21:10:56.939 INFO
(OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
[c:test1_backup_NEW ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica
command.
2020-04-08 21:10:56.945 INFO
(OverseerStateUpdate-72057883336638464-localhost:8983_solr-n_0000000003) [
] o.a.s.c.o.SliceMutator createReplica() {
"operation":"addreplica",
"collection":"test1_backup_NEW",
"shard":"shard1",
"core":"test1_backup_NEW_shard1_replica_n1",
"state":"down",
"base_url":"http://localhost:8983/solr",
"node_name":"localhost:8983_solr",
"type":"NRT"}
2020-04-08 21:10:57.348 INFO (qtp1099855928-19) [
x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core
create command
qt=/admin/cores&coreNodeName=core_node2&collection.configName=AutopsyConfig&name=test1_backup_NEW_shard1_replica_n1&action=CREATE&collection=test1_backup_NEW&shard=shard1&wt=javabin&version=2&replicaType=NRT
2020-04-08 21:10:57.373 WARN (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.c.SolrConfig Couldn't add files from C:\total\crap\dir\ignored
filtered by null to classpath: C:\total\crap\dir\ignored
2020-04-08 21:10:57.374 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.c.SolrResourceLoader [test1_backup_NEW_shard1_replica_n1] Added 56
libs to classloader, from paths:
[/C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/clustering/lib,
/C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/extraction/lib,
/C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/langid/lib,
/C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/velocity/lib]
2020-04-08 21:10:57.395 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.2.0
2020-04-08 21:10:57.424 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.s.IndexSchema [test1_backup_NEW_shard1_replica_n1] Schema
name=Autopsy Keyword Search
2020-04-08 21:10:57.451 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.s.IndexSchema Loaded schema Autopsy Keyword Search/2.2 with uniqueid
field id
2020-04-08 21:10:57.473 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.c.CoreContainer Creating SolrCore
'test1_backup_NEW_shard1_replica_n1' using configuration from collection
test1_backup_NEW, trusted=true
2020-04-08 21:10:57.473 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.core.test1_backup_NEW.shard1.replica_n1' (registry
'solr.core.test1_backup_NEW.shard1.replica_n1') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@214b199c
2020-04-08 21:10:57.473 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.c.SolrCore [[test1_backup_NEW_shard1_replica_n1] ] Opening new
SolrCore at
[C:\Bitnami\solr-8.2.0-2\apache-solr\server\solr\test1_backup_NEW_shard1_replica_n1],
dataDir=[C:\Bitnami\solr-8.2.0-2\apache-solr\server\solr\test1_backup_NEW_shard1_replica_n1\data\]
2020-04-08 21:10:57.491 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
2020-04-08 21:10:57.527 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.u.CommitTracker Hard AutoCommit: if 15000 uncommitted docs;
2020-04-08 21:10:57.527 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.u.CommitTracker Soft AutoCommit: disabled
2020-04-08 21:10:57.535 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@59a2b753[test1_backup_NEW_shard1_replica_n1]
main]
2020-04-08 21:10:57.536 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with
znodeBase: /configs/AutopsyConfig
2020-04-08 21:10:57.537 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/AutopsyConfig
2020-04-08 21:10:57.537 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.h.c.SpellCheckComponent Initializing spell checkers
2020-04-08 21:10:57.537 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.s.DirectSolrSpellChecker init:
{name=default,field=name,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
2020-04-08 21:10:57.540 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
2020-04-08 21:10:57.541 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.c.SolrCore [test1_backup_NEW_shard1_replica_n1] CLOSING SolrCore
org.apache.solr.core.SolrCore@3a4eb909
2020-04-08 21:10:57.541 INFO
(searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.QuerySenderListener
QuerySenderListener sending requests to
Searcher@59a2b753[test1_backup_NEW_shard1_replica_n1]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
2020-04-08 21:10:57.541 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.m.SolrMetricManager Closing metric reporters for
registry=solr.core.test1_backup_NEW.shard1.replica_n1, tag=3a4eb909
2020-04-08 21:10:57.542 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.m.r.SolrJmxReporter Closing reporter
[org.apache.solr.metrics.reporters.SolrJmxReporter@54eb018f: rootName =
null, domain = solr.core.test1_backup_NEW.shard1.replica_n1, service url =
null, agent id = null] for registry
solr.core.test1_backup_NEW.shard1.replica_n1 /
com.codahale.metrics.MetricRegistry@55bc77fd
2020-04-08 21:10:57.542 INFO
(searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.S.Request
[test1_backup_NEW_shard1_replica_n1] webapp=null path=null
params={q=static+firstSearcher+warming+in+solrconfig.xml&distrib=false&event=firstSearcher}
hits=0 status=0 QTime=1
2020-04-08 21:10:57.542 INFO
(searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.QuerySenderListener
QuerySenderListener done.
2020-04-08 21:10:57.542 INFO
(searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading
spell index for spellchecker: default
2020-04-08 21:10:57.542 INFO
(searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading
spell index for spellchecker: wordbreak
2020-04-08 21:10:57.542 INFO
(searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.SolrCore
[test1_backup_NEW_shard1_replica_n1] Registered new searcher
Searcher@59a2b753[test1_backup_NEW_shard1_replica_n1]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
2020-04-08 21:10:57.552 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.m.SolrMetricManager Closing metric reporters for
registry=solr.collection.test1_backup_NEW.shard1.leader, tag=3a4eb909
2020-04-08 21:10:57.560 ERROR (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error
CREATEing SolrCore 'test1_backup_NEW_shard1_replica_n1': Unable to create
core [test1_backup_NEW_shard1_replica_n1] Caused by: null
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1195)
at
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
at
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
at
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:396)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:820)
at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:786)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:505)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:311)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
at
org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
at
org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: Unable to create core
[test1_backup_NEW_shard1_replica_n1]
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1266)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1156)
... 50 more
Caused by: org.apache.solr.common.SolrException
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1055)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:882)
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1245)
... 51 more
Caused by: java.lang.NullPointerException
at
org.apache.solr.core.SolrCore.bufferUpdatesIfConstructing(SolrCore.java:1097)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1029)
... 53 more
2020-04-08 21:10:57.561 INFO (qtp1099855928-19) [c:test1_backup_NEW
s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=AutopsyConfig&name=test1_backup_NEW_shard1_replica_n1&action=CREATE&collection=test1_backup_NEW&shard=shard1&wt=javabin&version=2&replicaType=NRT}
status=400 QTime=213
2020-04-08 21:10:57.577 ERROR
(OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
[c:test1_backup_NEW ] o.a.s.c.a.c.OverseerCollectionMessageHandler Error
from shard: http://localhost:8983/solr =>
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:
Error from server at null: Error CREATEing SolrCore
'test1_backup_NEW_shard1_replica_n1': Unable to create core
[test1_backup_NEW_shard1_replica_n1] Caused by: null
at
org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:727)
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:
Error from server at null: Error CREATEing SolrCore
'test1_backup_NEW_shard1_replica_n1': Unable to create core
[test1_backup_NEW_shard1_replica_n1] Caused by: null
at
org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:727)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:400)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:746)
~[?:?]
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274)
~[?:?]
at
org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238)
~[?:?]
at
org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199)
~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_222-1-ojdkbuild]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_222-1-ojdkbuild]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_222-1-ojdkbuild]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
~[metrics-core-4.0.5.jar:4.0.5]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_222-1-ojdkbuild]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_222-1-ojdkbuild]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222-1-ojdkbuild]
2020-04-08 21:10:57.578 ERROR
(OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
[c:test1_backup_NEW ] o.a.s.c.a.c.OverseerCollectionMessageHandler
Collection: test1_backup_NEW operation: restore
failed:org.apache.solr.common.SolrException: ADDREPLICA failed to create
replica
at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1030)
at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1013)
at
org.apache.solr.cloud.api.collections.AddReplicaCmd.lambda$addReplica$1(AddReplicaCmd.java:177)
at
org.apache.solr.cloud.api.collections.AddReplicaCmd.addReplica(AddReplicaCmd.java:199)
at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.addReplica(OverseerCollectionMessageHandler.java:708)
at
org.apache.solr.cloud.api.collections.RestoreCmd.call(RestoreCmd.java:286)
at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:264)
at
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:505)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:
Error from server at null: Error CREATEing SolrCore
'test1_backup_NEW_shard1_replica_n1': Unable to create core
[test1_backup_NEW_shard1_replica_n1] Caused by: null
at
org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:727)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:400)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:746)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274)
at
org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238)
at
org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
... 4 more
2020-04-08 21:10:57.604 INFO (qtp1099855928-24) [c:test1_backup_NEW ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
params={name=test1_backup2&action=RESTORE&location=C:\TEST\DELETE\BACKUPS&collection=test1_backup_NEW}
status=500 QTime=850
2020-04-08 21:10:57.605 ERROR (qtp1099855928-24) [c:test1_backup_NEW ]
o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: ADDREPLICA
failed to create replica
at
org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)
at
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:280)
at
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:252)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:820)
at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:786)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:505)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
at java.lang.Thread.run(Thread.java:748)
2020-04-08 21:10:58.775 INFO
(OverseerCollectionConfigSetProcessor-72057883336638464-localhost:8983_solr-n_0000000003)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000014 doesn't exist. Requestor
may have disconnected from ZooKeeper
2020-04-08 21:11:16.690 INFO (qtp1099855928-21) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.test1_backup_NEW2.shard1.replica_n1:QUERY./select.requests&key=solr.core.test1_20200407_170438_20200407_170441.shard1.replica_n1:UPDATE./update.requests&key=solr.core.test1_backup_NEW2.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.test1_20200407_170438_20200407_170441.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.test1_backup_NEW2.shard1.replica_n1:UPDATE./update.requests&key=solr.core.xp_20200408_162046_20200408_162051.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.test1_backup_NEW.shard1.replica_n1:UPDATE./update.requests&key=solr.core.xp_20200408_162046_20200408_162051.shard1.replica_n1:UPDATE./update.requests&key=solr.core.test1_20200407_170438_20200407_170441.shard1.replica_n1:QUERY./select.requests&key=solr.core.xp_20200408_162046_20200408_162051.shard1.replica_n1:QUERY./select.requests&key=solr.core.test1_backup_NEW.shard1.replica_n1:QUERY./select.requests&key=solr.core.test1_backup_NEW.shard1.replica_n1:INDEX.sizeInBytes}
status=0 QTime=51
2020-04-08 21:11:16.731 INFO (qtp1099855928-24) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=27
Re: Unable to RESTORE collections via Collections API
Posted by Jan Høydahl <ja...@cominvent.com>.
Thanks for sharing your workaround!
Agree that error message is extremely unhelpful :)
Would you care to report it as a bug in JIRA?
Jan
> 15. apr. 2020 kl. 21:54 skrev Eugene Livis <el...@basistech.com>:
>
> In case this helps somebody in the future, given how completely unhelpful
> the Solr error message is - turns out the problem was occurring because in
> solrconfig.xml the updateLog was disabled. I have enabled updateLog the
> following way and "restore" operation started working:
>
> <updateLog>
> <str name="dir">${solr.ulog.dir:}</str>
> <int
> name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>
> </updateLog>
>
>
> On Wed, Apr 8, 2020 at 5:17 PM Eugene Livis <el...@basistech.com> wrote:
>
>> Hello,
>>
>> I have been unsuccessfully trying to find a way to restore a collection
>> using the Collections API for the last several days. *I would
>> greatly appreciate any help as I am now stuck.* I am using Solr 8.2 in
>> cloud mode. To simplify things, I have only a single Solr node on my local
>> machine (though I have also tried multi-node setups). I am able to
>> successfully create collections, index documents, and run searches against
>> the index. General gist of what I am trying to do is to be able to backup
>> an existing collection, delete that collection, and then restore the
>> collection from backup when needed. Our use case requires creating
>> thousands of collections, which I believe is highly not recommended in Solr
>> Cloud (is that correct?), so this backup-delete-restore mechanism is our
>> way of reducing the number of collections in Solr at any given time. Just
>> FYI, in our use case it is ok for searches to take a couple of minutes to
>> produce results.
>>
>> So far I have not been able to restore a single collection using the
>> collection RESTORE API. I am able to create backups by running the
>> following HTTP command in Solr admin console:
>>
>>
>> http://localhost:8983/solr/admin/collections?action=BACKUP&name=test1_backup2&collection=test1_20200407_170438_20200407_170441&location=C:\TEST\DELETE\BACKUPS
>>
>>
>> The backup appears to be successfully created. It contains a snapshot from
>> "shard1", which is the only shard in the cluster. It also contains the
>> "zk_backup" directory, which contains my Solr config and
>> "collection_state.json" file. And it contains a "backup.properties" file.
>> Everything looks good to me.
>>
>> However, when I attempt to restore the collection using the following HTTP
>> command:
>>
>>
>> http://localhost:8983/solr/admin/collections?action=RESTORE&name=test1_backup2&location=C:\TEST\DELETE\BACKUPS&collection=test1_backup_NEW
>>
>>
>> I get the same extremely vague error messages in Solr logs:
>>
>>
>> *RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing
>> SolrCore 'test1_backup_NEW_shard1_replica_n1': Unable to create core
>> [test1_backup_NEW_shard1_replica_n1] Caused by: nullERROR
>> (qtp1099855928-24) [c:test1_backup_NEW ] o.a.s.s.HttpSolrCall
>> null:org.apache.solr.common.SolrException: ADDREPLICA failed to create
>> replica *
>>
>> Below is the full Solr log of the restore command:
>>
>> 2020-04-08 21:10:56.753 INFO (qtp1099855928-24) [ ]
>> o.a.s.h.a.CollectionsHandler Invoked Collection Action :restore with params
>> name=test1_backup2&action=RESTORE&location=C:\TEST\DELETE\BACKUPS&collection=test1_backup_NEW
>> and sendToOCPQueue=true
>> 2020-04-08 21:10:56.783 INFO
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.RestoreCmd Using existing config
>> AutopsyConfig
>> 2020-04-08 21:10:56.783 INFO
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.RestoreCmd Starting restore into
>> collection=test1_backup_NEW with backup_name=test1_backup2 at
>> location=file:///C:/TEST/DELETE/BACKUPS/
>> 2020-04-08 21:10:56.784 INFO
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.CreateCollectionCmd Create collection
>> test1_backup_NEW
>> 2020-04-08 21:10:56.900 WARN
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to
>> create a collection (test1_backup_NEW) without cores.
>> 2020-04-08 21:10:56.908 INFO
>> (OverseerStateUpdate-72057883336638464-localhost:8983_solr-n_0000000003) [
>> ] o.a.s.c.o.SliceMutator Update shard state invoked for collection:
>> test1_backup_NEW with message: {
>> "shard1":"construction",
>> "collection":"test1_backup_NEW",
>> "operation":"updateshardstate"}
>> 2020-04-08 21:10:56.908 INFO
>> (OverseerStateUpdate-72057883336638464-localhost:8983_solr-n_0000000003) [
>> ] o.a.s.c.o.SliceMutator Update shard state shard1 to construction
>> 2020-04-08 21:10:56.925 INFO
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.RestoreCmd Adding replica for
>> shard=shard1
>> collection=DocCollection(test1_backup_NEW//collections/test1_backup_NEW/state.json/0)={
>> "pullReplicas":0,
>> "replicationFactor":1,
>> "shards":{"shard1":{
>> "range":"80000000-7fffffff",
>> "state":"active",
>> "replicas":{}}},
>> "router":{"name":"compositeId"},
>> "maxShardsPerNode":"1",
>> "autoAddReplicas":"false",
>> "nrtReplicas":1,
>> "tlogReplicas":0}
>> 2020-04-08 21:10:56.928 INFO
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.AddReplicaCmd Node Identified
>> localhost:8983_solr for creating new replica of shard shard1 for collection
>> test1_backup_NEW
>> 2020-04-08 21:10:56.939 INFO
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica
>> command.
>> 2020-04-08 21:10:56.945 INFO
>> (OverseerStateUpdate-72057883336638464-localhost:8983_solr-n_0000000003) [
>> ] o.a.s.c.o.SliceMutator createReplica() {
>> "operation":"addreplica",
>> "collection":"test1_backup_NEW",
>> "shard":"shard1",
>> "core":"test1_backup_NEW_shard1_replica_n1",
>> "state":"down",
>> "base_url":"http://localhost:8983/solr",
>> "node_name":"localhost:8983_solr",
>> "type":"NRT"}
>> 2020-04-08 21:10:57.348 INFO (qtp1099855928-19) [
>> x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core
>> create command
>> qt=/admin/cores&coreNodeName=core_node2&collection.configName=AutopsyConfig&name=test1_backup_NEW_shard1_replica_n1&action=CREATE&collection=test1_backup_NEW&shard=shard1&wt=javabin&version=2&replicaType=NRT
>> 2020-04-08 21:10:57.373 WARN (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.c.SolrConfig Couldn't add files from C:\total\crap\dir\ignored
>> filtered by null to classpath: C:\total\crap\dir\ignored
>> 2020-04-08 21:10:57.374 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.c.SolrResourceLoader [test1_backup_NEW_shard1_replica_n1] Added 56
>> libs to classloader, from paths:
>> [/C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/clustering/lib,
>> /C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/extraction/lib,
>> /C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/langid/lib,
>> /C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/velocity/lib]
>> 2020-04-08 21:10:57.395 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.2.0
>> 2020-04-08 21:10:57.424 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.s.IndexSchema [test1_backup_NEW_shard1_replica_n1] Schema
>> name=Autopsy Keyword Search
>> 2020-04-08 21:10:57.451 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.s.IndexSchema Loaded schema Autopsy Keyword Search/2.2 with uniqueid
>> field id
>> 2020-04-08 21:10:57.473 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.c.CoreContainer Creating SolrCore
>> 'test1_backup_NEW_shard1_replica_n1' using configuration from collection
>> test1_backup_NEW, trusted=true
>> 2020-04-08 21:10:57.473 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.m.r.SolrJmxReporter JMX monitoring for
>> 'solr.core.test1_backup_NEW.shard1.replica_n1' (registry
>> 'solr.core.test1_backup_NEW.shard1.replica_n1') enabled at server:
>> com.sun.jmx.mbeanserver.JmxMBeanServer@214b199c
>> 2020-04-08 21:10:57.473 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.c.SolrCore [[test1_backup_NEW_shard1_replica_n1] ] Opening new
>> SolrCore at
>> [C:\Bitnami\solr-8.2.0-2\apache-solr\server\solr\test1_backup_NEW_shard1_replica_n1],
>> dataDir=[C:\Bitnami\solr-8.2.0-2\apache-solr\server\solr\test1_backup_NEW_shard1_replica_n1\data\]
>> 2020-04-08 21:10:57.491 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
>> 2020-04-08 21:10:57.527 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.u.CommitTracker Hard AutoCommit: if 15000 uncommitted docs;
>> 2020-04-08 21:10:57.527 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.u.CommitTracker Soft AutoCommit: disabled
>> 2020-04-08 21:10:57.535 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.s.SolrIndexSearcher Opening [Searcher@59a2b753[test1_backup_NEW_shard1_replica_n1]
>> main]
>> 2020-04-08 21:10:57.536 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with
>> znodeBase: /configs/AutopsyConfig
>> 2020-04-08 21:10:57.537 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
>> ZooKeeperStorageIO:path=/configs/AutopsyConfig
>> 2020-04-08 21:10:57.537 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.h.c.SpellCheckComponent Initializing spell checkers
>> 2020-04-08 21:10:57.537 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.s.DirectSolrSpellChecker init:
>> {name=default,field=name,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
>> 2020-04-08 21:10:57.540 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
>> 2020-04-08 21:10:57.541 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.c.SolrCore [test1_backup_NEW_shard1_replica_n1] CLOSING SolrCore
>> org.apache.solr.core.SolrCore@3a4eb909
>> 2020-04-08 21:10:57.541 INFO
>> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
>> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
>> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
>> x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.QuerySenderListener
>> QuerySenderListener sending requests to Searcher@59a2b753[test1_backup_NEW_shard1_replica_n1]
>> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>> 2020-04-08 21:10:57.541 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.m.SolrMetricManager Closing metric reporters for
>> registry=solr.core.test1_backup_NEW.shard1.replica_n1, tag=3a4eb909
>> 2020-04-08 21:10:57.542 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.m.r.SolrJmxReporter Closing reporter
>> [org.apache.solr.metrics.reporters.SolrJmxReporter@54eb018f: rootName =
>> null, domain = solr.core.test1_backup_NEW.shard1.replica_n1, service url =
>> null, agent id = null] for registry
>> solr.core.test1_backup_NEW.shard1.replica_n1 /
>> com.codahale.metrics.MetricRegistry@55bc77fd
>> 2020-04-08 21:10:57.542 INFO
>> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
>> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
>> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
>> x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.S.Request
>> [test1_backup_NEW_shard1_replica_n1] webapp=null path=null
>> params={q=static+firstSearcher+warming+in+solrconfig.xml&distrib=false&event=firstSearcher}
>> hits=0 status=0 QTime=1
>> 2020-04-08 21:10:57.542 INFO
>> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
>> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
>> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
>> x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.QuerySenderListener
>> QuerySenderListener done.
>> 2020-04-08 21:10:57.542 INFO
>> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
>> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
>> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
>> x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading
>> spell index for spellchecker: default
>> 2020-04-08 21:10:57.542 INFO
>> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
>> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
>> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
>> x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading
>> spell index for spellchecker: wordbreak
>> 2020-04-08 21:10:57.542 INFO
>> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
>> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
>> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
>> x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.SolrCore
>> [test1_backup_NEW_shard1_replica_n1] Registered new searcher
>> Searcher@59a2b753[test1_backup_NEW_shard1_replica_n1]
>> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>> 2020-04-08 21:10:57.552 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.m.SolrMetricManager Closing metric reporters for
>> registry=solr.collection.test1_backup_NEW.shard1.leader, tag=3a4eb909
>> 2020-04-08 21:10:57.560 ERROR (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error
>> CREATEing SolrCore 'test1_backup_NEW_shard1_replica_n1': Unable to create
>> core [test1_backup_NEW_shard1_replica_n1] Caused by: null
>> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1195)
>> at
>> org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
>> at
>> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
>> at
>> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:396)
>> at
>> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
>> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
>> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:820)
>> at
>> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:786)
>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>> at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>> at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
>> at
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> at
>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> at org.eclipse.jetty.server.Server.handle(Server.java:505)
>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
>> at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:311)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
>> at
>> org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
>> at
>> org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
>> at
>> org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>> at
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: org.apache.solr.common.SolrException: Unable to create core
>> [test1_backup_NEW_shard1_replica_n1]
>> at
>> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1266)
>> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1156)
>> ... 50 more
>> Caused by: org.apache.solr.common.SolrException
>> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1055)
>> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:882)
>> at
>> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1245)
>> ... 51 more
>> Caused by: java.lang.NullPointerException
>> at
>> org.apache.solr.core.SolrCore.bufferUpdatesIfConstructing(SolrCore.java:1097)
>> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1029)
>> ... 53 more
>>
>> 2020-04-08 21:10:57.561 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
>> params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=AutopsyConfig&name=test1_backup_NEW_shard1_replica_n1&action=CREATE&collection=test1_backup_NEW&shard=shard1&wt=javabin&version=2&replicaType=NRT}
>> status=400 QTime=213
>> 2020-04-08 21:10:57.577 ERROR
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.OverseerCollectionMessageHandler Error
>> from shard: http://localhost:8983/solr =>
>> org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:
>> Error from server at null: Error CREATEing SolrCore
>> 'test1_backup_NEW_shard1_replica_n1': Unable to create core
>> [test1_backup_NEW_shard1_replica_n1] Caused by: null
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:727)
>> org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:
>> Error from server at null: Error CREATEing SolrCore
>> 'test1_backup_NEW_shard1_replica_n1': Unable to create core
>> [test1_backup_NEW_shard1_replica_n1] Caused by: null
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:727)
>> ~[?:?]
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:400)
>> ~[?:?]
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:746)
>> ~[?:?]
>> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274)
>> ~[?:?]
>> at
>> org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238)
>> ~[?:?]
>> at
>> org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199)
>> ~[?:?]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> ~[?:1.8.0_222-1-ojdkbuild]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> ~[?:1.8.0_222-1-ojdkbuild]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> ~[?:1.8.0_222-1-ojdkbuild]
>> at
>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
>> ~[metrics-core-4.0.5.jar:4.0.5]
>> at
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
>> ~[?:?]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> ~[?:1.8.0_222-1-ojdkbuild]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> ~[?:1.8.0_222-1-ojdkbuild]
>> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222-1-ojdkbuild]
>> 2020-04-08 21:10:57.578 ERROR
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.OverseerCollectionMessageHandler
>> Collection: test1_backup_NEW operation: restore
>> failed:org.apache.solr.common.SolrException: ADDREPLICA failed to create
>> replica
>> at
>> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1030)
>> at
>> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1013)
>> at
>> org.apache.solr.cloud.api.collections.AddReplicaCmd.lambda$addReplica$1(AddReplicaCmd.java:177)
>> at
>> org.apache.solr.cloud.api.collections.AddReplicaCmd.addReplica(AddReplicaCmd.java:199)
>> at
>> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.addReplica(OverseerCollectionMessageHandler.java:708)
>> at
>> org.apache.solr.cloud.api.collections.RestoreCmd.call(RestoreCmd.java:286)
>> at
>> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:264)
>> at
>> org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:505)
>> at
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by:
>> org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:
>> Error from server at null: Error CREATEing SolrCore
>> 'test1_backup_NEW_shard1_replica_n1': Unable to create core
>> [test1_backup_NEW_shard1_replica_n1] Caused by: null
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:727)
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:400)
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:746)
>> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274)
>> at
>> org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238)
>> at
>> org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
>> ... 4 more
>>
>> 2020-04-08 21:10:57.604 INFO (qtp1099855928-24) [c:test1_backup_NEW ]
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
>> params={name=test1_backup2&action=RESTORE&location=C:\TEST\DELETE\BACKUPS&collection=test1_backup_NEW}
>> status=500 QTime=850
>> 2020-04-08 21:10:57.605 ERROR (qtp1099855928-24) [c:test1_backup_NEW ]
>> o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: ADDREPLICA
>> failed to create replica
>> at
>> org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)
>> at
>> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:280)
>> at
>> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:252)
>> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
>> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:820)
>> at
>> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:786)
>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>> at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>> at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
>> at
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> at
>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> at org.eclipse.jetty.server.Server.handle(Server.java:505)
>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
>> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
>> at
>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>> at
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
>> at java.lang.Thread.run(Thread.java:748)
>>
>> 2020-04-08 21:10:58.775 INFO
>> (OverseerCollectionConfigSetProcessor-72057883336638464-localhost:8983_solr-n_0000000003)
>> [ ] o.a.s.c.OverseerTaskQueue Response ZK path:
>> /overseer/collection-queue-work/qnr-0000000014 doesn't exist. Requestor
>> may have disconnected from ZooKeeper
>> 2020-04-08 21:11:16.690 INFO (qtp1099855928-21) [ ]
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
>> params={wt=javabin&version=2&key=solr.core.test1_backup_NEW2.shard1.replica_n1:QUERY./select.requests&key=solr.core.test1_20200407_170438_20200407_170441.shard1.replica_n1:UPDATE./update.requests&key=solr.core.test1_backup_NEW2.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.test1_20200407_170438_20200407_170441.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.test1_backup_NEW2.shard1.replica_n1:UPDATE./update.requests&key=solr.core.xp_20200408_162046_20200408_162051.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.test1_backup_NEW.shard1.replica_n1:UPDATE./update.requests&key=solr.core.xp_20200408_162046_20200408_162051.shard1.replica_n1:UPDATE./update.requests&key=solr.core.test1_20200407_170438_20200407_170441.shard1.replica_n1:QUERY./select.requests&key=solr.core.xp_20200408_162046_20200408_162051.shard1.replica_n1:QUERY./select.requests&key=solr.core.test1_backup_NEW.shard1.replica_n1:QUERY./select.requests&key=solr.core.test1_backup_NEW.shard1.replica_n1:INDEX.sizeInBytes}
>> status=0 QTime=51
>> 2020-04-08 21:11:16.731 INFO (qtp1099855928-24) [ ]
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
>> params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
>> status=0 QTime=27
>>
>>
>>
Re: Unable to RESTORE collections via Collections API
Posted by Eugene Livis <el...@basistech.com>.
In case this helps somebody in the future, given how completely unhelpful
the Solr error message is - turns out the problem was occurring because in
solrconfig.xml the updateLog was disabled. I have enabled updateLog the
following way and "restore" operation started working:
<updateLog>
<str name="dir">${solr.ulog.dir:}</str>
<int
name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>
</updateLog>
On Wed, Apr 8, 2020 at 5:17 PM Eugene Livis <el...@basistech.com> wrote:
> Hello,
>
> I have been unsuccessfully trying to find a way to restore a collection
> using the Collections API for the last several days. *I would
> greatly appreciate any help as I am now stuck.* I am using Solr 8.2 in
> cloud mode. To simplify things, I have only a single Solr node on my local
> machine (though I have also tried multi-node setups). I am able to
> successfully create collections, index documents, and run searches against
> the index. General gist of what I am trying to do is to be able to backup
> an existing collection, delete that collection, and then restore the
> collection from backup when needed. Our use case requires creating
> thousands of collections, which I believe is highly not recommended in Solr
> Cloud (is that correct?), so this backup-delete-restore mechanism is our
> way of reducing the number of collections in Solr at any given time. Just
> FYI, in our use case it is ok for searches to take a couple of minutes to
> produce results.
>
> So far I have not been able to restore a single collection using the
> collection RESTORE API. I am able to create backups by running the
> following HTTP command in Solr admin console:
>
>
> http://localhost:8983/solr/admin/collections?action=BACKUP&name=test1_backup2&collection=test1_20200407_170438_20200407_170441&location=C:\TEST\DELETE\BACKUPS
>
>
> The backup appears to be successfully created. It contains a snapshot from
> "shard1", which is the only shard in the cluster. It also contains the
> "zk_backup" directory, which contains my Solr config and
> "collection_state.json" file. And it contains a "backup.properties" file.
> Everything looks good to me.
>
> However, when I attempt to restore the collection using the following HTTP
> command:
>
>
> http://localhost:8983/solr/admin/collections?action=RESTORE&name=test1_backup2&location=C:\TEST\DELETE\BACKUPS&collection=test1_backup_NEW
>
>
> I get the same extremely vague error messages in Solr logs:
>
>
> *RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing
> SolrCore 'test1_backup_NEW_shard1_replica_n1': Unable to create core
> [test1_backup_NEW_shard1_replica_n1] Caused by: nullERROR
> (qtp1099855928-24) [c:test1_backup_NEW ] o.a.s.s.HttpSolrCall
> null:org.apache.solr.common.SolrException: ADDREPLICA failed to create
> replica *
>
> Below is the full Solr log of the restore command:
>
> 2020-04-08 21:10:56.753 INFO (qtp1099855928-24) [ ]
> o.a.s.h.a.CollectionsHandler Invoked Collection Action :restore with params
> name=test1_backup2&action=RESTORE&location=C:\TEST\DELETE\BACKUPS&collection=test1_backup_NEW
> and sendToOCPQueue=true
> 2020-04-08 21:10:56.783 INFO
> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
> [c:test1_backup_NEW ] o.a.s.c.a.c.RestoreCmd Using existing config
> AutopsyConfig
> 2020-04-08 21:10:56.783 INFO
> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
> [c:test1_backup_NEW ] o.a.s.c.a.c.RestoreCmd Starting restore into
> collection=test1_backup_NEW with backup_name=test1_backup2 at
> location=file:///C:/TEST/DELETE/BACKUPS/
> 2020-04-08 21:10:56.784 INFO
> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
> [c:test1_backup_NEW ] o.a.s.c.a.c.CreateCollectionCmd Create collection
> test1_backup_NEW
> 2020-04-08 21:10:56.900 WARN
> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
> [c:test1_backup_NEW ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to
> create a collection (test1_backup_NEW) without cores.
> 2020-04-08 21:10:56.908 INFO
> (OverseerStateUpdate-72057883336638464-localhost:8983_solr-n_0000000003) [
> ] o.a.s.c.o.SliceMutator Update shard state invoked for collection:
> test1_backup_NEW with message: {
> "shard1":"construction",
> "collection":"test1_backup_NEW",
> "operation":"updateshardstate"}
> 2020-04-08 21:10:56.908 INFO
> (OverseerStateUpdate-72057883336638464-localhost:8983_solr-n_0000000003) [
> ] o.a.s.c.o.SliceMutator Update shard state shard1 to construction
> 2020-04-08 21:10:56.925 INFO
> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
> [c:test1_backup_NEW ] o.a.s.c.a.c.RestoreCmd Adding replica for
> shard=shard1
> collection=DocCollection(test1_backup_NEW//collections/test1_backup_NEW/state.json/0)={
> "pullReplicas":0,
> "replicationFactor":1,
> "shards":{"shard1":{
> "range":"80000000-7fffffff",
> "state":"active",
> "replicas":{}}},
> "router":{"name":"compositeId"},
> "maxShardsPerNode":"1",
> "autoAddReplicas":"false",
> "nrtReplicas":1,
> "tlogReplicas":0}
> 2020-04-08 21:10:56.928 INFO
> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
> [c:test1_backup_NEW ] o.a.s.c.a.c.AddReplicaCmd Node Identified
> localhost:8983_solr for creating new replica of shard shard1 for collection
> test1_backup_NEW
> 2020-04-08 21:10:56.939 INFO
> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
> [c:test1_backup_NEW ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica
> command.
> 2020-04-08 21:10:56.945 INFO
> (OverseerStateUpdate-72057883336638464-localhost:8983_solr-n_0000000003) [
> ] o.a.s.c.o.SliceMutator createReplica() {
> "operation":"addreplica",
> "collection":"test1_backup_NEW",
> "shard":"shard1",
> "core":"test1_backup_NEW_shard1_replica_n1",
> "state":"down",
> "base_url":"http://localhost:8983/solr",
> "node_name":"localhost:8983_solr",
> "type":"NRT"}
> 2020-04-08 21:10:57.348 INFO (qtp1099855928-19) [
> x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core
> create command
> qt=/admin/cores&coreNodeName=core_node2&collection.configName=AutopsyConfig&name=test1_backup_NEW_shard1_replica_n1&action=CREATE&collection=test1_backup_NEW&shard=shard1&wt=javabin&version=2&replicaType=NRT
> 2020-04-08 21:10:57.373 WARN (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.c.SolrConfig Couldn't add files from C:\total\crap\dir\ignored
> filtered by null to classpath: C:\total\crap\dir\ignored
> 2020-04-08 21:10:57.374 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.c.SolrResourceLoader [test1_backup_NEW_shard1_replica_n1] Added 56
> libs to classloader, from paths:
> [/C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/clustering/lib,
> /C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/extraction/lib,
> /C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/langid/lib,
> /C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/velocity/lib]
> 2020-04-08 21:10:57.395 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.2.0
> 2020-04-08 21:10:57.424 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.s.IndexSchema [test1_backup_NEW_shard1_replica_n1] Schema
> name=Autopsy Keyword Search
> 2020-04-08 21:10:57.451 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.s.IndexSchema Loaded schema Autopsy Keyword Search/2.2 with uniqueid
> field id
> 2020-04-08 21:10:57.473 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.c.CoreContainer Creating SolrCore
> 'test1_backup_NEW_shard1_replica_n1' using configuration from collection
> test1_backup_NEW, trusted=true
> 2020-04-08 21:10:57.473 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr.core.test1_backup_NEW.shard1.replica_n1' (registry
> 'solr.core.test1_backup_NEW.shard1.replica_n1') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@214b199c
> 2020-04-08 21:10:57.473 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.c.SolrCore [[test1_backup_NEW_shard1_replica_n1] ] Opening new
> SolrCore at
> [C:\Bitnami\solr-8.2.0-2\apache-solr\server\solr\test1_backup_NEW_shard1_replica_n1],
> dataDir=[C:\Bitnami\solr-8.2.0-2\apache-solr\server\solr\test1_backup_NEW_shard1_replica_n1\data\]
> 2020-04-08 21:10:57.491 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
> 2020-04-08 21:10:57.527 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.u.CommitTracker Hard AutoCommit: if 15000 uncommitted docs;
> 2020-04-08 21:10:57.527 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.u.CommitTracker Soft AutoCommit: disabled
> 2020-04-08 21:10:57.535 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.s.SolrIndexSearcher Opening [Searcher@59a2b753[test1_backup_NEW_shard1_replica_n1]
> main]
> 2020-04-08 21:10:57.536 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with
> znodeBase: /configs/AutopsyConfig
> 2020-04-08 21:10:57.537 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
> ZooKeeperStorageIO:path=/configs/AutopsyConfig
> 2020-04-08 21:10:57.537 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.h.c.SpellCheckComponent Initializing spell checkers
> 2020-04-08 21:10:57.537 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.s.DirectSolrSpellChecker init:
> {name=default,field=name,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
> 2020-04-08 21:10:57.540 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
> 2020-04-08 21:10:57.541 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.c.SolrCore [test1_backup_NEW_shard1_replica_n1] CLOSING SolrCore
> org.apache.solr.core.SolrCore@3a4eb909
> 2020-04-08 21:10:57.541 INFO
> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
> x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.QuerySenderListener
> QuerySenderListener sending requests to Searcher@59a2b753[test1_backup_NEW_shard1_replica_n1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> 2020-04-08 21:10:57.541 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.core.test1_backup_NEW.shard1.replica_n1, tag=3a4eb909
> 2020-04-08 21:10:57.542 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.m.r.SolrJmxReporter Closing reporter
> [org.apache.solr.metrics.reporters.SolrJmxReporter@54eb018f: rootName =
> null, domain = solr.core.test1_backup_NEW.shard1.replica_n1, service url =
> null, agent id = null] for registry
> solr.core.test1_backup_NEW.shard1.replica_n1 /
> com.codahale.metrics.MetricRegistry@55bc77fd
> 2020-04-08 21:10:57.542 INFO
> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
> x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.S.Request
> [test1_backup_NEW_shard1_replica_n1] webapp=null path=null
> params={q=static+firstSearcher+warming+in+solrconfig.xml&distrib=false&event=firstSearcher}
> hits=0 status=0 QTime=1
> 2020-04-08 21:10:57.542 INFO
> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
> x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.QuerySenderListener
> QuerySenderListener done.
> 2020-04-08 21:10:57.542 INFO
> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
> x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading
> spell index for spellchecker: default
> 2020-04-08 21:10:57.542 INFO
> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
> x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading
> spell index for spellchecker: wordbreak
> 2020-04-08 21:10:57.542 INFO
> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
> x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.SolrCore
> [test1_backup_NEW_shard1_replica_n1] Registered new searcher
> Searcher@59a2b753[test1_backup_NEW_shard1_replica_n1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> 2020-04-08 21:10:57.552 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.m.SolrMetricManager Closing metric reporters for
> registry=solr.collection.test1_backup_NEW.shard1.leader, tag=3a4eb909
> 2020-04-08 21:10:57.560 ERROR (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error
> CREATEing SolrCore 'test1_backup_NEW_shard1_replica_n1': Unable to create
> core [test1_backup_NEW_shard1_replica_n1] Caused by: null
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1195)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
> at
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:396)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:820)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:786)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.server.Server.handle(Server.java:505)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
> at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:311)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
> at
> org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
> at
> org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
> at
> org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.solr.common.SolrException: Unable to create core
> [test1_backup_NEW_shard1_replica_n1]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1266)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1156)
> ... 50 more
> Caused by: org.apache.solr.common.SolrException
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1055)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:882)
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1245)
> ... 51 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.solr.core.SolrCore.bufferUpdatesIfConstructing(SolrCore.java:1097)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1029)
> ... 53 more
>
> 2020-04-08 21:10:57.561 INFO (qtp1099855928-19) [c:test1_backup_NEW
> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=AutopsyConfig&name=test1_backup_NEW_shard1_replica_n1&action=CREATE&collection=test1_backup_NEW&shard=shard1&wt=javabin&version=2&replicaType=NRT}
> status=400 QTime=213
> 2020-04-08 21:10:57.577 ERROR
> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
> [c:test1_backup_NEW ] o.a.s.c.a.c.OverseerCollectionMessageHandler Error
> from shard: http://localhost:8983/solr =>
> org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:
> Error from server at null: Error CREATEing SolrCore
> 'test1_backup_NEW_shard1_replica_n1': Unable to create core
> [test1_backup_NEW_shard1_replica_n1] Caused by: null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:727)
> org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:
> Error from server at null: Error CREATEing SolrCore
> 'test1_backup_NEW_shard1_replica_n1': Unable to create core
> [test1_backup_NEW_shard1_replica_n1] Caused by: null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:727)
> ~[?:?]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:400)
> ~[?:?]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:746)
> ~[?:?]
> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274)
> ~[?:?]
> at
> org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238)
> ~[?:?]
> at
> org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199)
> ~[?:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_222-1-ojdkbuild]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_222-1-ojdkbuild]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_222-1-ojdkbuild]
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
> ~[metrics-core-4.0.5.jar:4.0.5]
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_222-1-ojdkbuild]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_222-1-ojdkbuild]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222-1-ojdkbuild]
> 2020-04-08 21:10:57.578 ERROR
> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
> [c:test1_backup_NEW ] o.a.s.c.a.c.OverseerCollectionMessageHandler
> Collection: test1_backup_NEW operation: restore
> failed:org.apache.solr.common.SolrException: ADDREPLICA failed to create
> replica
> at
> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1030)
> at
> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1013)
> at
> org.apache.solr.cloud.api.collections.AddReplicaCmd.lambda$addReplica$1(AddReplicaCmd.java:177)
> at
> org.apache.solr.cloud.api.collections.AddReplicaCmd.addReplica(AddReplicaCmd.java:199)
> at
> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.addReplica(OverseerCollectionMessageHandler.java:708)
> at
> org.apache.solr.cloud.api.collections.RestoreCmd.call(RestoreCmd.java:286)
> at
> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:264)
> at
> org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:505)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:
> Error from server at null: Error CREATEing SolrCore
> 'test1_backup_NEW_shard1_replica_n1': Unable to create core
> [test1_backup_NEW_shard1_replica_n1] Caused by: null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:727)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:400)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:746)
> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274)
> at
> org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238)
> at
> org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
> ... 4 more
>
> 2020-04-08 21:10:57.604 INFO (qtp1099855928-24) [c:test1_backup_NEW ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> params={name=test1_backup2&action=RESTORE&location=C:\TEST\DELETE\BACKUPS&collection=test1_backup_NEW}
> status=500 QTime=850
> 2020-04-08 21:10:57.605 ERROR (qtp1099855928-24) [c:test1_backup_NEW ]
> o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: ADDREPLICA
> failed to create replica
> at
> org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)
> at
> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:280)
> at
> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:252)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:820)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:786)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.server.Server.handle(Server.java:505)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
> at java.lang.Thread.run(Thread.java:748)
>
> 2020-04-08 21:10:58.775 INFO
> (OverseerCollectionConfigSetProcessor-72057883336638464-localhost:8983_solr-n_0000000003)
> [ ] o.a.s.c.OverseerTaskQueue Response ZK path:
> /overseer/collection-queue-work/qnr-0000000014 doesn't exist. Requestor
> may have disconnected from ZooKeeper
> 2020-04-08 21:11:16.690 INFO (qtp1099855928-21) [ ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
> params={wt=javabin&version=2&key=solr.core.test1_backup_NEW2.shard1.replica_n1:QUERY./select.requests&key=solr.core.test1_20200407_170438_20200407_170441.shard1.replica_n1:UPDATE./update.requests&key=solr.core.test1_backup_NEW2.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.test1_20200407_170438_20200407_170441.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.test1_backup_NEW2.shard1.replica_n1:UPDATE./update.requests&key=solr.core.xp_20200408_162046_20200408_162051.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.test1_backup_NEW.shard1.replica_n1:UPDATE./update.requests&key=solr.core.xp_20200408_162046_20200408_162051.shard1.replica_n1:UPDATE./update.requests&key=solr.core.test1_20200407_170438_20200407_170441.shard1.replica_n1:QUERY./select.requests&key=solr.core.xp_20200408_162046_20200408_162051.shard1.replica_n1:QUERY./select.requests&key=solr.core.test1_backup_NEW.shard1.replica_n1:QUERY./select.requests&key=solr.core.test1_backup_NEW.shard1.replica_n1:INDEX.sizeInBytes}
> status=0 QTime=51
> 2020-04-08 21:11:16.731 INFO (qtp1099855928-24) [ ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
> params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
> status=0 QTime=27
>
>
>