You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by HaiXin Tie <ht...@tivo.com> on 2013/09/17 03:44:16 UTC
SPLITSHARD failure right before publishing the new sub-shards
Hi Solr experts,
I am using Solr 4.4 with ZK 3.4.5, trying to split "shard1" of a
collection named "body". There is only one core on one machine for this
collection. When I call SPLITSHARD to split this collection, Solr is
able to create two sub-shards, but failed with a NPE in SolrCore.java
while publishing the new shards. It seems that either the updateHandler
or its updateLog is null, though they work fine in the original shard:
SolrCore.java
if (cc != null && cc.isZooKeeperAware() &&
Slice.CONSTRUCTION.equals(cd.getCloudDescriptor().getShardState())) {
// set update log to buffer before publishing the core
862: getUpdateHandler().getUpdateLog().bufferUpdates();
cd.getCloudDescriptor().setShardState(null);
cd.getCloudDescriptor().setShardRange(null);
}
Here are the details. Any pointers to aid debugging this issue is
greatly appreciated!
# curl request/response to split the shard:
curl -s
"http://localhost:8983/solr/admin/collections?action=SPLITSHARD&collection=body&shard=shard1"
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">500</int><int
name="QTime">2688</int></lst><lst
name="failure"><str>org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:Error
CREATEing SolrCore 'body_shard1_0_replica1': Unable to create core:
body_shard1_0_replica1 Caused by: null</str></lst><str name="Operation
splitshard caused
exception:">org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
SPLTSHARD failed to create subshard leaders</str><lst
name="exception"><str name="msg">SPLTSHARD failed to create subshard
leaders</str><int name="rspCode">500</int></lst><lst name="error"><str
name="msg">SPLTSHARD failed to create subshard leaders</str><str
name="trace">org.apache.solr.common.SolrException: SPLTSHARD failed to
create subshard leaders
at
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:171)
at
org.apache.solr.handler.admin.CollectionsHandler.handleSplitShardAction(CollectionsHandler.java:322)
at
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:136)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:218)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
</str><int name="code">500</int></lst>
</response>
# Full solr log for the split shard call:
384779 [qtp334936591-10] INFO
org.apache.solr.handler.admin.CollectionsHandler ? Splitting shard :
shard=shard1&action=SPLITSHARD&collection=body
384791 [main-EventThread] INFO org.apache.solr.cloud.DistributedQueue ?
Watcher fired on path: /overseer/collection-queue-work state:
SyncConnected type NodeChildrenChanged
384791 [main-EventThread] INFO org.apache.solr.cloud.DistributedQueue ?
Watcher fired on path: /overseer/collection-queue-work state:
SyncConnected type NodeChildrenChanged
384797 [main-EventThread] INFO org.apache.solr.cloud.DistributedQueue ?
Watcher fired on path: /overseer/collection-queue-work state:
SyncConnected type NodeChildrenChanged
384798 [Overseer-90358802380161262-10.100.131.76:8983_solr-n_0000000012]
INFO org.apache.solr.cloud.OverseerCollectionProcessor ? Overseer
Collection Processor: Get the message
id:/overseer/collection-queue-work/qn-0000000040 message:{
"operation":"splitshard",
"shard":"shard1",
"collection":"body"}
384799 [Overseer-90358802380161262-10.100.131.76:8983_solr-n_0000000012]
INFO org.apache.solr.cloud.OverseerCollectionProcessor ? Split shard
invoked
384800 [Overseer-90358802380161262-10.100.131.76:8983_solr-n_0000000012]
INFO org.apache.solr.cloud.OverseerCollectionProcessor ? Creating
shard body_shard1_0_replica1 as part of slice shard1_0 of collection
body on 10.100.131.76:8983_solr
384801 [Overseer-90358802380161262-10.100.131.76:8983_solr-n_0000000012]
INFO org.apache.solr.cloud.OverseerCollectionProcessor ? Creating
shard body_shard1_1_replica1 as part of slice shard1_1 of collection
body on 10.100.131.76:8983_solr
384916 [qtp334936591-13] INFO org.apache.solr.cloud.ZkController ?
publishing core=body_shard1_1_replica1 state=down
384916 [qtp334936591-12] INFO org.apache.solr.cloud.ZkController ?
publishing core=body_shard1_0_replica1 state=down
384917 [qtp334936591-12] INFO org.apache.solr.cloud.ZkController ?
numShards not found on descriptor - reading it from system property
384917 [qtp334936591-13] INFO org.apache.solr.cloud.ZkController ?
numShards not found on descriptor - reading it from system property
384926 [qtp334936591-12] INFO org.apache.solr.cloud.ZkController ?
waiting to find shard id in clusterstate for body_shard1_0_replica1
384930 [qtp334936591-13] INFO org.apache.solr.cloud.ZkController ?
waiting to find shard id in clusterstate for body_shard1_1_replica1
386369 [Thread-13] INFO org.apache.solr.common.cloud.ZkStateReader ?
Updating cloud state from ZooKeeper...
386372 [Thread-13] INFO org.apache.solr.cloud.Overseer ? Update state
numShards=null message={
"operation":"state",
"core_node_name":"10.100.131.76:8983_solr_body_shard1_0_replica1",
"numShards":null,
"shard":"shard1_0",
"roles":null,
"state":"down",
"shard_state":"construction",
"core":"body_shard1_0_replica1",
"collection":"body",
"shard_range":"80000000-ffffffff",
"node_name":"10.100.131.76:8983_solr",
"base_url":"http://10.100.131.76:8983/solr"}
386392 [Thread-13] INFO org.apache.solr.cloud.Overseer ? Update state
numShards=null message={
"operation":"state",
"core_node_name":"10.100.131.76:8983_solr_body_shard1_1_replica1",
"numShards":null,
"shard":"shard1_1",
"roles":null,
"state":"down",
"shard_state":"construction",
"core":"body_shard1_1_replica1",
"collection":"body",
"shard_range":"0-7fffffff",
"node_name":"10.100.131.76:8983_solr",
"base_url":"http://10.100.131.76:8983/solr"}
386481 [main-EventThread] INFO
org.apache.solr.common.cloud.ZkStateReader ? A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
386930 [qtp334936591-12] INFO org.apache.solr.core.CoreContainer ?
Creating SolrCore 'body_shard1_0_replica1' using instanceDir:
mind/body_shard1_0_replica1
386931 [qtp334936591-12] INFO org.apache.solr.cloud.ZkController ?
Check for collection zkNode:body
386932 [qtp334936591-12] INFO org.apache.solr.cloud.ZkController ?
Collection zkNode exists
386932 [qtp334936591-12] INFO org.apache.solr.cloud.ZkController ? Load
collection config from:/collections/body
386934 [qtp334936591-13] INFO org.apache.solr.core.CoreContainer ?
Creating SolrCore 'body_shard1_1_replica1' using instanceDir:
mind/body_shard1_1_replica1
386934 [qtp334936591-13] INFO org.apache.solr.cloud.ZkController ?
Check for collection zkNode:body
386935 [qtp334936591-12] INFO org.apache.solr.core.SolrResourceLoader ?
new SolrResourceLoader for directory: 'mind/body_shard1_0_replica1/'
386936 [qtp334936591-13] INFO org.apache.solr.cloud.ZkController ?
Collection zkNode exists
386936 [qtp334936591-13] INFO org.apache.solr.cloud.ZkController ? Load
collection config from:/collections/body
386939 [qtp334936591-13] INFO org.apache.solr.core.SolrResourceLoader ?
new SolrResourceLoader for directory: 'mind/body_shard1_1_replica1/'
386971 [qtp334936591-12] INFO org.apache.solr.core.SolrConfig ? Adding
specified lib dirs to ClassLoader
386971 [qtp334936591-12] WARN org.apache.solr.core.SolrResourceLoader ?
Can't find (or read) directory to add to classloader: ../lib/ (resolved
as: /TivoData/solr-4.4.0/example/mind/body_shard1_0_replica1/../lib).
386974 [qtp334936591-13] INFO org.apache.solr.core.SolrConfig ? Adding
specified lib dirs to ClassLoader
386974 [qtp334936591-13] WARN org.apache.solr.core.SolrResourceLoader ?
Can't find (or read) directory to add to classloader: ../lib/ (resolved
as: /TivoData/solr-4.4.0/example/mind/body_shard1_1_replica1/../lib).
387016 [qtp334936591-12] INFO org.apache.solr.core.SolrConfig ? Using
Lucene MatchVersion: LUCENE_44
387023 [qtp334936591-13] INFO org.apache.solr.core.SolrConfig ? Using
Lucene MatchVersion: LUCENE_44
387072 [qtp334936591-12] INFO org.apache.solr.core.Config ? Loaded
SolrConfig: solrconfig.xml
387075 [qtp334936591-12] INFO org.apache.solr.schema.IndexSchema ?
Reading Solr Schema from schema.xml
387078 [qtp334936591-13] INFO org.apache.solr.core.Config ? Loaded
SolrConfig: solrconfig.xml
387080 [qtp334936591-13] INFO org.apache.solr.schema.IndexSchema ?
Reading Solr Schema from schema.xml
387095 [qtp334936591-12] INFO org.apache.solr.schema.IndexSchema ?
[body_shard1_0_replica1] Schema name=body
387099 [qtp334936591-13] INFO org.apache.solr.schema.IndexSchema ?
[body_shard1_1_replica1] Schema name=body
387195 [qtp334936591-12] INFO org.apache.solr.schema.IndexSchema ?
default search field in schema is bodyId
387196 [qtp334936591-13] INFO org.apache.solr.schema.IndexSchema ?
default search field in schema is bodyId
387197 [qtp334936591-12] INFO org.apache.solr.schema.IndexSchema ?
query parser default operator is OR
387197 [qtp334936591-13] INFO org.apache.solr.schema.IndexSchema ?
query parser default operator is OR
387198 [qtp334936591-13] INFO org.apache.solr.schema.IndexSchema ?
unique key field: bodyId
387198 [qtp334936591-12] INFO org.apache.solr.schema.IndexSchema ?
unique key field: bodyId
387218 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ?
solr.NRTCachingDirectoryFactory
387219 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ?
[body_shard1_1_replica1] Opening new SolrCore at
mind/body_shard1_1_replica1/,
dataDir=/TivoData/solr-4.4.0/example/mind/body_shard1_1_replica1/data/
387219 [qtp334936591-13] INFO org.apache.solr.core.JmxMonitoredMap ? No
JMX servers found, not exposing Solr information with JMX.
387219 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ?
[body_shard1_1_replica1] Added SolrEventListener for newSearcher:
org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10},
{q=rocks,start=0,rows=10}, {q=static newSearcher warming query from
solrconfig.xml}]}
387219 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ?
[body_shard1_1_replica1] Added SolrEventListener for firstSearcher:
org.apache.solr.core.QuerySenderListener{queries=[]}
387220 [qtp334936591-13] INFO
org.apache.solr.core.CachingDirectoryFactory ? return new directory for
/TivoData/solr-4.4.0/example/mind/body_shard1_1_replica1/data
387220 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ? New index
directory detected: old=null
new=/TivoData/solr-4.4.0/example/mind/body_shard1_1_replica1/data/index/
387220 [qtp334936591-13] WARN org.apache.solr.core.SolrCore ?
[body_shard1_1_replica1] Solr index directory
'/TivoData/solr-4.4.0/example/mind/body_shard1_1_replica1/data/index'
doesn't exist. Creating new index...
387221 [qtp334936591-13] INFO
org.apache.solr.core.CachingDirectoryFactory ? return new directory for
/TivoData/solr-4.4.0/example/mind/body_shard1_1_replica1/data/index
387225 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ?
solr.NRTCachingDirectoryFactory
387226 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ?
[body_shard1_0_replica1] Opening new SolrCore at
mind/body_shard1_0_replica1/,
dataDir=/TivoData/solr-4.4.0/example/mind/body_shard1_0_replica1/data/
387227 [qtp334936591-12] INFO org.apache.solr.core.JmxMonitoredMap ? No
JMX servers found, not exposing Solr information with JMX.
387227 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ?
[body_shard1_0_replica1] Added SolrEventListener for newSearcher:
org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10},
{q=rocks,start=0,rows=10}, {q=static newSearcher warming query from
solrconfig.xml}]}
387228 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ?
[body_shard1_0_replica1] Added SolrEventListener for firstSearcher:
org.apache.solr.core.QuerySenderListener{queries=[]}
387229 [qtp334936591-12] INFO
org.apache.solr.core.CachingDirectoryFactory ? return new directory for
/TivoData/solr-4.4.0/example/mind/body_shard1_0_replica1/data
387229 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ? New index
directory detected: old=null
new=/TivoData/solr-4.4.0/example/mind/body_shard1_0_replica1/data/index/
387230 [qtp334936591-12] WARN org.apache.solr.core.SolrCore ?
[body_shard1_0_replica1] Solr index directory
'/TivoData/solr-4.4.0/example/mind/body_shard1_0_replica1/data/index'
doesn't exist. Creating new index...
387230 [qtp334936591-12] INFO
org.apache.solr.core.CachingDirectoryFactory ? return new directory for
/TivoData/solr-4.4.0/example/mind/body_shard1_0_replica1/data/index
387346 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ?
SolrDeletionPolicy.onCommit: commits: num=1
commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/TivoData/solr-4.4.0/example/mind/body_shard1_1_replica1/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@3bea817f;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
387346 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ?
SolrDeletionPolicy.onCommit: commits: num=1
commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/TivoData/solr-4.4.0/example/mind/body_shard1_0_replica1/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@20dbd794;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
387346 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ? newest
commit generation = 1
387346 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ? newest
commit generation = 1
387348 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ? created
xslt: solr.XSLTResponseWriter
387348 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ? created
xslt: solr.XSLTResponseWriter
387348 [qtp334936591-12] INFO
org.apache.solr.response.XSLTResponseWriter ? xsltCacheLifetimeSeconds=5
387348 [qtp334936591-13] INFO
org.apache.solr.response.XSLTResponseWriter ? xsltCacheLifetimeSeconds=5
387350 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
created standard: solr.StandardRequestHandler
387350 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
adding lazy requestHandler: solr.SpellCheckerRequestHandler
387350 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
created standard: solr.StandardRequestHandler
387350 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
created spellchecker: solr.SpellCheckerRequestHandler
387351 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
adding lazy requestHandler: solr.SpellCheckerRequestHandler
387351 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
created /mlt: solr.MoreLikeThisHandler
387351 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
created spellchecker: solr.SpellCheckerRequestHandler
387352 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
387352 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
created /mlt: solr.MoreLikeThisHandler
387352 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
created /search: org.apache.solr.handler.component.SearchHandler
387353 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
adding lazy requestHandler: org.apache.solr.handler.component.SearchHandler
387353 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
387353 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
created /elevate: org.apache.solr.handler.component.SearchHandler
387354 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
created /search: org.apache.solr.handler.component.SearchHandler
387354 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
adding lazy requestHandler: org.apache.solr.handler.component.SearchHandler
387355 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
created /elevate: org.apache.solr.handler.component.SearchHandler
387354 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
created /update: solr.UpdateRequestHandler
387355 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
created /update: solr.UpdateRequestHandler
387355 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
created /admin/: org.apache.solr.handler.admin.AdminHandlers
387356 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
created /admin/: org.apache.solr.handler.admin.AdminHandlers
387356 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
created /admin/ping: solr.PingRequestHandler
387356 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
created /admin/ping: solr.PingRequestHandler
387357 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
created /debug/dump: solr.DumpRequestHandler
387357 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
created /debug/dump: solr.DumpRequestHandler
387358 [qtp334936591-12] INFO org.apache.solr.handler.loader.XMLLoader
? xsltCacheLifetimeSeconds=60
387358 [qtp334936591-13] INFO org.apache.solr.handler.loader.XMLLoader
? xsltCacheLifetimeSeconds=60
387362 [qtp334936591-12] INFO org.apache.solr.search.SolrIndexSearcher
? Opening Searcher@6745934e main
387363 [qtp334936591-13] INFO org.apache.solr.search.SolrIndexSearcher
? Opening Searcher@23bca486 main
387363 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ? Hard
AutoCommit: disabled
387363 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ? Hard
AutoCommit: disabled
387363 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ? Soft
AutoCommit: disabled
387364 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ? Soft
AutoCommit: disabled
387364 [qtp334936591-12] INFO org.apache.solr.cloud.ZkController ? Load
collection config from:/collections/body
387364 [qtp334936591-13] INFO org.apache.solr.cloud.ZkController ? Load
collection config from:/collections/body
387370 [searcherExecutor-36-thread-1] INFO
org.apache.solr.core.SolrCore ? QuerySenderListener sending requests to
Searcher@6745934e main{StandardDirectoryReader(segments_1:1)}
387371 [searcherExecutor-36-thread-1] INFO
org.apache.solr.core.SolrCore ? QuerySenderListener done.
387371 [searcherExecutor-35-thread-1] INFO
org.apache.solr.core.SolrCore ? QuerySenderListener sending requests to
Searcher@23bca486 main{StandardDirectoryReader(segments_1:1)}
387372 [qtp334936591-13] ERROR org.apache.solr.core.CoreContainer ?
Unable to create core: body_shard1_1_replica1
java.lang.NullPointerException
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:862)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:629)
at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:270)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:655)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:523)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:145)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:209)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
387372 [searcherExecutor-35-thread-1] INFO
org.apache.solr.core.SolrCore ? QuerySenderListener done.
387372 [qtp334936591-12] ERROR org.apache.solr.core.CoreContainer ?
Unable to create core: body_shard1_0_replica1
java.lang.NullPointerException
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:862)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:629)
at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:270)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:655)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:523)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:145)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:209)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
387371 [searcherExecutor-36-thread-1] INFO
org.apache.solr.core.SolrCore ? [body_shard1_0_replica1] Registered new
searcher Searcher@6745934e main{StandardDirectoryReader(segments_1:1)}
387375 [searcherExecutor-35-thread-1] INFO
org.apache.solr.core.SolrCore ? [body_shard1_1_replica1] Registered new
searcher Searcher@23bca486 main{StandardDirectoryReader(segments_1:1)}
387389 [qtp334936591-13] ERROR org.apache.solr.core.SolrCore ?
org.apache.solr.common.SolrException: Error CREATEing SolrCore
'body_shard1_1_replica1': Unable to create core: body_shard1_1_replica1
Caused by: null
at
org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:554)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:145)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:209)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.solr.common.SolrException: Unable to create core:
body_shard1_1_replica1
at
org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:1150)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:666)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:523)
... 30 more
Caused by: java.lang.NullPointerException
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:862)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:629)
at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:270)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:655)
... 31 more
387389 [qtp334936591-13] INFO
org.apache.solr.servlet.SolrDispatchFilter ? [admin] webapp=null
path=/admin/cores
params={shard.range=0-7fffffff&shard.state=construction&shard=shard1_1&name=body_shard1_1_replica1&action=CREATE&collection=body&wt=javabin&qt=/admin/cores&version=2}
status=400 QTime=2474
387392 [qtp334936591-12] ERROR org.apache.solr.core.SolrCore ?
org.apache.solr.common.SolrException: Error CREATEing SolrCore
'body_shard1_0_replica1': Unable to create core: body_shard1_0_replica1
Caused by: null
at
org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:554)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:145)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:209)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.solr.common.SolrException: Unable to create core:
body_shard1_0_replica1
at
org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:1150)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:666)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:523)
... 30 more
Caused by: java.lang.NullPointerException
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:862)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:629)
at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:270)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:655)
... 31 more
387393 [qtp334936591-12] INFO
org.apache.solr.servlet.SolrDispatchFilter ? [admin] webapp=null
path=/admin/cores
params={shard.range=80000000-ffffffff&shard.state=construction&shard=shard1_0&name=body_shard1_0_replica1&action=CREATE&collection=body&wt=javabin&qt=/admin/cores&version=2}
status=400 QTime=2478
387405 [Overseer-90358802380161262-10.100.131.76:8983_solr-n_0000000012]
ERROR org.apache.solr.cloud.OverseerCollectionProcessor ? Error from
shard: 10.100.131.76:8983/solr
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
Error CREATEing SolrCore 'body_shard1_0_replica1': Unable to create
core: body_shard1_0_replica1 Caused by: null
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:424)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
387406 [Overseer-90358802380161262-10.100.131.76:8983_solr-n_0000000012]
ERROR org.apache.solr.cloud.OverseerCollectionProcessor ? Collection
splitshard of splitshard failed:org.apache.solr.common.SolrException:
SPLTSHARD failed to create subshard leaders
at
org.apache.solr.cloud.OverseerCollectionProcessor.collectShardResponses(OverseerCollectionProcessor.java:647)
at
org.apache.solr.cloud.OverseerCollectionProcessor.splitShard(OverseerCollectionProcessor.java:422)
at
org.apache.solr.cloud.OverseerCollectionProcessor.processMessage(OverseerCollectionProcessor.java:187)
at
org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:127)
at java.lang.Thread.run(Thread.java:662)
Caused by:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
Error CREATEing SolrCore 'body_shard1_0_replica1': Unable to create
core: body_shard1_0_replica1 Caused by: null
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:424)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
... 1 more
387450 [main-EventThread] INFO org.apache.solr.cloud.DistributedQueue ?
Watcher fired on path: /overseer/collection-queue-work/qnr-0000000040
state: SyncConnected type NodeDataChanged
387465 [Overseer-90358802380161262-10.100.131.76:8983_solr-n_0000000012]
INFO org.apache.solr.cloud.OverseerCollectionProcessor ? Overseer
Collection Processor: Message
id:/overseer/collection-queue-work/qn-0000000040 complete,
response:{failure={null=org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:Error
CREATEing SolrCore 'body_shard1_0_replica1': Unable to create core:
body_shard1_0_replica1 Caused by: null},Operation splitshard caused
exception:=org.apache.solr.common.SolrException: SPLTSHARD failed to
create subshard leaders,exception={msg=SPLTSHARD failed to create
subshard leaders,rspCode=500}}
387466 [qtp334936591-10] INFO
org.apache.solr.servlet.SolrDispatchFilter ? [admin] webapp=null
path=/admin/collections
params={shard=shard1&action=SPLITSHARD&collection=body} status=500
QTime=2688
387467 [qtp334936591-10] ERROR
org.apache.solr.servlet.SolrDispatchFilter ?
null:org.apache.solr.common.SolrException: SPLTSHARD failed to create
subshard leaders
at
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:171)
at
org.apache.solr.handler.admin.CollectionsHandler.handleSplitShardAction(CollectionsHandler.java:322)
at
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:136)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:218)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
388009 [Thread-13] INFO org.apache.solr.common.cloud.ZkStateReader ?
Updating cloud state from ZooKeeper...
388066 [main-EventThread] INFO
org.apache.solr.common.cloud.ZkStateReader ? A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
738810 [Finalizer] ERROR org.apache.solr.core.SolrCore ? REFCOUNT
ERROR: unreferenced org.apache.solr.core.SolrCore@529801f4
(body_shard1_0_replica1) has a reference count of 1
738810 [Finalizer] ERROR org.apache.solr.util.ConcurrentLRUCache ?
ConcurrentLRUCache was not destroyed prior to finalize(), indicates a
bug -- POSSIBLE RESOURCE LEAK!!!
738835 [Finalizer] ERROR org.apache.solr.core.SolrCore ? REFCOUNT
ERROR: unreferenced org.apache.solr.core.SolrCore@25927275
(body_shard1_1_replica1) has a reference count of 1
738836 [Finalizer] ERROR org.apache.solr.util.ConcurrentLRUCache ?
ConcurrentLRUCache was not destroyed prior to finalize(), indicates a
bug -- POSSIBLE RESOURCE LEAK!!!
941518 [qtp334936591-14] ERROR org.apache.solr.core.SolrCore ?
org.apache.solr.common.SolrException: no servers hosting shard:
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:149)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
--
Regards,
HaiXin
=============================
AIM : tivohtie
Work : 408.914.9835
Mobile : 408.368.9289
Schedule : http://htie-linux/
=============================
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
Re: SPLITSHARD failure right before publishing the new sub-shards
Posted by HaiXin Tie <ht...@tivo.com>.
Never mind. I figured it out. It was due to a NPE on the missing
updateLog in solrconfig.xml. My solrconfig.xml is from an older Solr
release, which doesn't have certain required sections, etc. After adding
them to solrconfig.xml per this official doc, everything started to
work. It'd be great if null checks were there to produce informative
error on SolrCore.java, so as to make it easier to find the root cause.
http://wiki.apache.org/solr/SolrCloud#Required_Config
Regards,
HaiXin
On 09/16/2013 06:44 PM, HaiXin Tie wrote:
> Hi Solr experts,
>
> I am using Solr 4.4 with ZK 3.4.5, trying to split "shard1" of a
> collection named "body". There is only one core on one machine for
> this collection. When I call SPLITSHARD to split this collection, Solr
> is able to create two sub-shards, but failed with a NPE in
> SolrCore.java while publishing the new shards. It seems that either
> the updateHandler or its updateLog is null, though they work fine in
> the original shard:
>
> SolrCore.java
> if (cc != null && cc.isZooKeeperAware() &&
> Slice.CONSTRUCTION.equals(cd.getCloudDescriptor().getShardState())) {
> // set update log to buffer before publishing the core
> 862: getUpdateHandler().getUpdateLog().bufferUpdates();
>
> cd.getCloudDescriptor().setShardState(null);
> cd.getCloudDescriptor().setShardRange(null);
>
> }
>
>
> Here are the details. Any pointers to aid debugging this issue is
> greatly appreciated!
>
> # curl request/response to split the shard:
>
> curl -s
> "http://localhost:8983/solr/admin/collections?action=SPLITSHARD&collection=body&shard=shard1"
>
> <?xml version="1.0" encoding="UTF-8"?>
> <response>
> <lst name="responseHeader"><int name="status">500</int><int
> name="QTime">2688</int></lst><lst
> name="failure"><str>org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:Error
> CREATEing SolrCore 'body_shard1_0_replica1': Unable to create core:
> body_shard1_0_replica1 Caused by: null</str></lst><str name="Operation
> splitshard caused
> exception:">org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
> SPLTSHARD failed to create subshard leaders</str><lst
> name="exception"><str name="msg">SPLTSHARD failed to create subshard
> leaders</str><int name="rspCode">500</int></lst><lst name="error"><str
> name="msg">SPLTSHARD failed to create subshard leaders</str><str
> name="trace">org.apache.solr.common.SolrException: SPLTSHARD failed to
> create subshard leaders
> at
> org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:171)
> at
> org.apache.solr.handler.admin.CollectionsHandler.handleSplitShardAction(CollectionsHandler.java:322)
> at
> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:136)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at
> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:218)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:662)
> </str><int name="code">500</int></lst>
> </response>
>
>
> # Full solr log for the split shard call:
>
> 384779 [qtp334936591-10] INFO
> org.apache.solr.handler.admin.CollectionsHandler ? Splitting shard :
> shard=shard1&action=SPLITSHARD&collection=body
> 384791 [main-EventThread] INFO org.apache.solr.cloud.DistributedQueue
> ? Watcher fired on path: /overseer/collection-queue-work state:
> SyncConnected type NodeChildrenChanged
> 384791 [main-EventThread] INFO org.apache.solr.cloud.DistributedQueue
> ? Watcher fired on path: /overseer/collection-queue-work state:
> SyncConnected type NodeChildrenChanged
> 384797 [main-EventThread] INFO org.apache.solr.cloud.DistributedQueue
> ? Watcher fired on path: /overseer/collection-queue-work state:
> SyncConnected type NodeChildrenChanged
> 384798
> [Overseer-90358802380161262-10.100.131.76:8983_solr-n_0000000012]
> INFO org.apache.solr.cloud.OverseerCollectionProcessor ? Overseer
> Collection Processor: Get the message
> id:/overseer/collection-queue-work/qn-0000000040 message:{
> "operation":"splitshard",
> "shard":"shard1",
> "collection":"body"}
> 384799
> [Overseer-90358802380161262-10.100.131.76:8983_solr-n_0000000012]
> INFO org.apache.solr.cloud.OverseerCollectionProcessor ? Split shard
> invoked
> 384800
> [Overseer-90358802380161262-10.100.131.76:8983_solr-n_0000000012]
> INFO org.apache.solr.cloud.OverseerCollectionProcessor ? Creating
> shard body_shard1_0_replica1 as part of slice shard1_0 of collection
> body on 10.100.131.76:8983_solr
> 384801
> [Overseer-90358802380161262-10.100.131.76:8983_solr-n_0000000012]
> INFO org.apache.solr.cloud.OverseerCollectionProcessor ? Creating
> shard body_shard1_1_replica1 as part of slice shard1_1 of collection
> body on 10.100.131.76:8983_solr
> 384916 [qtp334936591-13] INFO org.apache.solr.cloud.ZkController ?
> publishing core=body_shard1_1_replica1 state=down
> 384916 [qtp334936591-12] INFO org.apache.solr.cloud.ZkController ?
> publishing core=body_shard1_0_replica1 state=down
> 384917 [qtp334936591-12] INFO org.apache.solr.cloud.ZkController ?
> numShards not found on descriptor - reading it from system property
> 384917 [qtp334936591-13] INFO org.apache.solr.cloud.ZkController ?
> numShards not found on descriptor - reading it from system property
> 384926 [qtp334936591-12] INFO org.apache.solr.cloud.ZkController ?
> waiting to find shard id in clusterstate for body_shard1_0_replica1
> 384930 [qtp334936591-13] INFO org.apache.solr.cloud.ZkController ?
> waiting to find shard id in clusterstate for body_shard1_1_replica1
> 386369 [Thread-13] INFO org.apache.solr.common.cloud.ZkStateReader ?
> Updating cloud state from ZooKeeper...
> 386372 [Thread-13] INFO org.apache.solr.cloud.Overseer ? Update
> state numShards=null message={
> "operation":"state",
> "core_node_name":"10.100.131.76:8983_solr_body_shard1_0_replica1",
> "numShards":null,
> "shard":"shard1_0",
> "roles":null,
> "state":"down",
> "shard_state":"construction",
> "core":"body_shard1_0_replica1",
> "collection":"body",
> "shard_range":"80000000-ffffffff",
> "node_name":"10.100.131.76:8983_solr",
> "base_url":"http://10.100.131.76:8983/solr"}
> 386392 [Thread-13] INFO org.apache.solr.cloud.Overseer ? Update
> state numShards=null message={
> "operation":"state",
> "core_node_name":"10.100.131.76:8983_solr_body_shard1_1_replica1",
> "numShards":null,
> "shard":"shard1_1",
> "roles":null,
> "state":"down",
> "shard_state":"construction",
> "core":"body_shard1_1_replica1",
> "collection":"body",
> "shard_range":"0-7fffffff",
> "node_name":"10.100.131.76:8983_solr",
> "base_url":"http://10.100.131.76:8983/solr"}
> 386481 [main-EventThread] INFO
> org.apache.solr.common.cloud.ZkStateReader ? A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> 386930 [qtp334936591-12] INFO org.apache.solr.core.CoreContainer ?
> Creating SolrCore 'body_shard1_0_replica1' using instanceDir:
> mind/body_shard1_0_replica1
> 386931 [qtp334936591-12] INFO org.apache.solr.cloud.ZkController ?
> Check for collection zkNode:body
> 386932 [qtp334936591-12] INFO org.apache.solr.cloud.ZkController ?
> Collection zkNode exists
> 386932 [qtp334936591-12] INFO org.apache.solr.cloud.ZkController ?
> Load collection config from:/collections/body
> 386934 [qtp334936591-13] INFO org.apache.solr.core.CoreContainer ?
> Creating SolrCore 'body_shard1_1_replica1' using instanceDir:
> mind/body_shard1_1_replica1
> 386934 [qtp334936591-13] INFO org.apache.solr.cloud.ZkController ?
> Check for collection zkNode:body
> 386935 [qtp334936591-12] INFO org.apache.solr.core.SolrResourceLoader
> ? new SolrResourceLoader for directory: 'mind/body_shard1_0_replica1/'
> 386936 [qtp334936591-13] INFO org.apache.solr.cloud.ZkController ?
> Collection zkNode exists
> 386936 [qtp334936591-13] INFO org.apache.solr.cloud.ZkController ?
> Load collection config from:/collections/body
> 386939 [qtp334936591-13] INFO org.apache.solr.core.SolrResourceLoader
> ? new SolrResourceLoader for directory: 'mind/body_shard1_1_replica1/'
> 386971 [qtp334936591-12] INFO org.apache.solr.core.SolrConfig ?
> Adding specified lib dirs to ClassLoader
> 386971 [qtp334936591-12] WARN org.apache.solr.core.SolrResourceLoader
> ? Can't find (or read) directory to add to classloader: ../lib/
> (resolved as:
> /TivoData/solr-4.4.0/example/mind/body_shard1_0_replica1/../lib).
> 386974 [qtp334936591-13] INFO org.apache.solr.core.SolrConfig ?
> Adding specified lib dirs to ClassLoader
> 386974 [qtp334936591-13] WARN org.apache.solr.core.SolrResourceLoader
> ? Can't find (or read) directory to add to classloader: ../lib/
> (resolved as:
> /TivoData/solr-4.4.0/example/mind/body_shard1_1_replica1/../lib).
> 387016 [qtp334936591-12] INFO org.apache.solr.core.SolrConfig ? Using
> Lucene MatchVersion: LUCENE_44
> 387023 [qtp334936591-13] INFO org.apache.solr.core.SolrConfig ? Using
> Lucene MatchVersion: LUCENE_44
> 387072 [qtp334936591-12] INFO org.apache.solr.core.Config ? Loaded
> SolrConfig: solrconfig.xml
> 387075 [qtp334936591-12] INFO org.apache.solr.schema.IndexSchema ?
> Reading Solr Schema from schema.xml
> 387078 [qtp334936591-13] INFO org.apache.solr.core.Config ? Loaded
> SolrConfig: solrconfig.xml
> 387080 [qtp334936591-13] INFO org.apache.solr.schema.IndexSchema ?
> Reading Solr Schema from schema.xml
> 387095 [qtp334936591-12] INFO org.apache.solr.schema.IndexSchema ?
> [body_shard1_0_replica1] Schema name=body
> 387099 [qtp334936591-13] INFO org.apache.solr.schema.IndexSchema ?
> [body_shard1_1_replica1] Schema name=body
> 387195 [qtp334936591-12] INFO org.apache.solr.schema.IndexSchema ?
> default search field in schema is bodyId
> 387196 [qtp334936591-13] INFO org.apache.solr.schema.IndexSchema ?
> default search field in schema is bodyId
> 387197 [qtp334936591-12] INFO org.apache.solr.schema.IndexSchema ?
> query parser default operator is OR
> 387197 [qtp334936591-13] INFO org.apache.solr.schema.IndexSchema ?
> query parser default operator is OR
> 387198 [qtp334936591-13] INFO org.apache.solr.schema.IndexSchema ?
> unique key field: bodyId
> 387198 [qtp334936591-12] INFO org.apache.solr.schema.IndexSchema ?
> unique key field: bodyId
> 387218 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ?
> solr.NRTCachingDirectoryFactory
> 387219 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ?
> [body_shard1_1_replica1] Opening new SolrCore at
> mind/body_shard1_1_replica1/,
> dataDir=/TivoData/solr-4.4.0/example/mind/body_shard1_1_replica1/data/
> 387219 [qtp334936591-13] INFO org.apache.solr.core.JmxMonitoredMap ?
> No JMX servers found, not exposing Solr information with JMX.
> 387219 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ?
> [body_shard1_1_replica1] Added SolrEventListener for newSearcher:
> org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10},
> {q=rocks,start=0,rows=10}, {q=static newSearcher warming query from
> solrconfig.xml}]}
> 387219 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ?
> [body_shard1_1_replica1] Added SolrEventListener for firstSearcher:
> org.apache.solr.core.QuerySenderListener{queries=[]}
> 387220 [qtp334936591-13] INFO
> org.apache.solr.core.CachingDirectoryFactory ? return new directory
> for /TivoData/solr-4.4.0/example/mind/body_shard1_1_replica1/data
> 387220 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ? New
> index directory detected: old=null
> new=/TivoData/solr-4.4.0/example/mind/body_shard1_1_replica1/data/index/
> 387220 [qtp334936591-13] WARN org.apache.solr.core.SolrCore ?
> [body_shard1_1_replica1] Solr index directory
> '/TivoData/solr-4.4.0/example/mind/body_shard1_1_replica1/data/index'
> doesn't exist. Creating new index...
> 387221 [qtp334936591-13] INFO
> org.apache.solr.core.CachingDirectoryFactory ? return new directory
> for /TivoData/solr-4.4.0/example/mind/body_shard1_1_replica1/data/index
> 387225 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ?
> solr.NRTCachingDirectoryFactory
> 387226 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ?
> [body_shard1_0_replica1] Opening new SolrCore at
> mind/body_shard1_0_replica1/,
> dataDir=/TivoData/solr-4.4.0/example/mind/body_shard1_0_replica1/data/
> 387227 [qtp334936591-12] INFO org.apache.solr.core.JmxMonitoredMap ?
> No JMX servers found, not exposing Solr information with JMX.
> 387227 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ?
> [body_shard1_0_replica1] Added SolrEventListener for newSearcher:
> org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10},
> {q=rocks,start=0,rows=10}, {q=static newSearcher warming query from
> solrconfig.xml}]}
> 387228 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ?
> [body_shard1_0_replica1] Added SolrEventListener for firstSearcher:
> org.apache.solr.core.QuerySenderListener{queries=[]}
> 387229 [qtp334936591-12] INFO
> org.apache.solr.core.CachingDirectoryFactory ? return new directory
> for /TivoData/solr-4.4.0/example/mind/body_shard1_0_replica1/data
> 387229 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ? New
> index directory detected: old=null
> new=/TivoData/solr-4.4.0/example/mind/body_shard1_0_replica1/data/index/
> 387230 [qtp334936591-12] WARN org.apache.solr.core.SolrCore ?
> [body_shard1_0_replica1] Solr index directory
> '/TivoData/solr-4.4.0/example/mind/body_shard1_0_replica1/data/index'
> doesn't exist. Creating new index...
> 387230 [qtp334936591-12] INFO
> org.apache.solr.core.CachingDirectoryFactory ? return new directory
> for /TivoData/solr-4.4.0/example/mind/body_shard1_0_replica1/data/index
> 387346 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ?
> SolrDeletionPolicy.onCommit: commits: num=1
> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/TivoData/solr-4.4.0/example/mind/body_shard1_1_replica1/data/index
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@3bea817f;
> maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
> 387346 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ?
> SolrDeletionPolicy.onCommit: commits: num=1
> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/TivoData/solr-4.4.0/example/mind/body_shard1_0_replica1/data/index
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@20dbd794;
> maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
> 387346 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ? newest
> commit generation = 1
> 387346 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ? newest
> commit generation = 1
> 387348 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ? created
> xslt: solr.XSLTResponseWriter
> 387348 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ? created
> xslt: solr.XSLTResponseWriter
> 387348 [qtp334936591-12] INFO
> org.apache.solr.response.XSLTResponseWriter ? xsltCacheLifetimeSeconds=5
> 387348 [qtp334936591-13] INFO
> org.apache.solr.response.XSLTResponseWriter ? xsltCacheLifetimeSeconds=5
> 387350 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
> created standard: solr.StandardRequestHandler
> 387350 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
> adding lazy requestHandler: solr.SpellCheckerRequestHandler
> 387350 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
> created standard: solr.StandardRequestHandler
> 387350 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
> created spellchecker: solr.SpellCheckerRequestHandler
> 387351 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
> adding lazy requestHandler: solr.SpellCheckerRequestHandler
> 387351 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
> created /mlt: solr.MoreLikeThisHandler
> 387351 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
> created spellchecker: solr.SpellCheckerRequestHandler
> 387352 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
> created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
> 387352 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
> created /mlt: solr.MoreLikeThisHandler
> 387352 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
> created /search: org.apache.solr.handler.component.SearchHandler
> 387353 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
> adding lazy requestHandler:
> org.apache.solr.handler.component.SearchHandler
> 387353 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
> created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
> 387353 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
> created /elevate: org.apache.solr.handler.component.SearchHandler
> 387354 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
> created /search: org.apache.solr.handler.component.SearchHandler
> 387354 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
> adding lazy requestHandler:
> org.apache.solr.handler.component.SearchHandler
> 387355 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
> created /elevate: org.apache.solr.handler.component.SearchHandler
> 387354 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
> created /update: solr.UpdateRequestHandler
> 387355 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
> created /update: solr.UpdateRequestHandler
> 387355 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
> 387356 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
> 387356 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
> created /admin/ping: solr.PingRequestHandler
> 387356 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
> created /admin/ping: solr.PingRequestHandler
> 387357 [qtp334936591-12] INFO org.apache.solr.core.RequestHandlers ?
> created /debug/dump: solr.DumpRequestHandler
> 387357 [qtp334936591-13] INFO org.apache.solr.core.RequestHandlers ?
> created /debug/dump: solr.DumpRequestHandler
> 387358 [qtp334936591-12] INFO
> org.apache.solr.handler.loader.XMLLoader ? xsltCacheLifetimeSeconds=60
> 387358 [qtp334936591-13] INFO
> org.apache.solr.handler.loader.XMLLoader ? xsltCacheLifetimeSeconds=60
> 387362 [qtp334936591-12] INFO
> org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@6745934e
> main
> 387363 [qtp334936591-13] INFO
> org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@23bca486
> main
> 387363 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ? Hard
> AutoCommit: disabled
> 387363 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ? Hard
> AutoCommit: disabled
> 387363 [qtp334936591-12] INFO org.apache.solr.core.SolrCore ? Soft
> AutoCommit: disabled
> 387364 [qtp334936591-13] INFO org.apache.solr.core.SolrCore ? Soft
> AutoCommit: disabled
> 387364 [qtp334936591-12] INFO org.apache.solr.cloud.ZkController ?
> Load collection config from:/collections/body
> 387364 [qtp334936591-13] INFO org.apache.solr.cloud.ZkController ?
> Load collection config from:/collections/body
> 387370 [searcherExecutor-36-thread-1] INFO
> org.apache.solr.core.SolrCore ? QuerySenderListener sending requests
> to Searcher@6745934e main{StandardDirectoryReader(segments_1:1)}
> 387371 [searcherExecutor-36-thread-1] INFO
> org.apache.solr.core.SolrCore ? QuerySenderListener done.
> 387371 [searcherExecutor-35-thread-1] INFO
> org.apache.solr.core.SolrCore ? QuerySenderListener sending requests
> to Searcher@23bca486 main{StandardDirectoryReader(segments_1:1)}
> 387372 [qtp334936591-13] ERROR org.apache.solr.core.CoreContainer ?
> Unable to create core: body_shard1_1_replica1
> java.lang.NullPointerException
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:862)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:629)
> at
> org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:270)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:655)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:523)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:145)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at
> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:209)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:662)
> 387372 [searcherExecutor-35-thread-1] INFO
> org.apache.solr.core.SolrCore ? QuerySenderListener done.
> 387372 [qtp334936591-12] ERROR org.apache.solr.core.CoreContainer ?
> Unable to create core: body_shard1_0_replica1
> java.lang.NullPointerException
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:862)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:629)
> at
> org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:270)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:655)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:523)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:145)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at
> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:209)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:662)
> 387371 [searcherExecutor-36-thread-1] INFO
> org.apache.solr.core.SolrCore ? [body_shard1_0_replica1] Registered
> new searcher Searcher@6745934e
> main{StandardDirectoryReader(segments_1:1)}
> 387375 [searcherExecutor-35-thread-1] INFO
> org.apache.solr.core.SolrCore ? [body_shard1_1_replica1] Registered
> new searcher Searcher@23bca486
> main{StandardDirectoryReader(segments_1:1)}
> 387389 [qtp334936591-13] ERROR org.apache.solr.core.SolrCore ?
> org.apache.solr.common.SolrException: Error CREATEing SolrCore
> 'body_shard1_1_replica1': Unable to create core:
> body_shard1_1_replica1 Caused by: null
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:554)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:145)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at
> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:209)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.solr.common.SolrException: Unable to create
> core: body_shard1_1_replica1
> at
> org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:1150)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:666)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:523)
> ... 30 more
> Caused by: java.lang.NullPointerException
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:862)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:629)
> at
> org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:270)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:655)
> ... 31 more
>
> 387389 [qtp334936591-13] INFO
> org.apache.solr.servlet.SolrDispatchFilter ? [admin] webapp=null
> path=/admin/cores
> params={shard.range=0-7fffffff&shard.state=construction&shard=shard1_1&name=body_shard1_1_replica1&action=CREATE&collection=body&wt=javabin&qt=/admin/cores&version=2}
> status=400 QTime=2474
> 387392 [qtp334936591-12] ERROR org.apache.solr.core.SolrCore ?
> org.apache.solr.common.SolrException: Error CREATEing SolrCore
> 'body_shard1_0_replica1': Unable to create core:
> body_shard1_0_replica1 Caused by: null
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:554)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:145)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at
> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:209)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.solr.common.SolrException: Unable to create
> core: body_shard1_0_replica1
> at
> org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:1150)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:666)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:523)
> ... 30 more
> Caused by: java.lang.NullPointerException
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:862)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:629)
> at
> org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:270)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:655)
> ... 31 more
>
> 387393 [qtp334936591-12] INFO
> org.apache.solr.servlet.SolrDispatchFilter ? [admin] webapp=null
> path=/admin/cores
> params={shard.range=80000000-ffffffff&shard.state=construction&shard=shard1_0&name=body_shard1_0_replica1&action=CREATE&collection=body&wt=javabin&qt=/admin/cores&version=2}
> status=400 QTime=2478
> 387405
> [Overseer-90358802380161262-10.100.131.76:8983_solr-n_0000000012]
> ERROR org.apache.solr.cloud.OverseerCollectionProcessor ? Error from
> shard: 10.100.131.76:8983/solr
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> Error CREATEing SolrCore 'body_shard1_0_replica1': Unable to create
> core: body_shard1_0_replica1 Caused by: null
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:424)
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
> at
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156)
> at
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 387406
> [Overseer-90358802380161262-10.100.131.76:8983_solr-n_0000000012]
> ERROR org.apache.solr.cloud.OverseerCollectionProcessor ? Collection
> splitshard of splitshard failed:org.apache.solr.common.SolrException:
> SPLTSHARD failed to create subshard leaders
> at
> org.apache.solr.cloud.OverseerCollectionProcessor.collectShardResponses(OverseerCollectionProcessor.java:647)
> at
> org.apache.solr.cloud.OverseerCollectionProcessor.splitShard(OverseerCollectionProcessor.java:422)
> at
> org.apache.solr.cloud.OverseerCollectionProcessor.processMessage(OverseerCollectionProcessor.java:187)
> at
> org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:127)
> at java.lang.Thread.run(Thread.java:662)
> Caused by:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> Error CREATEing SolrCore 'body_shard1_0_replica1': Unable to create
> core: body_shard1_0_replica1 Caused by: null
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:424)
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
> at
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156)
> at
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> ... 1 more
>
> 387450 [main-EventThread] INFO org.apache.solr.cloud.DistributedQueue
> ? Watcher fired on path:
> /overseer/collection-queue-work/qnr-0000000040 state: SyncConnected
> type NodeDataChanged
> 387465
> [Overseer-90358802380161262-10.100.131.76:8983_solr-n_0000000012]
> INFO org.apache.solr.cloud.OverseerCollectionProcessor ? Overseer
> Collection Processor: Message
> id:/overseer/collection-queue-work/qn-0000000040 complete,
> response:{failure={null=org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:Error
> CREATEing SolrCore 'body_shard1_0_replica1': Unable to create core:
> body_shard1_0_replica1 Caused by: null},Operation splitshard caused
> exception:=org.apache.solr.common.SolrException: SPLTSHARD failed to
> create subshard leaders,exception={msg=SPLTSHARD failed to create
> subshard leaders,rspCode=500}}
> 387466 [qtp334936591-10] INFO
> org.apache.solr.servlet.SolrDispatchFilter ? [admin] webapp=null
> path=/admin/collections
> params={shard=shard1&action=SPLITSHARD&collection=body} status=500
> QTime=2688
> 387467 [qtp334936591-10] ERROR
> org.apache.solr.servlet.SolrDispatchFilter ?
> null:org.apache.solr.common.SolrException: SPLTSHARD failed to create
> subshard leaders
> at
> org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:171)
> at
> org.apache.solr.handler.admin.CollectionsHandler.handleSplitShardAction(CollectionsHandler.java:322)
> at
> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:136)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at
> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:218)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:662)
>
> 388009 [Thread-13] INFO org.apache.solr.common.cloud.ZkStateReader ?
> Updating cloud state from ZooKeeper...
> 388066 [main-EventThread] INFO
> org.apache.solr.common.cloud.ZkStateReader ? A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> 738810 [Finalizer] ERROR org.apache.solr.core.SolrCore ? REFCOUNT
> ERROR: unreferenced org.apache.solr.core.SolrCore@529801f4
> (body_shard1_0_replica1) has a reference count of 1
> 738810 [Finalizer] ERROR org.apache.solr.util.ConcurrentLRUCache ?
> ConcurrentLRUCache was not destroyed prior to finalize(), indicates a
> bug -- POSSIBLE RESOURCE LEAK!!!
> 738835 [Finalizer] ERROR org.apache.solr.core.SolrCore ? REFCOUNT
> ERROR: unreferenced org.apache.solr.core.SolrCore@25927275
> (body_shard1_1_replica1) has a reference count of 1
> 738836 [Finalizer] ERROR org.apache.solr.util.ConcurrentLRUCache ?
> ConcurrentLRUCache was not destroyed prior to finalize(), indicates a
> bug -- POSSIBLE RESOURCE LEAK!!!
> 941518 [qtp334936591-14] ERROR org.apache.solr.core.SolrCore ?
> org.apache.solr.common.SolrException: no servers hosting shard:
> at
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:149)
> at
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
>
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.