You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ray Cheng <rc...@rocketmail.com> on 2014/02/05 04:26:15 UTC
SolrCloud fails to create new collections
Hi,
Our SolrCloud with Solr 4.6.0 has two replicas and uses three external ZooKeeper servers. We have created 20 collections successfully in the last two weeks. Then, creating new collections started failing since yesterday. New collections are created with curl like this one:
curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=abcd.xyz&numShards=1&replicationFactor=2&collection.configName=oursolrconfig'
telnet to port 8983 from one to the other Solr host was okay.
The largest ZooKeeper node is about 17K bytes.
curl http://localhost:8983/solr/admin/ping printed OK on either SolrCloud host.
The load on these two SolrCloud host is very low. So, these two systems are not busy at all.
After Updated/inserted a document in one SolrCloud server, the document appeared on the other SolrCloud server correctly.
Thus far only creating new collections failed. (I haven't tried to delete collections.)
The create collection printed the following lines on the shell that I ran the above curl command. Solr on the other Linux didn't print anything during this time frame. If I run the same curl on the second SolrCloud host, it printed the same error message and the first SolrCloud host didn't print anything.
Any idea on the cause of this problem or the fix? Thanks in advance.
$ curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=abcd.xyz&numShards=1&replicationFactor=2&collection.configName=oursolrconfig'
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">500</int><int name="QTime">60016</int></lst><lst name="error"><str name="msg">createcollection the collection time out:60s</str><str name="trace">org.apache.solr.common.SolrException: createcollection the collection time out:60s
at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:204)
at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:185)
at org.apache.solr.handler.admin.CollectionsHandler.handleCreateAction(CollectionsHandler.java:319)
at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:134)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:744)
</str><int name="code">500</int></lst>
</response>
Ray
Re: SolrCloud fails to create new collections
Posted by Ray Cheng <rc...@rocketmail.com>.
I restarted one of the Solr server of the SolrCloud and then domain creation worked. Not sure why this happened, whether anybody has seen it, and when this problem will happen again.
Thanks,
Ray
On Wednesday, February 5, 2014 11:19 AM, Ray Cheng <rc...@rocketmail.com> wrote:
Some more information that may help developers find out the cause.
>
>
>INFO 2014-02-04 14:47:08,931 DistributedQueue.java (line 211) Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
>...
>(there were 393 of these "Watcher fired on path" lines in total)
>...
>
>
>Thanks,
>Ray
>
>
>INFO 2014-02-04 14:47:08,923 CollectionsHandler.java (line 295) Creating Collection : numShards=1&name=abc.edu.xyz&replicationFactor=2&action=CREATE
> INFO 2014-02-04 14:47:08,931 DistributedQueue.java (line 211) Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
>INFO 2014-02-04 14:47:08,931 DistributedQueue.java (line 211) Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
>...
>(there were 393 of these "Watcher fired on path" lines in total)
>...
> INFO 2014-02-04 14:47:08,955 DistributedQueue.java (line 211) Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
> INFO 2014-02-04 14:48:08,944 DistributedQueue.java (line 211) Watcher fired on path: /overseer/collection-queue-work/qnr-0000000134 state: SyncConnected type NodeDeleted
>ERROR 2014-02-04 14:48:08,945 SolrException.java (line 108) org.apache.solr.common.SolrException: createcollection the collection time out:60s
> at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:204)
> at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:185)
> at org.apache.solr.handler.admin.CollectionsHandler.handleCreateAction(CollectionsHandler.java:319)
>
>
>
>On Tuesday, February 4, 2014 7:26 PM, Ray Cheng <rc...@rocketmail.com> wrote:
>
>Hi,
>>
>>Our SolrCloud with Solr 4.6.0 has two replicas and uses three external ZooKeeper servers. We have created 20 collections successfully in the last two weeks. Then, creating new collections started failing since yesterday. New collections are created with curl like this one:
>>
>>curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=abcd.xyz&numShards=1&replicationFactor=2&collection.configName=oursolrconfig'
>>
>>telnet to port 8983 from one to the other Solr host was okay.
>>The largest ZooKeeper node is about 17K bytes.
>>curl http://localhost:8983/solr/admin/ping printed OK on either SolrCloud host.
>>
>>The load on these two SolrCloud host is very low. So, these two systems are not busy at all.
>>After Updated/inserted a document in one SolrCloud server, the document appeared on the other SolrCloud server correctly.
>>Thus far only creating new collections failed. (I haven't tried to delete collections.)
>>
>>The create collection printed the following lines on the shell that I ran the above curl command. Solr on the other Linux didn't print anything during this time frame. If I run the same curl on the second SolrCloud host, it printed the same error message and the first SolrCloud host didn't print anything.
>>
>>Any idea on the cause of this problem or the fix? Thanks in advance.
>>
>>$ curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=abcd.xyz&numShards=1&replicationFactor=2&collection.configName=oursolrconfig'
>><?xml version="1.0" encoding="UTF-8"?>
>><response>
>><lst name="responseHeader"><int name="status">500</int><int name="QTime">60016</int></lst><lst name="error"><str name="msg">createcollection the collection time out:60s</str><str name="trace">org.apache.solr.common.SolrException: createcollection the collection time out:60s
>>at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:204)
>>at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:185)
>>at org.apache.solr.handler.admin.CollectionsHandler.handleCreateAction(CollectionsHandler.java:319)
>>at
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:134)
>>at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>>at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662)
>>at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
>>at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
>>at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
>>at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
>>at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
>>at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
>>at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
>>at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
>>at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
>>at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
>>at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
>>at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>>at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
>>at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
>>at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>>at org.eclipse.jetty.server.Server.handle(Server.java:368)
>>at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
>>at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
>>at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
>>at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
>>at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
>>at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>>at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
>>at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
>>at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>at java.lang.Thread.run(Thread.java:744)
>></str><int
name="code">500</int></lst>
>></response>
>>
>>
>>Ray
>>
>>
>
>
Re: SolrCloud fails to create new collections
Posted by Ray Cheng <rc...@rocketmail.com>.
Some more information that may help developers find out the cause.
INFO 2014-02-04 14:47:08,931 DistributedQueue.java (line 211) Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
...
(there were 393 of these "Watcher fired on path" lines in total)
...
Thanks,
Ray
INFO 2014-02-04 14:47:08,923 CollectionsHandler.java (line 295) Creating Collection : numShards=1&name=abc.edu.xyz&replicationFactor=2&action=CREATE
INFO 2014-02-04 14:47:08,931 DistributedQueue.java (line 211) Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
INFO 2014-02-04 14:47:08,931 DistributedQueue.java (line 211) Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
...
(there were 393 of these "Watcher fired on path" lines in total)
...
INFO 2014-02-04 14:47:08,955 DistributedQueue.java (line 211) Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
INFO 2014-02-04 14:48:08,944 DistributedQueue.java (line 211) Watcher fired on path: /overseer/collection-queue-work/qnr-0000000134 state: SyncConnected type NodeDeleted
ERROR 2014-02-04 14:48:08,945 SolrException.java (line 108) org.apache.solr.common.SolrException: createcollection the collection time out:60s
at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:204)
at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:185)
at org.apache.solr.handler.admin.CollectionsHandler.handleCreateAction(CollectionsHandler.java:319)
On Tuesday, February 4, 2014 7:26 PM, Ray Cheng <rc...@rocketmail.com> wrote:
Hi,
>
>Our SolrCloud with Solr 4.6.0 has two replicas and uses three external ZooKeeper servers. We have created 20 collections successfully in the last two weeks. Then, creating new collections started failing since yesterday. New collections are created with curl like this one:
>
>curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=abcd.xyz&numShards=1&replicationFactor=2&collection.configName=oursolrconfig'
>
>telnet to port 8983 from one to the other Solr host was okay.
>The largest ZooKeeper node is about 17K bytes.
>curl http://localhost:8983/solr/admin/ping printed OK on either SolrCloud host.
>
>The load on these two SolrCloud host is very low. So, these two systems are not busy at all.
>After Updated/inserted a document in one SolrCloud server, the document appeared on the other SolrCloud server correctly.
>Thus far only creating new collections failed. (I haven't tried to delete collections.)
>
>The create collection printed the following lines on the shell that I ran the above curl command. Solr on the other Linux didn't print anything during this time frame. If I run the same curl on the second SolrCloud host, it printed the same error message and the first SolrCloud host didn't print anything.
>
>Any idea on the cause of this problem or the fix? Thanks in advance.
>
>$ curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=abcd.xyz&numShards=1&replicationFactor=2&collection.configName=oursolrconfig'
><?xml version="1.0" encoding="UTF-8"?>
><response>
><lst name="responseHeader"><int name="status">500</int><int name="QTime">60016</int></lst><lst name="error"><str name="msg">createcollection the collection time out:60s</str><str name="trace">org.apache.solr.common.SolrException: createcollection the collection time out:60s
>at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:204)
>at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:185)
>at org.apache.solr.handler.admin.CollectionsHandler.handleCreateAction(CollectionsHandler.java:319)
>at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:134)
>at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662)
>at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
>at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
>at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
>at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
>at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
>at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
>at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
>at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
>at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
>at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
>at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
>at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
>at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
>at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>at org.eclipse.jetty.server.Server.handle(Server.java:368)
>at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
>at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
>at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
>at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
>at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
>at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
>at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
>at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>at java.lang.Thread.run(Thread.java:744)
></str><int name="code">500</int></lst>
></response>
>
>
>Ray
>
>