You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Susheel Kumar <su...@gmail.com> on 2016/05/04 19:48:34 UTC
Solr cloud 6.0.0 with ZooKeeper 3.4.8 Errors
Hello,
I am trying to setup 2 node Solr cloud 6 cluster with ZK 3.4.8 and used the
install service to setup solr.
After launching Solr Admin Panel on server1, it looses connections in few
seconds and then comes back and other node server2 is marked as Down in
cloud graph. After few seconds its loosing the connection and comes back.
Any idea what may be going wrong? Has anyone used Solr 6 with ZK 3.4.8.
Have never seen this error before with solr 5.x with ZK 3.4.6.
Below log from server1 & server2. The ZK has 3 nodes with chroot enabled.
Thanks,
Susheel
server1/solr.log
====
2016-05-04 19:20:53.804 INFO (qtp1989972246-14) [ ]
o.a.s.c.c.ZkStateReader path=[/collections/collection1]
[configName]=[collection1] specified config exists in ZooKeeper
2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 QTime=25
2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system
params={wt=json&_=1462389588126} status=0 QTime=25
2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 QTime=25
2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system
params={wt=json&_=1462389588126} status=0 QTime=25
2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
2016-05-04 19:21:29.145 INFO (qtp1989972246-5983) [ ]
o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
2016-05-04 19:21:29.146 INFO (qtp1989972246-5985) [ ]
o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
when connecting to {}->http://server2:8983: Too many open files
…
…
2016-05-04 19:21:29.150 INFO (qtp1989972246-5980) [ ]
o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
2016-05-04 19:21:29.151 ERROR (qtp1989972246-5985) [ ]
o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
trying to proxy request for url:
http://server2:8983/solr/collection1/admin/luke
at
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:588)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
server2/solr.log
======
016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.ServerConnector Started
ServerConnector@4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
2016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.Server Started @3850ms
2016-05-04 19:20:20.243 INFO (qtp1989972246-18) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=34
2016-05-04 19:20:20.254 INFO (qtp1989972246-15) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/zookeeper
params={path=/live_nodes&wt=json&_=1462389620113} status=0 QTime=42
2016-05-04 19:20:20.264 INFO (qtp1989972246-20) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system
params={wt=json&_=1462389620002} status=0 QTime=50
2016-05-04 19:20:20.321 INFO (qtp1989972246-19) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/zookeeper
params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1462389620113}
status=0 QTime=6
2016-05-04 19:20:20.360 INFO (qtp1989972246-21) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
2016-05-04 19:20:20.362 INFO (qtp1989972246-21) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={action=LIST&wt=json&_=1462389620002} status=0 QTime=19
2016-05-04 19:34:35.760 WARN
(qtp1989972246-16-acceptor-0@42e5f722-ServerConnector
@4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}) [ ] o.e.j.s.ServerConnector
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at
org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
2016-05-04 19:38:09.442 INFO (qtp1989972246-5289) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=0
2016-05-04 19:38:09.496 INFO (qtp1989972246-1224) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
params={wt=json&_=1462390689357&since=0} status=0 QTime=4
2016-05-04 19:38:09.587 INFO (qtp1989972246-5765) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system
params={wt=json&_=1462389620002} status=0 QTime=99
2016-05-04 19:38:09.656 INFO (qtp1989972246-4352) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
2016-05-04 19:38:09.658 INFO (qtp1989972246-4352) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
params={action=LIST&wt=json&_=1462389620002} status=0 QTime=2
2016-05-04 19:38:19.442 INFO (qtp1989972246-4324) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
params={wt=json&_=1462390689357&since=0} status=0 QTime=0
Re: Solr cloud 6.0.0 with ZooKeeper 3.4.8 Errors
Posted by Susheel Kumar <su...@gmail.com>.
Yes, Nick, I am using the chroot to share the ZK for different instances.
On Thu, May 5, 2016 at 3:08 PM, Nick Vasilyev <ni...@gmail.com>
wrote:
> Just out of curiosity, are you using sharing the zookeepers between the
> different versions of Solr? If So, are you specifying a zookeeper chroot?
> On May 5, 2016 2:05 PM, "Susheel Kumar" <su...@gmail.com> wrote:
>
> > Nick, Hoss - Things are back to normal with ZK 3.4.8 and ZK-6.0.0. I
> > switched to Solr 5.5.0 with ZK 3.4.8 which worked fine and then installed
> > 6.0.0. I suspect (not 100% sure) i left ZK dataDir / Solr collection
> > directory data from previous ZK/solr version which probably was making
> Solr
> > 6 in unstable state.
> >
> > Thanks,
> > Susheel
> >
> > On Wed, May 4, 2016 at 9:56 PM, Susheel Kumar <su...@gmail.com>
> > wrote:
> >
> > > Thanks, Nick & Hoss. I am using the exact same machine, have wiped out
> > > solr 5.5.0 and installed solr-6.0.0 with external ZK 3.4.8. I checked
> > the
> > > File Description limit for user solr, which is 12000 and increased to
> > > 52000. Don't see "too many files open..." error now in Solr log but
> still
> > > Solr connection getting lost in Admin panel.
> > >
> > > Let me do some more tests and install older version back to confirm and
> > > will share the findings.
> > >
> > > Thanks,
> > > Susheel
> > >
> > > On Wed, May 4, 2016 at 8:11 PM, Chris Hostetter <
> > hossman_lucene@fucit.org>
> > > wrote:
> > >
> > >>
> > >> : Thanks, Nick. Do we know any suggested # for file descriptor limit
> > with
> > >> : Solr6? Also wondering why i haven't seen this problem before with
> > Solr
> > >> 5.x?
> > >>
> > >> are you running Solr6 on the exact same host OS that you were running
> > >> Solr5 on?
> > >>
> > >> even if you are using the "same OS version" on a diff machine, that
> > could
> > >> explain the discrepency if you (or someone else) increased the file
> > >> descriptor limit on the "old machine" but that neverh appened on the
> > 'new
> > >> machine"
> > >>
> > >>
> > >>
> > >> : On Wed, May 4, 2016 at 4:54 PM, Nick Vasilyev <
> > nick.vasilyev1@gmail.com
> > >> >
> > >> : wrote:
> > >> :
> > >> : > It looks like you have too many open files, try increasing the
> file
> > >> : > descriptor limit.
> > >> : >
> > >> : > On Wed, May 4, 2016 at 3:48 PM, Susheel Kumar <
> > susheel2777@gmail.com>
> > >> : > wrote:
> > >> : >
> > >> : > > Hello,
> > >> : > >
> > >> : > > I am trying to setup 2 node Solr cloud 6 cluster with ZK 3.4.8
> and
> > >> used
> > >> : > the
> > >> : > > install service to setup solr.
> > >> : > >
> > >> : > > After launching Solr Admin Panel on server1, it looses
> connections
> > >> in few
> > >> : > > seconds and then comes back and other node server2 is marked as
> > >> Down in
> > >> : > > cloud graph. After few seconds its loosing the connection and
> > comes
> > >> back.
> > >> : > >
> > >> : > > Any idea what may be going wrong? Has anyone used Solr 6 with ZK
> > >> 3.4.8.
> > >> : > > Have never seen this error before with solr 5.x with ZK 3.4.6.
> > >> : > >
> > >> : > > Below log from server1 & server2. The ZK has 3 nodes with
> chroot
> > >> : > enabled.
> > >> : > >
> > >> : > > Thanks,
> > >> : > > Susheel
> > >> : > >
> > >> : > > server1/solr.log
> > >> : > >
> > >> : > > ====
> > >> : > >
> > >> : > >
> > >> : > > 2016-05-04 19:20:53.804 INFO (qtp1989972246-14) [ ]
> > >> : > > o.a.s.c.c.ZkStateReader path=[/collections/collection1]
> > >> : > > [configName]=[collection1] specified config exists in ZooKeeper
> > >> : > >
> > >> : > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
> > >> : > o.a.s.s.HttpSolrCall
> > >> : > > [admin] webapp=null path=/admin/collections
> > >> : > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0
> > >> QTime=25
> > >> : > >
> > >> : > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
> > >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list
> with
> > >> params
> > >> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> > >> : > >
> > >> : > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
> > >> : > o.a.s.s.HttpSolrCall
> > >> : > > [admin] webapp=null path=/admin/collections
> > >> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
> > >> : > >
> > >> : > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
> > >> : > o.a.s.s.HttpSolrCall
> > >> : > > [admin] webapp=null path=/admin/cores
> > >> : > > params={indexInfo=false&wt=json&_=1462389588124} status=0
> QTime=0
> > >> : > >
> > >> : > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
> > >> : > o.a.s.s.HttpSolrCall
> > >> : > > [admin] webapp=null path=/admin/info/system
> > >> : > > params={wt=json&_=1462389588126} status=0 QTime=25
> > >> : > >
> > >> : > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
> > >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list
> with
> > >> params
> > >> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> > >> : > >
> > >> : > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
> > >> : > o.a.s.s.HttpSolrCall
> > >> : > > [admin] webapp=null path=/admin/collections
> > >> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> > >> http://server2:8983
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> > >> http://server2:8983
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> > >> http://server2:8983
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> > >> http://server2:8983
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> > >> http://server2:8983
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> > >> http://server2:8983
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
> > >> : > o.a.s.s.HttpSolrCall
> > >> : > > [admin] webapp=null path=/admin/collections
> > >> : > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0
> > >> QTime=25
> > >> : > >
> > >> : > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
> > >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list
> with
> > >> params
> > >> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> > >> : > >
> > >> : > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
> > >> : > o.a.s.s.HttpSolrCall
> > >> : > > [admin] webapp=null path=/admin/collections
> > >> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
> > >> : > >
> > >> : > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
> > >> : > o.a.s.s.HttpSolrCall
> > >> : > > [admin] webapp=null path=/admin/cores
> > >> : > > params={indexInfo=false&wt=json&_=1462389588124} status=0
> QTime=0
> > >> : > >
> > >> : > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
> > >> : > o.a.s.s.HttpSolrCall
> > >> : > > [admin] webapp=null path=/admin/info/system
> > >> : > > params={wt=json&_=1462389588126} status=0 QTime=25
> > >> : > >
> > >> : > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
> > >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list
> with
> > >> params
> > >> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> > >> : > >
> > >> : > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
> > >> : > o.a.s.s.HttpSolrCall
> > >> : > > [admin] webapp=null path=/admin/collections
> > >> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> > >> http://server2:8983
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> > >> http://server2:8983
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> > >> http://server2:8983
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> > >> http://server2:8983
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> > >> http://server2:8983
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> > >> http://server2:8983
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.145 INFO (qtp1989972246-5983) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> > >> http://server2:8983
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.146 INFO (qtp1989972246-5985) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> > (java.net.SocketException)
> > >> : > caught
> > >> : > > when connecting to {}->http://server2:8983: Too many open files
> > >> : > >
> > >> : > > …
> > >> : > >
> > >> : > > …
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.150 INFO (qtp1989972246-5980) [ ]
> > >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> > >> http://server2:8983
> > >> : > >
> > >> : > > 2016-05-04 19:21:29.151 ERROR (qtp1989972246-5985) [ ]
> > >> : > > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException:
> > >> Error
> > >> : > > trying to proxy request for url:
> > >> : > > http://server2:8983/solr/collection1/admin/luke
> > >> : > >
> > >> : > > at
> > >> : > >
> > >>
> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:588)
> > >> : > >
> > >> : > > at
> > >> : > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)
> > >> : > >
> > >> : > > at
> > >> : > >
> > >> : > >
> > >> : >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> > >> : > >
> > >> : > > at
> > >> : > >
> > >> : > >
> > >> : >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> > >> : > >
> > >> : > > at
> > >> : > >
> > >> : > >
> > >> : >
> > >>
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> > >> : > >
> > >> : > > at
> > >> : > >
> > >> : >
> > >>
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> > >> : > >
> > >> : > > at
> > >> : > >
> > >> : > >
> > >> : >
> > >>
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > >> : > >
> > >> : > > at
> > >> : > >
> > >> : >
> > >>
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> > >> : > >
> > >> : > > at
> > >> : > >
> > >> : > >
> > >> : >
> > >>
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> > >> : > >
> > >> : > > at
> > >> : > >
> > >> : > >
> > >> : >
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> > >> : > >
> > >> : > >
> > >> : > >
> > >> : > >
> > >> : > >
> > >> : > > server2/solr.log
> > >> : > >
> > >> : > > ======
> > >> : > >
> > >> : > >
> > >> : > > 016-05-04 18:14:32.936 INFO (main) [ ]
> o.e.j.s.ServerConnector
> > >> Started
> > >> : > > ServerConnector@4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
> > >> : > >
> > >> : > > 2016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.Server
> Started
> > >> @3850ms
> > >> : > >
> > >> : > > 2016-05-04 19:20:20.243 INFO (qtp1989972246-18) [ ]
> > >> : > o.a.s.s.HttpSolrCall
> > >> : > > [admin] webapp=null path=/admin/cores
> > >> : > > params={indexInfo=false&wt=json&_=1462389620001} status=0
> QTime=34
> > >> : > >
> > >> : > > 2016-05-04 19:20:20.254 INFO (qtp1989972246-15) [ ]
> > >> : > o.a.s.s.HttpSolrCall
> > >> : > > [admin] webapp=null path=/admin/zookeeper
> > >> : > > params={path=/live_nodes&wt=json&_=1462389620113} status=0
> > QTime=42
> > >> : > >
> > >> : > > 2016-05-04 19:20:20.264 INFO (qtp1989972246-20) [ ]
> > >> : > o.a.s.s.HttpSolrCall
> > >> : > > [admin] webapp=null path=/admin/info/system
> > >> : > > params={wt=json&_=1462389620002} status=0 QTime=50
> > >> : > >
> > >> : > > 2016-05-04 19:20:20.321 INFO (qtp1989972246-19) [ ]
> > >> : > o.a.s.s.HttpSolrCall
> > >> : > > [admin] webapp=null path=/admin/zookeeper
> > >> : > >
> > >> : > >
> > >> : >
> > >>
> >
> params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1462389620113}
> > >> : > > status=0 QTime=6
> > >> : > >
> > >> : > > 2016-05-04 19:20:20.360 INFO (qtp1989972246-21) [ ]
> > >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list
> with
> > >> params
> > >> : > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
> > >> : > >
> > >> : > > 2016-05-04 19:20:20.362 INFO (qtp1989972246-21) [ ]
> > >> : > o.a.s.s.HttpSolrCall
> > >> : > > [admin] webapp=null path=/admin/collections
> > >> : > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=19
> > >> : > >
> > >> : > > 2016-05-04 19:34:35.760 WARN
> > >> : > > (qtp1989972246-16-acceptor-0@42e5f722-ServerConnector
> > >> : > > @4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}) [ ]
> > >> : > > o.e.j.s.ServerConnector
> > >> : > >
> > >> : > > java.io.IOException: Too many open files
> > >> : > >
> > >> : > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native
> > Method)
> > >> : > >
> > >> : > > at
> > >> : > >
> > >> : >
> > >>
> >
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> > >> : > >
> > >> : > > at
> > >> : > >
> > >> : >
> > >>
> >
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> > >> : > >
> > >> : > > at
> > >> : > >
> > >>
> > org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
> > >> : > >
> > >> : > > at
> > >> : > >
> > >> : > >
> > >> : >
> > >>
> >
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
> > >> : > >
> > >> : > > at
> > >> : > >
> > >> : > >
> > >> : >
> > >>
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> > >> : > >
> > >> : > > at
> > >> : > >
> > >> : > >
> > >> : >
> > >>
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > >> : > >
> > >> : > > at java.lang.Thread.run(Thread.java:745)
> > >> : > >
> > >> : > > 2016-05-04 19:38:09.442 INFO (qtp1989972246-5289) [ ]
> > >> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> > >> : > > params={indexInfo=false&wt=json&_=1462389620001} status=0
> QTime=0
> > >> : > >
> > >> : > > 2016-05-04 19:38:09.496 INFO (qtp1989972246-1224) [ ]
> > >> : > > o.a.s.s.HttpSolrCall [admin] webapp=null
> path=/admin/info/logging
> > >> : > > params={wt=json&_=1462390689357&since=0} status=0 QTime=4
> > >> : > >
> > >> : > > 2016-05-04 19:38:09.587 INFO (qtp1989972246-5765) [ ]
> > >> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system
> > >> : > > params={wt=json&_=1462389620002} status=0 QTime=99
> > >> : > >
> > >> : > > 2016-05-04 19:38:09.656 INFO (qtp1989972246-4352) [ ]
> > >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list
> with
> > >> params
> > >> : > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
> > >> : > >
> > >> : > > 2016-05-04 19:38:09.658 INFO (qtp1989972246-4352) [ ]
> > >> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> > >> : > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=2
> > >> : > >
> > >> : > > 2016-05-04 19:38:19.442 INFO (qtp1989972246-4324) [ ]
> > >> : > > o.a.s.s.HttpSolrCall [admin] webapp=null
> path=/admin/info/logging
> > >> : > > params={wt=json&_=1462390689357&since=0} status=0 QTime=0
> > >> : > >
> > >> : >
> > >> :
> > >>
> > >> -Hoss
> > >> http://www.lucidworks.com/
> > >
> > >
> > >
> >
>
Re: Solr cloud 6.0.0 with ZooKeeper 3.4.8 Errors
Posted by Nick Vasilyev <ni...@gmail.com>.
Just out of curiosity, are you using sharing the zookeepers between the
different versions of Solr? If So, are you specifying a zookeeper chroot?
On May 5, 2016 2:05 PM, "Susheel Kumar" <su...@gmail.com> wrote:
> Nick, Hoss - Things are back to normal with ZK 3.4.8 and ZK-6.0.0. I
> switched to Solr 5.5.0 with ZK 3.4.8 which worked fine and then installed
> 6.0.0. I suspect (not 100% sure) i left ZK dataDir / Solr collection
> directory data from previous ZK/solr version which probably was making Solr
> 6 in unstable state.
>
> Thanks,
> Susheel
>
> On Wed, May 4, 2016 at 9:56 PM, Susheel Kumar <su...@gmail.com>
> wrote:
>
> > Thanks, Nick & Hoss. I am using the exact same machine, have wiped out
> > solr 5.5.0 and installed solr-6.0.0 with external ZK 3.4.8. I checked
> the
> > File Description limit for user solr, which is 12000 and increased to
> > 52000. Don't see "too many files open..." error now in Solr log but still
> > Solr connection getting lost in Admin panel.
> >
> > Let me do some more tests and install older version back to confirm and
> > will share the findings.
> >
> > Thanks,
> > Susheel
> >
> > On Wed, May 4, 2016 at 8:11 PM, Chris Hostetter <
> hossman_lucene@fucit.org>
> > wrote:
> >
> >>
> >> : Thanks, Nick. Do we know any suggested # for file descriptor limit
> with
> >> : Solr6? Also wondering why i haven't seen this problem before with
> Solr
> >> 5.x?
> >>
> >> are you running Solr6 on the exact same host OS that you were running
> >> Solr5 on?
> >>
> >> even if you are using the "same OS version" on a diff machine, that
> could
> >> explain the discrepency if you (or someone else) increased the file
> >> descriptor limit on the "old machine" but that neverh appened on the
> 'new
> >> machine"
> >>
> >>
> >>
> >> : On Wed, May 4, 2016 at 4:54 PM, Nick Vasilyev <
> nick.vasilyev1@gmail.com
> >> >
> >> : wrote:
> >> :
> >> : > It looks like you have too many open files, try increasing the file
> >> : > descriptor limit.
> >> : >
> >> : > On Wed, May 4, 2016 at 3:48 PM, Susheel Kumar <
> susheel2777@gmail.com>
> >> : > wrote:
> >> : >
> >> : > > Hello,
> >> : > >
> >> : > > I am trying to setup 2 node Solr cloud 6 cluster with ZK 3.4.8 and
> >> used
> >> : > the
> >> : > > install service to setup solr.
> >> : > >
> >> : > > After launching Solr Admin Panel on server1, it looses connections
> >> in few
> >> : > > seconds and then comes back and other node server2 is marked as
> >> Down in
> >> : > > cloud graph. After few seconds its loosing the connection and
> comes
> >> back.
> >> : > >
> >> : > > Any idea what may be going wrong? Has anyone used Solr 6 with ZK
> >> 3.4.8.
> >> : > > Have never seen this error before with solr 5.x with ZK 3.4.6.
> >> : > >
> >> : > > Below log from server1 & server2. The ZK has 3 nodes with chroot
> >> : > enabled.
> >> : > >
> >> : > > Thanks,
> >> : > > Susheel
> >> : > >
> >> : > > server1/solr.log
> >> : > >
> >> : > > ====
> >> : > >
> >> : > >
> >> : > > 2016-05-04 19:20:53.804 INFO (qtp1989972246-14) [ ]
> >> : > > o.a.s.c.c.ZkStateReader path=[/collections/collection1]
> >> : > > [configName]=[collection1] specified config exists in ZooKeeper
> >> : > >
> >> : > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
> >> : > o.a.s.s.HttpSolrCall
> >> : > > [admin] webapp=null path=/admin/collections
> >> : > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0
> >> QTime=25
> >> : > >
> >> : > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
> >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> >> params
> >> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> >> : > >
> >> : > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
> >> : > o.a.s.s.HttpSolrCall
> >> : > > [admin] webapp=null path=/admin/collections
> >> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
> >> : > >
> >> : > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
> >> : > o.a.s.s.HttpSolrCall
> >> : > > [admin] webapp=null path=/admin/cores
> >> : > > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
> >> : > >
> >> : > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
> >> : > o.a.s.s.HttpSolrCall
> >> : > > [admin] webapp=null path=/admin/info/system
> >> : > > params={wt=json&_=1462389588126} status=0 QTime=25
> >> : > >
> >> : > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
> >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> >> params
> >> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> >> : > >
> >> : > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
> >> : > o.a.s.s.HttpSolrCall
> >> : > > [admin] webapp=null path=/admin/collections
> >> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
> >> : > >
> >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> >> http://server2:8983
> >> : > >
> >> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> >> http://server2:8983
> >> : > >
> >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> >> http://server2:8983
> >> : > >
> >> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> >> http://server2:8983
> >> : > >
> >> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> >> http://server2:8983
> >> : > >
> >> : > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> >> http://server2:8983
> >> : > >
> >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
> >> : > o.a.s.s.HttpSolrCall
> >> : > > [admin] webapp=null path=/admin/collections
> >> : > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0
> >> QTime=25
> >> : > >
> >> : > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
> >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> >> params
> >> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> >> : > >
> >> : > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
> >> : > o.a.s.s.HttpSolrCall
> >> : > > [admin] webapp=null path=/admin/collections
> >> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
> >> : > >
> >> : > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
> >> : > o.a.s.s.HttpSolrCall
> >> : > > [admin] webapp=null path=/admin/cores
> >> : > > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
> >> : > >
> >> : > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
> >> : > o.a.s.s.HttpSolrCall
> >> : > > [admin] webapp=null path=/admin/info/system
> >> : > > params={wt=json&_=1462389588126} status=0 QTime=25
> >> : > >
> >> : > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
> >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> >> params
> >> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> >> : > >
> >> : > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
> >> : > o.a.s.s.HttpSolrCall
> >> : > > [admin] webapp=null path=/admin/collections
> >> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
> >> : > >
> >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> >> http://server2:8983
> >> : > >
> >> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> >> http://server2:8983
> >> : > >
> >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> >> http://server2:8983
> >> : > >
> >> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> >> http://server2:8983
> >> : > >
> >> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> >> http://server2:8983
> >> : > >
> >> : > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> >> http://server2:8983
> >> : > >
> >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > 2016-05-04 19:21:29.145 INFO (qtp1989972246-5983) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> >> http://server2:8983
> >> : > >
> >> : > > 2016-05-04 19:21:29.146 INFO (qtp1989972246-5985) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient I/O exception
> (java.net.SocketException)
> >> : > caught
> >> : > > when connecting to {}->http://server2:8983: Too many open files
> >> : > >
> >> : > > …
> >> : > >
> >> : > > …
> >> : > >
> >> : > > 2016-05-04 19:21:29.150 INFO (qtp1989972246-5980) [ ]
> >> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> >> http://server2:8983
> >> : > >
> >> : > > 2016-05-04 19:21:29.151 ERROR (qtp1989972246-5985) [ ]
> >> : > > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException:
> >> Error
> >> : > > trying to proxy request for url:
> >> : > > http://server2:8983/solr/collection1/admin/luke
> >> : > >
> >> : > > at
> >> : > >
> >> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:588)
> >> : > >
> >> : > > at
> >> : > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)
> >> : > >
> >> : > > at
> >> : > >
> >> : > >
> >> : >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> >> : > >
> >> : > > at
> >> : > >
> >> : > >
> >> : >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> >> : > >
> >> : > > at
> >> : > >
> >> : > >
> >> : >
> >>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> >> : > >
> >> : > > at
> >> : > >
> >> : >
> >>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> >> : > >
> >> : > > at
> >> : > >
> >> : > >
> >> : >
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> >> : > >
> >> : > > at
> >> : > >
> >> : >
> >>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> >> : > >
> >> : > > at
> >> : > >
> >> : > >
> >> : >
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> >> : > >
> >> : > > at
> >> : > >
> >> : > >
> >> : >
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> >> : > >
> >> : > >
> >> : > >
> >> : > >
> >> : > >
> >> : > > server2/solr.log
> >> : > >
> >> : > > ======
> >> : > >
> >> : > >
> >> : > > 016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.ServerConnector
> >> Started
> >> : > > ServerConnector@4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
> >> : > >
> >> : > > 2016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.Server Started
> >> @3850ms
> >> : > >
> >> : > > 2016-05-04 19:20:20.243 INFO (qtp1989972246-18) [ ]
> >> : > o.a.s.s.HttpSolrCall
> >> : > > [admin] webapp=null path=/admin/cores
> >> : > > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=34
> >> : > >
> >> : > > 2016-05-04 19:20:20.254 INFO (qtp1989972246-15) [ ]
> >> : > o.a.s.s.HttpSolrCall
> >> : > > [admin] webapp=null path=/admin/zookeeper
> >> : > > params={path=/live_nodes&wt=json&_=1462389620113} status=0
> QTime=42
> >> : > >
> >> : > > 2016-05-04 19:20:20.264 INFO (qtp1989972246-20) [ ]
> >> : > o.a.s.s.HttpSolrCall
> >> : > > [admin] webapp=null path=/admin/info/system
> >> : > > params={wt=json&_=1462389620002} status=0 QTime=50
> >> : > >
> >> : > > 2016-05-04 19:20:20.321 INFO (qtp1989972246-19) [ ]
> >> : > o.a.s.s.HttpSolrCall
> >> : > > [admin] webapp=null path=/admin/zookeeper
> >> : > >
> >> : > >
> >> : >
> >>
> params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1462389620113}
> >> : > > status=0 QTime=6
> >> : > >
> >> : > > 2016-05-04 19:20:20.360 INFO (qtp1989972246-21) [ ]
> >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> >> params
> >> : > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
> >> : > >
> >> : > > 2016-05-04 19:20:20.362 INFO (qtp1989972246-21) [ ]
> >> : > o.a.s.s.HttpSolrCall
> >> : > > [admin] webapp=null path=/admin/collections
> >> : > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=19
> >> : > >
> >> : > > 2016-05-04 19:34:35.760 WARN
> >> : > > (qtp1989972246-16-acceptor-0@42e5f722-ServerConnector
> >> : > > @4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}) [ ]
> >> : > > o.e.j.s.ServerConnector
> >> : > >
> >> : > > java.io.IOException: Too many open files
> >> : > >
> >> : > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native
> Method)
> >> : > >
> >> : > > at
> >> : > >
> >> : >
> >>
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> >> : > >
> >> : > > at
> >> : > >
> >> : >
> >>
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> >> : > >
> >> : > > at
> >> : > >
> >>
> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
> >> : > >
> >> : > > at
> >> : > >
> >> : > >
> >> : >
> >>
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
> >> : > >
> >> : > > at
> >> : > >
> >> : > >
> >> : >
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> >> : > >
> >> : > > at
> >> : > >
> >> : > >
> >> : >
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >> : > >
> >> : > > at java.lang.Thread.run(Thread.java:745)
> >> : > >
> >> : > > 2016-05-04 19:38:09.442 INFO (qtp1989972246-5289) [ ]
> >> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> >> : > > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=0
> >> : > >
> >> : > > 2016-05-04 19:38:09.496 INFO (qtp1989972246-1224) [ ]
> >> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
> >> : > > params={wt=json&_=1462390689357&since=0} status=0 QTime=4
> >> : > >
> >> : > > 2016-05-04 19:38:09.587 INFO (qtp1989972246-5765) [ ]
> >> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system
> >> : > > params={wt=json&_=1462389620002} status=0 QTime=99
> >> : > >
> >> : > > 2016-05-04 19:38:09.656 INFO (qtp1989972246-4352) [ ]
> >> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> >> params
> >> : > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
> >> : > >
> >> : > > 2016-05-04 19:38:09.658 INFO (qtp1989972246-4352) [ ]
> >> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> >> : > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=2
> >> : > >
> >> : > > 2016-05-04 19:38:19.442 INFO (qtp1989972246-4324) [ ]
> >> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
> >> : > > params={wt=json&_=1462390689357&since=0} status=0 QTime=0
> >> : > >
> >> : >
> >> :
> >>
> >> -Hoss
> >> http://www.lucidworks.com/
> >
> >
> >
>
Re: Solr cloud 6.0.0 with ZooKeeper 3.4.8 Errors
Posted by Susheel Kumar <su...@gmail.com>.
Nick, Hoss - Things are back to normal with ZK 3.4.8 and ZK-6.0.0. I
switched to Solr 5.5.0 with ZK 3.4.8 which worked fine and then installed
6.0.0. I suspect (not 100% sure) i left ZK dataDir / Solr collection
directory data from previous ZK/solr version which probably was making Solr
6 in unstable state.
Thanks,
Susheel
On Wed, May 4, 2016 at 9:56 PM, Susheel Kumar <su...@gmail.com> wrote:
> Thanks, Nick & Hoss. I am using the exact same machine, have wiped out
> solr 5.5.0 and installed solr-6.0.0 with external ZK 3.4.8. I checked the
> File Description limit for user solr, which is 12000 and increased to
> 52000. Don't see "too many files open..." error now in Solr log but still
> Solr connection getting lost in Admin panel.
>
> Let me do some more tests and install older version back to confirm and
> will share the findings.
>
> Thanks,
> Susheel
>
> On Wed, May 4, 2016 at 8:11 PM, Chris Hostetter <ho...@fucit.org>
> wrote:
>
>>
>> : Thanks, Nick. Do we know any suggested # for file descriptor limit with
>> : Solr6? Also wondering why i haven't seen this problem before with Solr
>> 5.x?
>>
>> are you running Solr6 on the exact same host OS that you were running
>> Solr5 on?
>>
>> even if you are using the "same OS version" on a diff machine, that could
>> explain the discrepency if you (or someone else) increased the file
>> descriptor limit on the "old machine" but that neverh appened on the 'new
>> machine"
>>
>>
>>
>> : On Wed, May 4, 2016 at 4:54 PM, Nick Vasilyev <nick.vasilyev1@gmail.com
>> >
>> : wrote:
>> :
>> : > It looks like you have too many open files, try increasing the file
>> : > descriptor limit.
>> : >
>> : > On Wed, May 4, 2016 at 3:48 PM, Susheel Kumar <su...@gmail.com>
>> : > wrote:
>> : >
>> : > > Hello,
>> : > >
>> : > > I am trying to setup 2 node Solr cloud 6 cluster with ZK 3.4.8 and
>> used
>> : > the
>> : > > install service to setup solr.
>> : > >
>> : > > After launching Solr Admin Panel on server1, it looses connections
>> in few
>> : > > seconds and then comes back and other node server2 is marked as
>> Down in
>> : > > cloud graph. After few seconds its loosing the connection and comes
>> back.
>> : > >
>> : > > Any idea what may be going wrong? Has anyone used Solr 6 with ZK
>> 3.4.8.
>> : > > Have never seen this error before with solr 5.x with ZK 3.4.6.
>> : > >
>> : > > Below log from server1 & server2. The ZK has 3 nodes with chroot
>> : > enabled.
>> : > >
>> : > > Thanks,
>> : > > Susheel
>> : > >
>> : > > server1/solr.log
>> : > >
>> : > > ====
>> : > >
>> : > >
>> : > > 2016-05-04 19:20:53.804 INFO (qtp1989972246-14) [ ]
>> : > > o.a.s.c.c.ZkStateReader path=[/collections/collection1]
>> : > > [configName]=[collection1] specified config exists in ZooKeeper
>> : > >
>> : > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
>> : > o.a.s.s.HttpSolrCall
>> : > > [admin] webapp=null path=/admin/collections
>> : > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0
>> QTime=25
>> : > >
>> : > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
>> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
>> params
>> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
>> : > >
>> : > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
>> : > o.a.s.s.HttpSolrCall
>> : > > [admin] webapp=null path=/admin/collections
>> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
>> : > >
>> : > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
>> : > o.a.s.s.HttpSolrCall
>> : > > [admin] webapp=null path=/admin/cores
>> : > > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
>> : > >
>> : > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
>> : > o.a.s.s.HttpSolrCall
>> : > > [admin] webapp=null path=/admin/info/system
>> : > > params={wt=json&_=1462389588126} status=0 QTime=25
>> : > >
>> : > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
>> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
>> params
>> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
>> : > >
>> : > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
>> : > o.a.s.s.HttpSolrCall
>> : > > [admin] webapp=null path=/admin/collections
>> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
>> : > >
>> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
>> http://server2:8983
>> : > >
>> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
>> http://server2:8983
>> : > >
>> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
>> http://server2:8983
>> : > >
>> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
>> http://server2:8983
>> : > >
>> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
>> http://server2:8983
>> : > >
>> : > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
>> http://server2:8983
>> : > >
>> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
>> : > o.a.s.s.HttpSolrCall
>> : > > [admin] webapp=null path=/admin/collections
>> : > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0
>> QTime=25
>> : > >
>> : > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
>> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
>> params
>> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
>> : > >
>> : > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
>> : > o.a.s.s.HttpSolrCall
>> : > > [admin] webapp=null path=/admin/collections
>> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
>> : > >
>> : > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
>> : > o.a.s.s.HttpSolrCall
>> : > > [admin] webapp=null path=/admin/cores
>> : > > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
>> : > >
>> : > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
>> : > o.a.s.s.HttpSolrCall
>> : > > [admin] webapp=null path=/admin/info/system
>> : > > params={wt=json&_=1462389588126} status=0 QTime=25
>> : > >
>> : > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
>> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
>> params
>> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
>> : > >
>> : > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
>> : > o.a.s.s.HttpSolrCall
>> : > > [admin] webapp=null path=/admin/collections
>> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
>> : > >
>> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
>> http://server2:8983
>> : > >
>> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
>> http://server2:8983
>> : > >
>> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
>> http://server2:8983
>> : > >
>> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
>> http://server2:8983
>> : > >
>> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
>> http://server2:8983
>> : > >
>> : > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
>> http://server2:8983
>> : > >
>> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > 2016-05-04 19:21:29.145 INFO (qtp1989972246-5983) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
>> http://server2:8983
>> : > >
>> : > > 2016-05-04 19:21:29.146 INFO (qtp1989972246-5985) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
>> : > caught
>> : > > when connecting to {}->http://server2:8983: Too many open files
>> : > >
>> : > > …
>> : > >
>> : > > …
>> : > >
>> : > > 2016-05-04 19:21:29.150 INFO (qtp1989972246-5980) [ ]
>> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
>> http://server2:8983
>> : > >
>> : > > 2016-05-04 19:21:29.151 ERROR (qtp1989972246-5985) [ ]
>> : > > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException:
>> Error
>> : > > trying to proxy request for url:
>> : > > http://server2:8983/solr/collection1/admin/luke
>> : > >
>> : > > at
>> : > >
>> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:588)
>> : > >
>> : > > at
>> : > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)
>> : > >
>> : > > at
>> : > >
>> : > >
>> : >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
>> : > >
>> : > > at
>> : > >
>> : > >
>> : >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
>> : > >
>> : > > at
>> : > >
>> : > >
>> : >
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>> : > >
>> : > > at
>> : > >
>> : >
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>> : > >
>> : > > at
>> : > >
>> : > >
>> : >
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>> : > >
>> : > > at
>> : > >
>> : >
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>> : > >
>> : > > at
>> : > >
>> : > >
>> : >
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>> : > >
>> : > > at
>> : > >
>> : > >
>> : >
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>> : > >
>> : > >
>> : > >
>> : > >
>> : > >
>> : > > server2/solr.log
>> : > >
>> : > > ======
>> : > >
>> : > >
>> : > > 016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.ServerConnector
>> Started
>> : > > ServerConnector@4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
>> : > >
>> : > > 2016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.Server Started
>> @3850ms
>> : > >
>> : > > 2016-05-04 19:20:20.243 INFO (qtp1989972246-18) [ ]
>> : > o.a.s.s.HttpSolrCall
>> : > > [admin] webapp=null path=/admin/cores
>> : > > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=34
>> : > >
>> : > > 2016-05-04 19:20:20.254 INFO (qtp1989972246-15) [ ]
>> : > o.a.s.s.HttpSolrCall
>> : > > [admin] webapp=null path=/admin/zookeeper
>> : > > params={path=/live_nodes&wt=json&_=1462389620113} status=0 QTime=42
>> : > >
>> : > > 2016-05-04 19:20:20.264 INFO (qtp1989972246-20) [ ]
>> : > o.a.s.s.HttpSolrCall
>> : > > [admin] webapp=null path=/admin/info/system
>> : > > params={wt=json&_=1462389620002} status=0 QTime=50
>> : > >
>> : > > 2016-05-04 19:20:20.321 INFO (qtp1989972246-19) [ ]
>> : > o.a.s.s.HttpSolrCall
>> : > > [admin] webapp=null path=/admin/zookeeper
>> : > >
>> : > >
>> : >
>> params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1462389620113}
>> : > > status=0 QTime=6
>> : > >
>> : > > 2016-05-04 19:20:20.360 INFO (qtp1989972246-21) [ ]
>> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
>> params
>> : > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
>> : > >
>> : > > 2016-05-04 19:20:20.362 INFO (qtp1989972246-21) [ ]
>> : > o.a.s.s.HttpSolrCall
>> : > > [admin] webapp=null path=/admin/collections
>> : > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=19
>> : > >
>> : > > 2016-05-04 19:34:35.760 WARN
>> : > > (qtp1989972246-16-acceptor-0@42e5f722-ServerConnector
>> : > > @4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}) [ ]
>> : > > o.e.j.s.ServerConnector
>> : > >
>> : > > java.io.IOException: Too many open files
>> : > >
>> : > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>> : > >
>> : > > at
>> : > >
>> : >
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>> : > >
>> : > > at
>> : > >
>> : >
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>> : > >
>> : > > at
>> : > >
>> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
>> : > >
>> : > > at
>> : > >
>> : > >
>> : >
>> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
>> : > >
>> : > > at
>> : > >
>> : > >
>> : >
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>> : > >
>> : > > at
>> : > >
>> : > >
>> : >
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>> : > >
>> : > > at java.lang.Thread.run(Thread.java:745)
>> : > >
>> : > > 2016-05-04 19:38:09.442 INFO (qtp1989972246-5289) [ ]
>> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
>> : > > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=0
>> : > >
>> : > > 2016-05-04 19:38:09.496 INFO (qtp1989972246-1224) [ ]
>> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
>> : > > params={wt=json&_=1462390689357&since=0} status=0 QTime=4
>> : > >
>> : > > 2016-05-04 19:38:09.587 INFO (qtp1989972246-5765) [ ]
>> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system
>> : > > params={wt=json&_=1462389620002} status=0 QTime=99
>> : > >
>> : > > 2016-05-04 19:38:09.656 INFO (qtp1989972246-4352) [ ]
>> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
>> params
>> : > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
>> : > >
>> : > > 2016-05-04 19:38:09.658 INFO (qtp1989972246-4352) [ ]
>> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
>> : > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=2
>> : > >
>> : > > 2016-05-04 19:38:19.442 INFO (qtp1989972246-4324) [ ]
>> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
>> : > > params={wt=json&_=1462390689357&since=0} status=0 QTime=0
>> : > >
>> : >
>> :
>>
>> -Hoss
>> http://www.lucidworks.com/
>
>
>
Re: Solr cloud 6.0.0 with ZooKeeper 3.4.8 Errors
Posted by Susheel Kumar <su...@gmail.com>.
Thanks, Nick & Hoss. I am using the exact same machine, have wiped out
solr 5.5.0 and installed solr-6.0.0 with external ZK 3.4.8. I checked the
File Description limit for user solr, which is 12000 and increased to
52000. Don't see "too many files open..." error now in Solr log but still
Solr connection getting lost in Admin panel.
Let me do some more tests and install older version back to confirm and
will share the findings.
Thanks,
Susheel
On Wed, May 4, 2016 at 8:11 PM, Chris Hostetter <ho...@fucit.org>
wrote:
>
> : Thanks, Nick. Do we know any suggested # for file descriptor limit with
> : Solr6? Also wondering why i haven't seen this problem before with Solr
> 5.x?
>
> are you running Solr6 on the exact same host OS that you were running
> Solr5 on?
>
> even if you are using the "same OS version" on a diff machine, that could
> explain the discrepency if you (or someone else) increased the file
> descriptor limit on the "old machine" but that neverh appened on the 'new
> machine"
>
>
>
> : On Wed, May 4, 2016 at 4:54 PM, Nick Vasilyev <ni...@gmail.com>
> : wrote:
> :
> : > It looks like you have too many open files, try increasing the file
> : > descriptor limit.
> : >
> : > On Wed, May 4, 2016 at 3:48 PM, Susheel Kumar <su...@gmail.com>
> : > wrote:
> : >
> : > > Hello,
> : > >
> : > > I am trying to setup 2 node Solr cloud 6 cluster with ZK 3.4.8 and
> used
> : > the
> : > > install service to setup solr.
> : > >
> : > > After launching Solr Admin Panel on server1, it looses connections
> in few
> : > > seconds and then comes back and other node server2 is marked as Down
> in
> : > > cloud graph. After few seconds its loosing the connection and comes
> back.
> : > >
> : > > Any idea what may be going wrong? Has anyone used Solr 6 with ZK
> 3.4.8.
> : > > Have never seen this error before with solr 5.x with ZK 3.4.6.
> : > >
> : > > Below log from server1 & server2. The ZK has 3 nodes with chroot
> : > enabled.
> : > >
> : > > Thanks,
> : > > Susheel
> : > >
> : > > server1/solr.log
> : > >
> : > > ====
> : > >
> : > >
> : > > 2016-05-04 19:20:53.804 INFO (qtp1989972246-14) [ ]
> : > > o.a.s.c.c.ZkStateReader path=[/collections/collection1]
> : > > [configName]=[collection1] specified config exists in ZooKeeper
> : > >
> : > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
> : > o.a.s.s.HttpSolrCall
> : > > [admin] webapp=null path=/admin/collections
> : > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0
> QTime=25
> : > >
> : > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> params
> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> : > >
> : > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
> : > o.a.s.s.HttpSolrCall
> : > > [admin] webapp=null path=/admin/collections
> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
> : > >
> : > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
> : > o.a.s.s.HttpSolrCall
> : > > [admin] webapp=null path=/admin/cores
> : > > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
> : > >
> : > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
> : > o.a.s.s.HttpSolrCall
> : > > [admin] webapp=null path=/admin/info/system
> : > > params={wt=json&_=1462389588126} status=0 QTime=25
> : > >
> : > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> params
> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> : > >
> : > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
> : > o.a.s.s.HttpSolrCall
> : > > [admin] webapp=null path=/admin/collections
> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
> : > >
> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> : > >
> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> : > >
> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> : > >
> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> : > >
> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> : > >
> : > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> : > >
> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
> : > o.a.s.s.HttpSolrCall
> : > > [admin] webapp=null path=/admin/collections
> : > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0
> QTime=25
> : > >
> : > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> params
> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> : > >
> : > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
> : > o.a.s.s.HttpSolrCall
> : > > [admin] webapp=null path=/admin/collections
> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
> : > >
> : > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
> : > o.a.s.s.HttpSolrCall
> : > > [admin] webapp=null path=/admin/cores
> : > > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
> : > >
> : > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
> : > o.a.s.s.HttpSolrCall
> : > > [admin] webapp=null path=/admin/info/system
> : > > params={wt=json&_=1462389588126} status=0 QTime=25
> : > >
> : > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> params
> : > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> : > >
> : > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
> : > o.a.s.s.HttpSolrCall
> : > > [admin] webapp=null path=/admin/collections
> : > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
> : > >
> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> : > >
> : > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> : > >
> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> : > >
> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> : > >
> : > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> : > >
> : > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> : > >
> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > 2016-05-04 19:21:29.145 INFO (qtp1989972246-5983) [ ]
> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> : > >
> : > > 2016-05-04 19:21:29.146 INFO (qtp1989972246-5985) [ ]
> : > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> : > caught
> : > > when connecting to {}->http://server2:8983: Too many open files
> : > >
> : > > …
> : > >
> : > > …
> : > >
> : > > 2016-05-04 19:21:29.150 INFO (qtp1989972246-5980) [ ]
> : > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> : > >
> : > > 2016-05-04 19:21:29.151 ERROR (qtp1989972246-5985) [ ]
> : > > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
> : > > trying to proxy request for url:
> : > > http://server2:8983/solr/collection1/admin/luke
> : > >
> : > > at
> : > >
> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:588)
> : > >
> : > > at
> : > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)
> : > >
> : > > at
> : > >
> : > >
> : >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> : > >
> : > > at
> : > >
> : > >
> : >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> : > >
> : > > at
> : > >
> : > >
> : >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> : > >
> : > > at
> : > >
> : >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> : > >
> : > > at
> : > >
> : > >
> : >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> : > >
> : > > at
> : > >
> : >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> : > >
> : > > at
> : > >
> : > >
> : >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> : > >
> : > > at
> : > >
> : > >
> : >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> : > >
> : > >
> : > >
> : > >
> : > >
> : > > server2/solr.log
> : > >
> : > > ======
> : > >
> : > >
> : > > 016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.ServerConnector
> Started
> : > > ServerConnector@4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
> : > >
> : > > 2016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.Server Started
> @3850ms
> : > >
> : > > 2016-05-04 19:20:20.243 INFO (qtp1989972246-18) [ ]
> : > o.a.s.s.HttpSolrCall
> : > > [admin] webapp=null path=/admin/cores
> : > > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=34
> : > >
> : > > 2016-05-04 19:20:20.254 INFO (qtp1989972246-15) [ ]
> : > o.a.s.s.HttpSolrCall
> : > > [admin] webapp=null path=/admin/zookeeper
> : > > params={path=/live_nodes&wt=json&_=1462389620113} status=0 QTime=42
> : > >
> : > > 2016-05-04 19:20:20.264 INFO (qtp1989972246-20) [ ]
> : > o.a.s.s.HttpSolrCall
> : > > [admin] webapp=null path=/admin/info/system
> : > > params={wt=json&_=1462389620002} status=0 QTime=50
> : > >
> : > > 2016-05-04 19:20:20.321 INFO (qtp1989972246-19) [ ]
> : > o.a.s.s.HttpSolrCall
> : > > [admin] webapp=null path=/admin/zookeeper
> : > >
> : > >
> : >
> params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1462389620113}
> : > > status=0 QTime=6
> : > >
> : > > 2016-05-04 19:20:20.360 INFO (qtp1989972246-21) [ ]
> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> params
> : > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
> : > >
> : > > 2016-05-04 19:20:20.362 INFO (qtp1989972246-21) [ ]
> : > o.a.s.s.HttpSolrCall
> : > > [admin] webapp=null path=/admin/collections
> : > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=19
> : > >
> : > > 2016-05-04 19:34:35.760 WARN
> : > > (qtp1989972246-16-acceptor-0@42e5f722-ServerConnector
> : > > @4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}) [ ]
> : > > o.e.j.s.ServerConnector
> : > >
> : > > java.io.IOException: Too many open files
> : > >
> : > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> : > >
> : > > at
> : > >
> : >
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> : > >
> : > > at
> : > >
> : >
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> : > >
> : > > at
> : > >
> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
> : > >
> : > > at
> : > >
> : > >
> : >
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
> : > >
> : > > at
> : > >
> : > >
> : >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> : > >
> : > > at
> : > >
> : > >
> : >
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> : > >
> : > > at java.lang.Thread.run(Thread.java:745)
> : > >
> : > > 2016-05-04 19:38:09.442 INFO (qtp1989972246-5289) [ ]
> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> : > > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=0
> : > >
> : > > 2016-05-04 19:38:09.496 INFO (qtp1989972246-1224) [ ]
> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
> : > > params={wt=json&_=1462390689357&since=0} status=0 QTime=4
> : > >
> : > > 2016-05-04 19:38:09.587 INFO (qtp1989972246-5765) [ ]
> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system
> : > > params={wt=json&_=1462389620002} status=0 QTime=99
> : > >
> : > > 2016-05-04 19:38:09.656 INFO (qtp1989972246-4352) [ ]
> : > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> params
> : > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
> : > >
> : > > 2016-05-04 19:38:09.658 INFO (qtp1989972246-4352) [ ]
> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> : > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=2
> : > >
> : > > 2016-05-04 19:38:19.442 INFO (qtp1989972246-4324) [ ]
> : > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
> : > > params={wt=json&_=1462390689357&since=0} status=0 QTime=0
> : > >
> : >
> :
>
> -Hoss
> http://www.lucidworks.com/
Re: Solr cloud 6.0.0 with ZooKeeper 3.4.8 Errors
Posted by Chris Hostetter <ho...@fucit.org>.
: Thanks, Nick. Do we know any suggested # for file descriptor limit with
: Solr6? Also wondering why i haven't seen this problem before with Solr 5.x?
are you running Solr6 on the exact same host OS that you were running
Solr5 on?
even if you are using the "same OS version" on a diff machine, that could
explain the discrepency if you (or someone else) increased the file
descriptor limit on the "old machine" but that neverh appened on the 'new
machine"
: On Wed, May 4, 2016 at 4:54 PM, Nick Vasilyev <ni...@gmail.com>
: wrote:
:
: > It looks like you have too many open files, try increasing the file
: > descriptor limit.
: >
: > On Wed, May 4, 2016 at 3:48 PM, Susheel Kumar <su...@gmail.com>
: > wrote:
: >
: > > Hello,
: > >
: > > I am trying to setup 2 node Solr cloud 6 cluster with ZK 3.4.8 and used
: > the
: > > install service to setup solr.
: > >
: > > After launching Solr Admin Panel on server1, it looses connections in few
: > > seconds and then comes back and other node server2 is marked as Down in
: > > cloud graph. After few seconds its loosing the connection and comes back.
: > >
: > > Any idea what may be going wrong? Has anyone used Solr 6 with ZK 3.4.8.
: > > Have never seen this error before with solr 5.x with ZK 3.4.6.
: > >
: > > Below log from server1 & server2. The ZK has 3 nodes with chroot
: > enabled.
: > >
: > > Thanks,
: > > Susheel
: > >
: > > server1/solr.log
: > >
: > > ====
: > >
: > >
: > > 2016-05-04 19:20:53.804 INFO (qtp1989972246-14) [ ]
: > > o.a.s.c.c.ZkStateReader path=[/collections/collection1]
: > > [configName]=[collection1] specified config exists in ZooKeeper
: > >
: > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/collections
: > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 QTime=25
: > >
: > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
: > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
: > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
: > >
: > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/collections
: > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
: > >
: > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/cores
: > > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
: > >
: > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/info/system
: > > params={wt=json&_=1462389588126} status=0 QTime=25
: > >
: > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
: > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
: > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
: > >
: > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/collections
: > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
: > >
: > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/collections
: > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 QTime=25
: > >
: > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
: > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
: > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
: > >
: > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/collections
: > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
: > >
: > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/cores
: > > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
: > >
: > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/info/system
: > > params={wt=json&_=1462389588126} status=0 QTime=25
: > >
: > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
: > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
: > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
: > >
: > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/collections
: > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
: > >
: > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > 2016-05-04 19:21:29.145 INFO (qtp1989972246-5983) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.146 INFO (qtp1989972246-5985) [ ]
: > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
: > caught
: > > when connecting to {}->http://server2:8983: Too many open files
: > >
: > > \u2026
: > >
: > > \u2026
: > >
: > > 2016-05-04 19:21:29.150 INFO (qtp1989972246-5980) [ ]
: > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
: > >
: > > 2016-05-04 19:21:29.151 ERROR (qtp1989972246-5985) [ ]
: > > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
: > > trying to proxy request for url:
: > > http://server2:8983/solr/collection1/admin/luke
: > >
: > > at
: > > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:588)
: > >
: > > at
: > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)
: > >
: > > at
: > >
: > >
: > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
: > >
: > > at
: > >
: > >
: > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
: > >
: > > at
: > >
: > >
: > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
: > >
: > > at
: > >
: > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
: > >
: > > at
: > >
: > >
: > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
: > >
: > > at
: > >
: > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
: > >
: > > at
: > >
: > >
: > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
: > >
: > > at
: > >
: > >
: > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
: > >
: > >
: > >
: > >
: > >
: > > server2/solr.log
: > >
: > > ======
: > >
: > >
: > > 016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.ServerConnector Started
: > > ServerConnector@4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
: > >
: > > 2016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.Server Started @3850ms
: > >
: > > 2016-05-04 19:20:20.243 INFO (qtp1989972246-18) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/cores
: > > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=34
: > >
: > > 2016-05-04 19:20:20.254 INFO (qtp1989972246-15) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/zookeeper
: > > params={path=/live_nodes&wt=json&_=1462389620113} status=0 QTime=42
: > >
: > > 2016-05-04 19:20:20.264 INFO (qtp1989972246-20) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/info/system
: > > params={wt=json&_=1462389620002} status=0 QTime=50
: > >
: > > 2016-05-04 19:20:20.321 INFO (qtp1989972246-19) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/zookeeper
: > >
: > >
: > params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1462389620113}
: > > status=0 QTime=6
: > >
: > > 2016-05-04 19:20:20.360 INFO (qtp1989972246-21) [ ]
: > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
: > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
: > >
: > > 2016-05-04 19:20:20.362 INFO (qtp1989972246-21) [ ]
: > o.a.s.s.HttpSolrCall
: > > [admin] webapp=null path=/admin/collections
: > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=19
: > >
: > > 2016-05-04 19:34:35.760 WARN
: > > (qtp1989972246-16-acceptor-0@42e5f722-ServerConnector
: > > @4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}) [ ]
: > > o.e.j.s.ServerConnector
: > >
: > > java.io.IOException: Too many open files
: > >
: > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
: > >
: > > at
: > >
: > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
: > >
: > > at
: > >
: > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
: > >
: > > at
: > > org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
: > >
: > > at
: > >
: > >
: > org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
: > >
: > > at
: > >
: > >
: > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
: > >
: > > at
: > >
: > >
: > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
: > >
: > > at java.lang.Thread.run(Thread.java:745)
: > >
: > > 2016-05-04 19:38:09.442 INFO (qtp1989972246-5289) [ ]
: > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
: > > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=0
: > >
: > > 2016-05-04 19:38:09.496 INFO (qtp1989972246-1224) [ ]
: > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
: > > params={wt=json&_=1462390689357&since=0} status=0 QTime=4
: > >
: > > 2016-05-04 19:38:09.587 INFO (qtp1989972246-5765) [ ]
: > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system
: > > params={wt=json&_=1462389620002} status=0 QTime=99
: > >
: > > 2016-05-04 19:38:09.656 INFO (qtp1989972246-4352) [ ]
: > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
: > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
: > >
: > > 2016-05-04 19:38:09.658 INFO (qtp1989972246-4352) [ ]
: > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
: > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=2
: > >
: > > 2016-05-04 19:38:19.442 INFO (qtp1989972246-4324) [ ]
: > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
: > > params={wt=json&_=1462390689357&since=0} status=0 QTime=0
: > >
: >
:
-Hoss
http://www.lucidworks.com/
Re: Solr cloud 6.0.0 with ZooKeeper 3.4.8 Errors
Posted by Nick Vasilyev <ni...@gmail.com>.
Not sure about your environment so it's hard to say why you haven't ran
into this issue before.
As for the suggested limit, I am not sure, it would depend on your system
and if you really want to limit it. I personally just jack it up to 50000.
On Wed, May 4, 2016 at 6:13 PM, Susheel Kumar <su...@gmail.com> wrote:
> Thanks, Nick. Do we know any suggested # for file descriptor limit with
> Solr6? Also wondering why i haven't seen this problem before with Solr
> 5.x?
>
> On Wed, May 4, 2016 at 4:54 PM, Nick Vasilyev <ni...@gmail.com>
> wrote:
>
> > It looks like you have too many open files, try increasing the file
> > descriptor limit.
> >
> > On Wed, May 4, 2016 at 3:48 PM, Susheel Kumar <su...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > I am trying to setup 2 node Solr cloud 6 cluster with ZK 3.4.8 and used
> > the
> > > install service to setup solr.
> > >
> > > After launching Solr Admin Panel on server1, it looses connections in
> few
> > > seconds and then comes back and other node server2 is marked as Down in
> > > cloud graph. After few seconds its loosing the connection and comes
> back.
> > >
> > > Any idea what may be going wrong? Has anyone used Solr 6 with ZK 3.4.8.
> > > Have never seen this error before with solr 5.x with ZK 3.4.6.
> > >
> > > Below log from server1 & server2. The ZK has 3 nodes with chroot
> > enabled.
> > >
> > > Thanks,
> > > Susheel
> > >
> > > server1/solr.log
> > >
> > > ====
> > >
> > >
> > > 2016-05-04 19:20:53.804 INFO (qtp1989972246-14) [ ]
> > > o.a.s.c.c.ZkStateReader path=[/collections/collection1]
> > > [configName]=[collection1] specified config exists in ZooKeeper
> > >
> > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
> > o.a.s.s.HttpSolrCall
> > > [admin] webapp=null path=/admin/collections
> > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 QTime=25
> > >
> > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
> > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> params
> > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> > >
> > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
> > o.a.s.s.HttpSolrCall
> > > [admin] webapp=null path=/admin/collections
> > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
> > >
> > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
> > o.a.s.s.HttpSolrCall
> > > [admin] webapp=null path=/admin/cores
> > > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
> > >
> > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
> > o.a.s.s.HttpSolrCall
> > > [admin] webapp=null path=/admin/info/system
> > > params={wt=json&_=1462389588126} status=0 QTime=25
> > >
> > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
> > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> params
> > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> > >
> > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
> > o.a.s.s.HttpSolrCall
> > > [admin] webapp=null path=/admin/collections
> > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
> > >
> > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> > >
> > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> > >
> > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> > >
> > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
> > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> > >
> > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
> > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> > >
> > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> > >
> > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
> > o.a.s.s.HttpSolrCall
> > > [admin] webapp=null path=/admin/collections
> > > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 QTime=25
> > >
> > > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
> > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> params
> > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> > >
> > > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
> > o.a.s.s.HttpSolrCall
> > > [admin] webapp=null path=/admin/collections
> > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
> > >
> > > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
> > o.a.s.s.HttpSolrCall
> > > [admin] webapp=null path=/admin/cores
> > > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
> > >
> > > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
> > o.a.s.s.HttpSolrCall
> > > [admin] webapp=null path=/admin/info/system
> > > params={wt=json&_=1462389588126} status=0 QTime=25
> > >
> > > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
> > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> params
> > > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> > >
> > > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
> > o.a.s.s.HttpSolrCall
> > > [admin] webapp=null path=/admin/collections
> > > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
> > >
> > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> > >
> > > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> > >
> > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> > >
> > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
> > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> > >
> > > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
> > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> > >
> > > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> > >
> > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > 2016-05-04 19:21:29.145 INFO (qtp1989972246-5983) [ ]
> > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> > >
> > > 2016-05-04 19:21:29.146 INFO (qtp1989972246-5985) [ ]
> > > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> > caught
> > > when connecting to {}->http://server2:8983: Too many open files
> > >
> > > …
> > >
> > > …
> > >
> > > 2016-05-04 19:21:29.150 INFO (qtp1989972246-5980) [ ]
> > > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->
> http://server2:8983
> > >
> > > 2016-05-04 19:21:29.151 ERROR (qtp1989972246-5985) [ ]
> > > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
> > > trying to proxy request for url:
> > > http://server2:8983/solr/collection1/admin/luke
> > >
> > > at
> > > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:588)
> > >
> > > at
> > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)
> > >
> > > at
> > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> > >
> > > at
> > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> > >
> > > at
> > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> > >
> > > at
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> > >
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > >
> > > at
> > >
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> > >
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> > >
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> > >
> > >
> > >
> > >
> > >
> > > server2/solr.log
> > >
> > > ======
> > >
> > >
> > > 016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.ServerConnector
> Started
> > > ServerConnector@4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
> > >
> > > 2016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.Server Started
> @3850ms
> > >
> > > 2016-05-04 19:20:20.243 INFO (qtp1989972246-18) [ ]
> > o.a.s.s.HttpSolrCall
> > > [admin] webapp=null path=/admin/cores
> > > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=34
> > >
> > > 2016-05-04 19:20:20.254 INFO (qtp1989972246-15) [ ]
> > o.a.s.s.HttpSolrCall
> > > [admin] webapp=null path=/admin/zookeeper
> > > params={path=/live_nodes&wt=json&_=1462389620113} status=0 QTime=42
> > >
> > > 2016-05-04 19:20:20.264 INFO (qtp1989972246-20) [ ]
> > o.a.s.s.HttpSolrCall
> > > [admin] webapp=null path=/admin/info/system
> > > params={wt=json&_=1462389620002} status=0 QTime=50
> > >
> > > 2016-05-04 19:20:20.321 INFO (qtp1989972246-19) [ ]
> > o.a.s.s.HttpSolrCall
> > > [admin] webapp=null path=/admin/zookeeper
> > >
> > >
> >
> params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1462389620113}
> > > status=0 QTime=6
> > >
> > > 2016-05-04 19:20:20.360 INFO (qtp1989972246-21) [ ]
> > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> params
> > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
> > >
> > > 2016-05-04 19:20:20.362 INFO (qtp1989972246-21) [ ]
> > o.a.s.s.HttpSolrCall
> > > [admin] webapp=null path=/admin/collections
> > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=19
> > >
> > > 2016-05-04 19:34:35.760 WARN
> > > (qtp1989972246-16-acceptor-0@42e5f722-ServerConnector
> > > @4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}) [ ]
> > > o.e.j.s.ServerConnector
> > >
> > > java.io.IOException: Too many open files
> > >
> > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> > >
> > > at
> > >
> >
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> > >
> > > at
> > >
> >
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> > >
> > > at
> > >
> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
> > >
> > > at
> > >
> > >
> >
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
> > >
> > > at
> > >
> > >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> > >
> > > at
> > >
> > >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > 2016-05-04 19:38:09.442 INFO (qtp1989972246-5289) [ ]
> > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> > > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=0
> > >
> > > 2016-05-04 19:38:09.496 INFO (qtp1989972246-1224) [ ]
> > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
> > > params={wt=json&_=1462390689357&since=0} status=0 QTime=4
> > >
> > > 2016-05-04 19:38:09.587 INFO (qtp1989972246-5765) [ ]
> > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system
> > > params={wt=json&_=1462389620002} status=0 QTime=99
> > >
> > > 2016-05-04 19:38:09.656 INFO (qtp1989972246-4352) [ ]
> > > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with
> params
> > > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
> > >
> > > 2016-05-04 19:38:09.658 INFO (qtp1989972246-4352) [ ]
> > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> > > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=2
> > >
> > > 2016-05-04 19:38:19.442 INFO (qtp1989972246-4324) [ ]
> > > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
> > > params={wt=json&_=1462390689357&since=0} status=0 QTime=0
> > >
> >
>
Re: Solr cloud 6.0.0 with ZooKeeper 3.4.8 Errors
Posted by Susheel Kumar <su...@gmail.com>.
Thanks, Nick. Do we know any suggested # for file descriptor limit with
Solr6? Also wondering why i haven't seen this problem before with Solr 5.x?
On Wed, May 4, 2016 at 4:54 PM, Nick Vasilyev <ni...@gmail.com>
wrote:
> It looks like you have too many open files, try increasing the file
> descriptor limit.
>
> On Wed, May 4, 2016 at 3:48 PM, Susheel Kumar <su...@gmail.com>
> wrote:
>
> > Hello,
> >
> > I am trying to setup 2 node Solr cloud 6 cluster with ZK 3.4.8 and used
> the
> > install service to setup solr.
> >
> > After launching Solr Admin Panel on server1, it looses connections in few
> > seconds and then comes back and other node server2 is marked as Down in
> > cloud graph. After few seconds its loosing the connection and comes back.
> >
> > Any idea what may be going wrong? Has anyone used Solr 6 with ZK 3.4.8.
> > Have never seen this error before with solr 5.x with ZK 3.4.6.
> >
> > Below log from server1 & server2. The ZK has 3 nodes with chroot
> enabled.
> >
> > Thanks,
> > Susheel
> >
> > server1/solr.log
> >
> > ====
> >
> >
> > 2016-05-04 19:20:53.804 INFO (qtp1989972246-14) [ ]
> > o.a.s.c.c.ZkStateReader path=[/collections/collection1]
> > [configName]=[collection1] specified config exists in ZooKeeper
> >
> > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/collections
> > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 QTime=25
> >
> > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> >
> > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/collections
> > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
> >
> > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/cores
> > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
> >
> > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/info/system
> > params={wt=json&_=1462389588126} status=0 QTime=25
> >
> > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> >
> > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/collections
> > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
> >
> > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/collections
> > params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 QTime=25
> >
> > 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> >
> > 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/collections
> > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
> >
> > 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/cores
> > params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
> >
> > 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/info/system
> > params={wt=json&_=1462389588126} status=0 QTime=25
> >
> > 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> > action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
> >
> > 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/collections
> > params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
> >
> > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > 2016-05-04 19:21:29.145 INFO (qtp1989972246-5983) [ ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.146 INFO (qtp1989972246-5985) [ ]
> > o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException)
> caught
> > when connecting to {}->http://server2:8983: Too many open files
> >
> > …
> >
> > …
> >
> > 2016-05-04 19:21:29.150 INFO (qtp1989972246-5980) [ ]
> > o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
> >
> > 2016-05-04 19:21:29.151 ERROR (qtp1989972246-5985) [ ]
> > o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
> > trying to proxy request for url:
> > http://server2:8983/solr/collection1/admin/luke
> >
> > at
> > org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:588)
> >
> > at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)
> >
> > at
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
> >
> > at
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> >
> > at
> >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> >
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> >
> > at
> >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> >
> > at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> >
> > at
> >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> >
> > at
> >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> >
> >
> >
> >
> >
> > server2/solr.log
> >
> > ======
> >
> >
> > 016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.ServerConnector Started
> > ServerConnector@4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
> >
> > 2016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.Server Started @3850ms
> >
> > 2016-05-04 19:20:20.243 INFO (qtp1989972246-18) [ ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/cores
> > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=34
> >
> > 2016-05-04 19:20:20.254 INFO (qtp1989972246-15) [ ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/zookeeper
> > params={path=/live_nodes&wt=json&_=1462389620113} status=0 QTime=42
> >
> > 2016-05-04 19:20:20.264 INFO (qtp1989972246-20) [ ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/info/system
> > params={wt=json&_=1462389620002} status=0 QTime=50
> >
> > 2016-05-04 19:20:20.321 INFO (qtp1989972246-19) [ ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/zookeeper
> >
> >
> params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1462389620113}
> > status=0 QTime=6
> >
> > 2016-05-04 19:20:20.360 INFO (qtp1989972246-21) [ ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
> >
> > 2016-05-04 19:20:20.362 INFO (qtp1989972246-21) [ ]
> o.a.s.s.HttpSolrCall
> > [admin] webapp=null path=/admin/collections
> > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=19
> >
> > 2016-05-04 19:34:35.760 WARN
> > (qtp1989972246-16-acceptor-0@42e5f722-ServerConnector
> > @4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}) [ ]
> > o.e.j.s.ServerConnector
> >
> > java.io.IOException: Too many open files
> >
> > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> >
> > at
> >
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> >
> > at
> >
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> >
> > at
> > org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
> >
> > at
> >
> >
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
> >
> > at
> >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> >
> > at
> >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> >
> > at java.lang.Thread.run(Thread.java:745)
> >
> > 2016-05-04 19:38:09.442 INFO (qtp1989972246-5289) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> > params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=0
> >
> > 2016-05-04 19:38:09.496 INFO (qtp1989972246-1224) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
> > params={wt=json&_=1462390689357&since=0} status=0 QTime=4
> >
> > 2016-05-04 19:38:09.587 INFO (qtp1989972246-5765) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system
> > params={wt=json&_=1462389620002} status=0 QTime=99
> >
> > 2016-05-04 19:38:09.656 INFO (qtp1989972246-4352) [ ]
> > o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> > action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
> >
> > 2016-05-04 19:38:09.658 INFO (qtp1989972246-4352) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> > params={action=LIST&wt=json&_=1462389620002} status=0 QTime=2
> >
> > 2016-05-04 19:38:19.442 INFO (qtp1989972246-4324) [ ]
> > o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
> > params={wt=json&_=1462390689357&since=0} status=0 QTime=0
> >
>
Re: Solr cloud 6.0.0 with ZooKeeper 3.4.8 Errors
Posted by Nick Vasilyev <ni...@gmail.com>.
It looks like you have too many open files, try increasing the file
descriptor limit.
On Wed, May 4, 2016 at 3:48 PM, Susheel Kumar <su...@gmail.com> wrote:
> Hello,
>
> I am trying to setup 2 node Solr cloud 6 cluster with ZK 3.4.8 and used the
> install service to setup solr.
>
> After launching Solr Admin Panel on server1, it looses connections in few
> seconds and then comes back and other node server2 is marked as Down in
> cloud graph. After few seconds its loosing the connection and comes back.
>
> Any idea what may be going wrong? Has anyone used Solr 6 with ZK 3.4.8.
> Have never seen this error before with solr 5.x with ZK 3.4.6.
>
> Below log from server1 & server2. The ZK has 3 nodes with chroot enabled.
>
> Thanks,
> Susheel
>
> server1/solr.log
>
> ====
>
>
> 2016-05-04 19:20:53.804 INFO (qtp1989972246-14) [ ]
> o.a.s.c.c.ZkStateReader path=[/collections/collection1]
> [configName]=[collection1] specified config exists in ZooKeeper
>
> 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/collections
> params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 QTime=25
>
> 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
> o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
>
> 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/collections
> params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
>
> 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/cores
> params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
>
> 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/info/system
> params={wt=json&_=1462389588126} status=0 QTime=25
>
> 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
> o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
>
> 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/collections
> params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
>
> 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
>
> 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
>
> 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
>
> 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
> o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
>
> 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
> o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
>
> 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
>
> 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:20:53.806 INFO (qtp1989972246-14) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/collections
> params={action=CLUSTERSTATUS&wt=json&_=1462389588125} status=0 QTime=25
>
> 2016-05-04 19:20:53.859 INFO (qtp1989972246-19) [ ]
> o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
>
> 2016-05-04 19:20:53.861 INFO (qtp1989972246-19) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/collections
> params={action=LIST&wt=json&_=1462389588125} status=0 QTime=2
>
> 2016-05-04 19:20:57.520 INFO (qtp1989972246-13) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/cores
> params={indexInfo=false&wt=json&_=1462389588124} status=0 QTime=0
>
> 2016-05-04 19:20:57.546 INFO (qtp1989972246-15) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/info/system
> params={wt=json&_=1462389588126} status=0 QTime=25
>
> 2016-05-04 19:20:57.610 INFO (qtp1989972246-13) [ ]
> o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> action=LIST&wt=json&_=1462389588125 and sendToOCPQueue=true
>
> 2016-05-04 19:20:57.613 INFO (qtp1989972246-13) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/collections
> params={action=LIST&wt=json&_=1462389588125} status=0 QTime=3
>
> 2016-05-04 19:21:29.139 INFO (qtp1989972246-5980) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:21:29.139 INFO (qtp1989972246-5983) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:21:29.139 INFO (qtp1989972246-5984) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
>
> 2016-05-04 19:21:29.141 INFO (qtp1989972246-5984) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
>
> 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:21:29.142 INFO (qtp1989972246-5984) [ ]
> o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
>
> 2016-05-04 19:21:29.140 INFO (qtp1989972246-5983) [ ]
> o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
>
> 2016-05-04 19:21:29.140 INFO (qtp1989972246-5980) [ ]
> o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
>
> 2016-05-04 19:21:29.143 INFO (qtp1989972246-5983) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
>
> 2016-05-04 19:21:29.144 INFO (qtp1989972246-5980) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:21:29.144 INFO (qtp1989972246-5983) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> 2016-05-04 19:21:29.145 INFO (qtp1989972246-5983) [ ]
> o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
>
> 2016-05-04 19:21:29.146 INFO (qtp1989972246-5985) [ ]
> o.a.h.i.c.DefaultHttpClient I/O exception (java.net.SocketException) caught
> when connecting to {}->http://server2:8983: Too many open files
>
> …
>
> …
>
> 2016-05-04 19:21:29.150 INFO (qtp1989972246-5980) [ ]
> o.a.h.i.c.DefaultHttpClient Retrying connect to {}->http://server2:8983
>
> 2016-05-04 19:21:29.151 ERROR (qtp1989972246-5985) [ ]
> o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
> trying to proxy request for url:
> http://server2:8983/solr/collection1/admin/luke
>
> at
> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:588)
>
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)
>
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
>
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
>
> at
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>
> at
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>
> at
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>
>
>
>
>
> server2/solr.log
>
> ======
>
>
> 016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.ServerConnector Started
> ServerConnector@4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
>
> 2016-05-04 18:14:32.936 INFO (main) [ ] o.e.j.s.Server Started @3850ms
>
> 2016-05-04 19:20:20.243 INFO (qtp1989972246-18) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/cores
> params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=34
>
> 2016-05-04 19:20:20.254 INFO (qtp1989972246-15) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/zookeeper
> params={path=/live_nodes&wt=json&_=1462389620113} status=0 QTime=42
>
> 2016-05-04 19:20:20.264 INFO (qtp1989972246-20) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/info/system
> params={wt=json&_=1462389620002} status=0 QTime=50
>
> 2016-05-04 19:20:20.321 INFO (qtp1989972246-19) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/zookeeper
>
> params={path=/clusterstate.json&view=graph&detail=true&wt=json&_=1462389620113}
> status=0 QTime=6
>
> 2016-05-04 19:20:20.360 INFO (qtp1989972246-21) [ ]
> o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
>
> 2016-05-04 19:20:20.362 INFO (qtp1989972246-21) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/collections
> params={action=LIST&wt=json&_=1462389620002} status=0 QTime=19
>
> 2016-05-04 19:34:35.760 WARN
> (qtp1989972246-16-acceptor-0@42e5f722-ServerConnector
> @4c22110e{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}) [ ]
> o.e.j.s.ServerConnector
>
> java.io.IOException: Too many open files
>
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>
> at
> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
>
> at
>
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
>
> at
>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>
> at
>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 2016-05-04 19:38:09.442 INFO (qtp1989972246-5289) [ ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> params={indexInfo=false&wt=json&_=1462389620001} status=0 QTime=0
>
> 2016-05-04 19:38:09.496 INFO (qtp1989972246-1224) [ ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
> params={wt=json&_=1462390689357&since=0} status=0 QTime=4
>
> 2016-05-04 19:38:09.587 INFO (qtp1989972246-5765) [ ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system
> params={wt=json&_=1462389620002} status=0 QTime=99
>
> 2016-05-04 19:38:09.656 INFO (qtp1989972246-4352) [ ]
> o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params
> action=LIST&wt=json&_=1462389620002 and sendToOCPQueue=true
>
> 2016-05-04 19:38:09.658 INFO (qtp1989972246-4352) [ ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> params={action=LIST&wt=json&_=1462389620002} status=0 QTime=2
>
> 2016-05-04 19:38:19.442 INFO (qtp1989972246-4324) [ ]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/logging
> params={wt=json&_=1462390689357&since=0} status=0 QTime=0
>