You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Jungtaek Lim <ka...@gmail.com> on 2016/08/27 09:37:49 UTC
Re: Storm nimbus election
We need to see Nimbus log to find out why Nimbus is down. Pasting or
attaching Nimbus log would be better.
And please explain your Storm version too.
Jungtaek Lim (HeartSaVioR)
2016년 8월 26일 (금) 오후 1:16, pradeep s <sr...@gmail.com>님이 작성:
> Hi ,
> While restarting nimbus and ui, i am getting
> NimbusLeaderNotFoundException..
> While setting up the cluster also this error came. That time i have
> pointed zookeper data directory to a new directory and issue was resolved.
> Any idea on below exception.
>
> org.apache.storm.utils.NimbusLeaderNotFoundException: Found nimbuses [] none of which is elected as leader, please try again after some time.
> at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:85)
> at org.apache.storm.ui.core$cluster_summary.invoke(core.clj:356)
> at org.apache.storm.ui.core$fn__12108.invoke(core.clj:936)
> at org.apache.storm.shade.compojure.core$make_route$fn__2467.invoke(core.clj:93)
> at org.apache.storm.shade.compojure.core$if_route$fn__2455.invoke(core.clj:39)
> at org.apache.storm.shade.compojure.core$if_method$fn__2448.invoke(core.clj:24)
> at org.apache.storm.shade.compojure.core$routing$fn__2473.invoke(core.clj:106)
> at clojure.core$some.invoke(core.clj:2570)
> at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106)
> at clojure.lang.RestFn.applyTo(RestFn.java:139)
> at clojure.core$apply.invoke(core.clj:632)
> at org.apache.storm.shade.compojure.core$routes$fn__2477.invoke(core.clj:111)
> at org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__11576.invoke(json.clj:56)
> at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__3543.invoke(multipart_params.clj:103)
> at org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__4286.invoke(reload.clj:22)
> at org.apache.storm.ui.helpers$requests_middleware$fn__3770.invoke(helpers.clj:46)
> at org.apache.storm.ui.core$catch_errors$fn__12301.invoke(core.clj:1230)
> at org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__3474.invoke(keyword_params.clj:27)
> at org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__3514.invoke(nested_params.clj:65)
> at org.apache.storm.shade.ring.middleware.params$wrap_params$fn__3445.invoke(params.clj:55)
> at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__3543.invoke(multipart_params.clj:103)
> at org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__3729.invoke(flash.clj:14)
> at org.apache.storm.shade.ring.middleware.session$wrap_session$fn__3717.invoke(session.clj:43)
> at org.apache.storm.shade.ring.middleware.cookies$wrap_cookies$fn__3645.invoke(cookies.clj:160)
> at org.apache.storm.shade.ring.util.servlet$make_service_method$fn__3351.invoke(servlet.clj:127)
> at org.apache.storm.shade.ring.util.servlet$servlet$fn__3355.invoke(servlet.clj:136)
> at org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown Source)
> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654)
> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320)
> at org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47)
> at org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39)
> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
> at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
> at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
> at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
> at org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
> at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
> at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
> at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
> at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
> at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
> at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
>
Re: Storm nimbus election
Posted by pradeep s <sr...@gmail.com>.
Thanks Jung.
It was a zookeeper configuration issue.myid file configuration was not
proper. Resolved after updating the myid file
On Sun, Aug 28, 2016 at 4:27 AM, Jungtaek Lim <ka...@gmail.com> wrote:
> As you suspected there seems to be ZK issue.
> Unfortunately I don't have experience to operate Storm cluster with EC2.
> You may want to check each Zookeeper instance is accessible from your Storm
> cluster.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 8월 28일 (일) 오후 5:27, pradeep s <sr...@gmail.com>님이 작성:
>
>> Hi Juntaek,
>> I am running Storm 1.0 on AWS .Figured that i have this issue when i have
>> multiple zookeeper servers. I was using 3 zookeeper servers.
>> When i make to only one zookeeper there is no issue found . Nimbus and UI
>> works fine.
>> Any reason for the nimbus failure when i have multiple zookeepers . Was
>> seeing below logs in nimbus when i had 3 zookeepers.
>> Do you have any recommendations on setting zookeeper cluster on ec2
>>
>> 016-08-26 23:46:47.822 o.a.s.d.common [INFO] Started statistics report
>> plugin...
>> 2016-08-26 23:46:47.854 o.a.s.d.nimbus [INFO] Starting nimbus server for
>> storm version '1.0.0'
>> 2016-08-27 00:07:37.520 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
>> additional data from server sessionid 0x156c9158b980003, likely server has
>> closed socket, closing socket connection and attempting reconnect
>> 2016-08-27 00:07:37.520 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
>> additional data from server sessionid 0x156c9158b980002, likely server has
>> closed socket, closing socket connection and attempting reconnect
>> 2016-08-27 00:07:37.621 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO]
>> State change: SUSPENDED
>> 2016-08-27 00:07:37.622 o.a.s.zookeeper [INFO] Zookeeper state update:
>> :disconnected:none
>> 2016-08-27 00:07:37.622 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO]
>> State change: SUSPENDED
>> 2016-08-27 00:07:37.623 o.a.s.c.zookeeper-state-factory [WARN] Received
>> event :disconnected::none: with disconnected Writer Zookeeper.
>> 2016-08-27 00:07:37.623 o.a.s.zookeeper [INFO] ip-xxxx21-38-235.cloud.
>> nordstrom.net lost leadership.
>> 2016-08-27 00:07:37.803 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
>> connection to server xxxx21.50.34/xxxx21.50.34:2181. Will not attempt to
>> authenticate using SASL (unknown error)
>> 2016-08-27 00:07:37.804 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
>> established to xxxx21.50.34/xxxx21.50.34:2181, initiating session
>> 2016-08-27 00:07:37.805 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
>> additional data from server sessionid 0x156c9158b980003, likely server has
>> closed socket, closing socket connection and attempting reconnect
>> 2016-08-27 00:07:38.152 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
>> connection to server xxxx21.81.19/xxxx21.81.19:2181. Will not attempt to
>> authenticate using SASL (unknown error)
>> 2016-08-27 00:07:38.153 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
>> established to xxxx21.81.19/xxxx21.81.19:2181, initiating session
>> 2016-08-27 00:07:38.155 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
>> additional data from server sessionid 0x156c9158b980002, likely server has
>> closed socket, closing socket connection and attempting reconnect
>> 2016-08-27 00:07:38.395 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
>> connection to server xxxx21.81.19/xxxx21.81.19:2181. Will not attempt to
>> authenticate using SASL (unknown error)
>> 2016-08-27 00:07:38.396 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
>> established to xxxx21.81.19/xxxx21.81.19:2181, initiating session
>> 2016-08-27 00:07:38.398 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
>> additional data from server sessionid 0x156c9158b980003, likely server has
>> closed socket, closing socket connection and attempting reconnect
>> 2016-08-27 00:07:38.979 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
>> connection to server xxxx21.50.34/xxxx21.50.34:2181. Will not attempt to
>> authenticate using SASL (unknown error)
>> 2016-08-27 00:07:38.979 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
>> established to xxxx21.50.34/xxxx21.50.34:2181, initiating session
>> 2016-08-27 00:07:38.980 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
>> additional data from server sessionid 0x156c9158b980002, likely server has
>> closed socket, closing socket connection and attempting reconnect
>> 2016-08-27 00:07:40.299 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
>> connection to server xxxx21.106.227/xxxx21.106.227:2181. Will not
>> attempt to authenticate using SASL (unknown error)
>> 2016-08-27 00:07:40.300 o.a.s.s.o.a.z.ClientCnxn [WARN] Session
>> 0x156c9158b980003 for server null, unexpected error, closing socket
>> connection and attempting reconnect
>> java.net.ConnectException: Connection refused
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at sun.nio.ch.SocketChannelImpl.finishConnect(
>> SocketChannelImpl.java:717)
>> at org.apache.storm.shade.org.apache.zookeeper.
>> ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$
>> SendThread.run(ClientCnxn.java:1081)
>> 2016-08-27 00:07:40.713 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
>> connection to server xxxx21.106.227/xxxx21.106.227:2181. Will not
>> attempt to authenticate using SASL (unknown error)
>>
>> Regards
>> Pradeep S
>>
>> On Sat, Aug 27, 2016 at 2:37 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>>
>>> We need to see Nimbus log to find out why Nimbus is down. Pasting or
>>> attaching Nimbus log would be better.
>>> And please explain your Storm version too.
>>>
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>> 2016년 8월 26일 (금) 오후 1:16, pradeep s <sr...@gmail.com>님이 작성:
>>>
>>>> Hi ,
>>>> While restarting nimbus and ui, i am getting
>>>> NimbusLeaderNotFoundException..
>>>> While setting up the cluster also this error came. That time i have
>>>> pointed zookeper data directory to a new directory and issue was resolved.
>>>> Any idea on below exception.
>>>>
>>>> org.apache.storm.utils.NimbusLeaderNotFoundException: Found nimbuses [] none of which is elected as leader, please try again after some time.
>>>> at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:85)
>>>> at org.apache.storm.ui.core$cluster_summary.invoke(core.clj:356)
>>>> at org.apache.storm.ui.core$fn__12108.invoke(core.clj:936)
>>>> at org.apache.storm.shade.compojure.core$make_route$fn__2467.invoke(core.clj:93)
>>>> at org.apache.storm.shade.compojure.core$if_route$fn__2455.invoke(core.clj:39)
>>>> at org.apache.storm.shade.compojure.core$if_method$fn__2448.invoke(core.clj:24)
>>>> at org.apache.storm.shade.compojure.core$routing$fn__2473.invoke(core.clj:106)
>>>> at clojure.core$some.invoke(core.clj:2570)
>>>> at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106)
>>>> at clojure.lang.RestFn.applyTo(RestFn.java:139)
>>>> at clojure.core$apply.invoke(core.clj:632)
>>>> at org.apache.storm.shade.compojure.core$routes$fn__2477.invoke(core.clj:111)
>>>> at org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__11576.invoke(json.clj:56)
>>>> at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__3543.invoke(multipart_params.clj:103)
>>>> at org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__4286.invoke(reload.clj:22)
>>>> at org.apache.storm.ui.helpers$requests_middleware$fn__3770.invoke(helpers.clj:46)
>>>> at org.apache.storm.ui.core$catch_errors$fn__12301.invoke(core.clj:1230)
>>>> at org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__3474.invoke(keyword_params.clj:27)
>>>> at org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__3514.invoke(nested_params.clj:65)
>>>> at org.apache.storm.shade.ring.middleware.params$wrap_params$fn__3445.invoke(params.clj:55)
>>>> at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__3543.invoke(multipart_params.clj:103)
>>>> at org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__3729.invoke(flash.clj:14)
>>>> at org.apache.storm.shade.ring.middleware.session$wrap_session$fn__3717.invoke(session.clj:43)
>>>> at org.apache.storm.shade.ring.middleware.cookies$wrap_cookies$fn__3645.invoke(cookies.clj:160)
>>>> at org.apache.storm.shade.ring.util.servlet$make_service_method$fn__3351.invoke(servlet.clj:127)
>>>> at org.apache.storm.shade.ring.util.servlet$servlet$fn__3355.invoke(servlet.clj:136)
>>>> at org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown Source)
>>>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654)
>>>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320)
>>>> at org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47)
>>>> at org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39)
>>>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
>>>> at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
>>>> at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
>>>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
>>>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
>>>> at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
>>>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
>>>> at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
>>>> at org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>>>> at org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>>>> at org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
>>>> at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
>>>> at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
>>>> at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
>>>> at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
>>>> at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>>>> at org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>>>> at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
>>>> at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>>>> at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>> at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>>
>>
Re: Storm nimbus election
Posted by Jungtaek Lim <ka...@gmail.com>.
As you suspected there seems to be ZK issue.
Unfortunately I don't have experience to operate Storm cluster with EC2.
You may want to check each Zookeeper instance is accessible from your Storm
cluster.
Thanks,
Jungtaek Lim (HeartSaVioR)
2016년 8월 28일 (일) 오후 5:27, pradeep s <sr...@gmail.com>님이 작성:
> Hi Juntaek,
> I am running Storm 1.0 on AWS .Figured that i have this issue when i have
> multiple zookeeper servers. I was using 3 zookeeper servers.
> When i make to only one zookeeper there is no issue found . Nimbus and UI
> works fine.
> Any reason for the nimbus failure when i have multiple zookeepers . Was
> seeing below logs in nimbus when i had 3 zookeepers.
> Do you have any recommendations on setting zookeeper cluster on ec2
>
> 016-08-26 23:46:47.822 o.a.s.d.common [INFO] Started statistics report
> plugin...
> 2016-08-26 23:46:47.854 o.a.s.d.nimbus [INFO] Starting nimbus server for
> storm version '1.0.0'
> 2016-08-27 00:07:37.520 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
> additional data from server sessionid 0x156c9158b980003, likely server has
> closed socket, closing socket connection and attempting reconnect
> 2016-08-27 00:07:37.520 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
> additional data from server sessionid 0x156c9158b980002, likely server has
> closed socket, closing socket connection and attempting reconnect
> 2016-08-27 00:07:37.621 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO]
> State change: SUSPENDED
> 2016-08-27 00:07:37.622 o.a.s.zookeeper [INFO] Zookeeper state update:
> :disconnected:none
> 2016-08-27 00:07:37.622 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO]
> State change: SUSPENDED
> 2016-08-27 00:07:37.623 o.a.s.c.zookeeper-state-factory [WARN] Received
> event :disconnected::none: with disconnected Writer Zookeeper.
> 2016-08-27 00:07:37.623 o.a.s.zookeeper [INFO]
> ip-xxxx21-38-235.cloud.nordstrom.net lost leadership.
> 2016-08-27 00:07:37.803 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
> connection to server xxxx21.50.34/xxxx21.50.34:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2016-08-27 00:07:37.804 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
> established to xxxx21.50.34/xxxx21.50.34:2181, initiating session
> 2016-08-27 00:07:37.805 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
> additional data from server sessionid 0x156c9158b980003, likely server has
> closed socket, closing socket connection and attempting reconnect
> 2016-08-27 00:07:38.152 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
> connection to server xxxx21.81.19/xxxx21.81.19:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2016-08-27 00:07:38.153 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
> established to xxxx21.81.19/xxxx21.81.19:2181, initiating session
> 2016-08-27 00:07:38.155 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
> additional data from server sessionid 0x156c9158b980002, likely server has
> closed socket, closing socket connection and attempting reconnect
> 2016-08-27 00:07:38.395 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
> connection to server xxxx21.81.19/xxxx21.81.19:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2016-08-27 00:07:38.396 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
> established to xxxx21.81.19/xxxx21.81.19:2181, initiating session
> 2016-08-27 00:07:38.398 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
> additional data from server sessionid 0x156c9158b980003, likely server has
> closed socket, closing socket connection and attempting reconnect
> 2016-08-27 00:07:38.979 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
> connection to server xxxx21.50.34/xxxx21.50.34:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2016-08-27 00:07:38.979 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
> established to xxxx21.50.34/xxxx21.50.34:2181, initiating session
> 2016-08-27 00:07:38.980 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
> additional data from server sessionid 0x156c9158b980002, likely server has
> closed socket, closing socket connection and attempting reconnect
> 2016-08-27 00:07:40.299 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
> connection to server xxxx21.106.227/xxxx21.106.227:2181. Will not attempt
> to authenticate using SASL (unknown error)
> 2016-08-27 00:07:40.300 o.a.s.s.o.a.z.ClientCnxn [WARN] Session
> 0x156c9158b980003 for server null, unexpected error, closing socket
> connection and attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> at
> org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> 2016-08-27 00:07:40.713 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
> connection to server xxxx21.106.227/xxxx21.106.227:2181. Will not attempt
> to authenticate using SASL (unknown error)
>
> Regards
> Pradeep S
>
> On Sat, Aug 27, 2016 at 2:37 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>
>> We need to see Nimbus log to find out why Nimbus is down. Pasting or
>> attaching Nimbus log would be better.
>> And please explain your Storm version too.
>>
>> Jungtaek Lim (HeartSaVioR)
>>
>> 2016년 8월 26일 (금) 오후 1:16, pradeep s <sr...@gmail.com>님이 작성:
>>
>>> Hi ,
>>> While restarting nimbus and ui, i am getting
>>> NimbusLeaderNotFoundException..
>>> While setting up the cluster also this error came. That time i have
>>> pointed zookeper data directory to a new directory and issue was resolved.
>>> Any idea on below exception.
>>>
>>> org.apache.storm.utils.NimbusLeaderNotFoundException: Found nimbuses [] none of which is elected as leader, please try again after some time.
>>> at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:85)
>>> at org.apache.storm.ui.core$cluster_summary.invoke(core.clj:356)
>>> at org.apache.storm.ui.core$fn__12108.invoke(core.clj:936)
>>> at org.apache.storm.shade.compojure.core$make_route$fn__2467.invoke(core.clj:93)
>>> at org.apache.storm.shade.compojure.core$if_route$fn__2455.invoke(core.clj:39)
>>> at org.apache.storm.shade.compojure.core$if_method$fn__2448.invoke(core.clj:24)
>>> at org.apache.storm.shade.compojure.core$routing$fn__2473.invoke(core.clj:106)
>>> at clojure.core$some.invoke(core.clj:2570)
>>> at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106)
>>> at clojure.lang.RestFn.applyTo(RestFn.java:139)
>>> at clojure.core$apply.invoke(core.clj:632)
>>> at org.apache.storm.shade.compojure.core$routes$fn__2477.invoke(core.clj:111)
>>> at org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__11576.invoke(json.clj:56)
>>> at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__3543.invoke(multipart_params.clj:103)
>>> at org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__4286.invoke(reload.clj:22)
>>> at org.apache.storm.ui.helpers$requests_middleware$fn__3770.invoke(helpers.clj:46)
>>> at org.apache.storm.ui.core$catch_errors$fn__12301.invoke(core.clj:1230)
>>> at org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__3474.invoke(keyword_params.clj:27)
>>> at org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__3514.invoke(nested_params.clj:65)
>>> at org.apache.storm.shade.ring.middleware.params$wrap_params$fn__3445.invoke(params.clj:55)
>>> at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__3543.invoke(multipart_params.clj:103)
>>> at org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__3729.invoke(flash.clj:14)
>>> at org.apache.storm.shade.ring.middleware.session$wrap_session$fn__3717.invoke(session.clj:43)
>>> at org.apache.storm.shade.ring.middleware.cookies$wrap_cookies$fn__3645.invoke(cookies.clj:160)
>>> at org.apache.storm.shade.ring.util.servlet$make_service_method$fn__3351.invoke(servlet.clj:127)
>>> at org.apache.storm.shade.ring.util.servlet$servlet$fn__3355.invoke(servlet.clj:136)
>>> at org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown Source)
>>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654)
>>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320)
>>> at org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47)
>>> at org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39)
>>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
>>> at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
>>> at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
>>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
>>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
>>> at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
>>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
>>> at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
>>> at org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>>> at org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>>> at org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
>>> at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
>>> at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
>>> at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
>>> at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
>>> at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>>> at org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>>> at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
>>> at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>>> at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>
Re: Storm nimbus election
Posted by pradeep s <sr...@gmail.com>.
Hi Juntaek,
I am running Storm 1.0 on AWS .Figured that i have this issue when i have
multiple zookeeper servers. I was using 3 zookeeper servers.
When i make to only one zookeeper there is no issue found . Nimbus and UI
works fine.
Any reason for the nimbus failure when i have multiple zookeepers . Was
seeing below logs in nimbus when i had 3 zookeepers.
Do you have any recommendations on setting zookeeper cluster on ec2
016-08-26 23:46:47.822 o.a.s.d.common [INFO] Started statistics report
plugin...
2016-08-26 23:46:47.854 o.a.s.d.nimbus [INFO] Starting nimbus server for
storm version '1.0.0'
2016-08-27 00:07:37.520 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
additional data from server sessionid 0x156c9158b980003, likely server has
closed socket, closing socket connection and attempting reconnect
2016-08-27 00:07:37.520 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
additional data from server sessionid 0x156c9158b980002, likely server has
closed socket, closing socket connection and attempting reconnect
2016-08-27 00:07:37.621 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO]
State change: SUSPENDED
2016-08-27 00:07:37.622 o.a.s.zookeeper [INFO] Zookeeper state update:
:disconnected:none
2016-08-27 00:07:37.622 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO]
State change: SUSPENDED
2016-08-27 00:07:37.623 o.a.s.c.zookeeper-state-factory [WARN] Received
event :disconnected::none: with disconnected Writer Zookeeper.
2016-08-27 00:07:37.623 o.a.s.zookeeper [INFO]
ip-xxxx21-38-235.cloud.nordstrom.net lost leadership.
2016-08-27 00:07:37.803 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xxxx21.50.34/xxxx21.50.34:2181. Will not attempt to
authenticate using SASL (unknown error)
2016-08-27 00:07:37.804 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xxxx21.50.34/xxxx21.50.34:2181, initiating session
2016-08-27 00:07:37.805 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
additional data from server sessionid 0x156c9158b980003, likely server has
closed socket, closing socket connection and attempting reconnect
2016-08-27 00:07:38.152 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xxxx21.81.19/xxxx21.81.19:2181. Will not attempt to
authenticate using SASL (unknown error)
2016-08-27 00:07:38.153 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xxxx21.81.19/xxxx21.81.19:2181, initiating session
2016-08-27 00:07:38.155 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
additional data from server sessionid 0x156c9158b980002, likely server has
closed socket, closing socket connection and attempting reconnect
2016-08-27 00:07:38.395 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xxxx21.81.19/xxxx21.81.19:2181. Will not attempt to
authenticate using SASL (unknown error)
2016-08-27 00:07:38.396 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xxxx21.81.19/xxxx21.81.19:2181, initiating session
2016-08-27 00:07:38.398 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
additional data from server sessionid 0x156c9158b980003, likely server has
closed socket, closing socket connection and attempting reconnect
2016-08-27 00:07:38.979 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xxxx21.50.34/xxxx21.50.34:2181. Will not attempt to
authenticate using SASL (unknown error)
2016-08-27 00:07:38.979 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xxxx21.50.34/xxxx21.50.34:2181, initiating session
2016-08-27 00:07:38.980 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
additional data from server sessionid 0x156c9158b980002, likely server has
closed socket, closing socket connection and attempting reconnect
2016-08-27 00:07:40.299 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xxxx21.106.227/xxxx21.106.227:2181. Will not attempt
to authenticate using SASL (unknown error)
2016-08-27 00:07:40.300 o.a.s.s.o.a.z.ClientCnxn [WARN] Session
0x156c9158b980003 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at
org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-27 00:07:40.713 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xxxx21.106.227/xxxx21.106.227:2181. Will not attempt
to authenticate using SASL (unknown error)
Regards
Pradeep S
On Sat, Aug 27, 2016 at 2:37 AM, Jungtaek Lim <ka...@gmail.com> wrote:
> We need to see Nimbus log to find out why Nimbus is down. Pasting or
> attaching Nimbus log would be better.
> And please explain your Storm version too.
>
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 8월 26일 (금) 오후 1:16, pradeep s <sr...@gmail.com>님이 작성:
>
>> Hi ,
>> While restarting nimbus and ui, i am getting
>> NimbusLeaderNotFoundException..
>> While setting up the cluster also this error came. That time i have
>> pointed zookeper data directory to a new directory and issue was resolved.
>> Any idea on below exception.
>>
>> org.apache.storm.utils.NimbusLeaderNotFoundException: Found nimbuses [] none of which is elected as leader, please try again after some time.
>> at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:85)
>> at org.apache.storm.ui.core$cluster_summary.invoke(core.clj:356)
>> at org.apache.storm.ui.core$fn__12108.invoke(core.clj:936)
>> at org.apache.storm.shade.compojure.core$make_route$fn__2467.invoke(core.clj:93)
>> at org.apache.storm.shade.compojure.core$if_route$fn__2455.invoke(core.clj:39)
>> at org.apache.storm.shade.compojure.core$if_method$fn__2448.invoke(core.clj:24)
>> at org.apache.storm.shade.compojure.core$routing$fn__2473.invoke(core.clj:106)
>> at clojure.core$some.invoke(core.clj:2570)
>> at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106)
>> at clojure.lang.RestFn.applyTo(RestFn.java:139)
>> at clojure.core$apply.invoke(core.clj:632)
>> at org.apache.storm.shade.compojure.core$routes$fn__2477.invoke(core.clj:111)
>> at org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__11576.invoke(json.clj:56)
>> at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__3543.invoke(multipart_params.clj:103)
>> at org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__4286.invoke(reload.clj:22)
>> at org.apache.storm.ui.helpers$requests_middleware$fn__3770.invoke(helpers.clj:46)
>> at org.apache.storm.ui.core$catch_errors$fn__12301.invoke(core.clj:1230)
>> at org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__3474.invoke(keyword_params.clj:27)
>> at org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__3514.invoke(nested_params.clj:65)
>> at org.apache.storm.shade.ring.middleware.params$wrap_params$fn__3445.invoke(params.clj:55)
>> at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__3543.invoke(multipart_params.clj:103)
>> at org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__3729.invoke(flash.clj:14)
>> at org.apache.storm.shade.ring.middleware.session$wrap_session$fn__3717.invoke(session.clj:43)
>> at org.apache.storm.shade.ring.middleware.cookies$wrap_cookies$fn__3645.invoke(cookies.clj:160)
>> at org.apache.storm.shade.ring.util.servlet$make_service_method$fn__3351.invoke(servlet.clj:127)
>> at org.apache.storm.shade.ring.util.servlet$servlet$fn__3355.invoke(servlet.clj:136)
>> at org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown Source)
>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654)
>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320)
>> at org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47)
>> at org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39)
>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
>> at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
>> at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
>> at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
>> at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
>> at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
>> at org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>> at org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>> at org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
>> at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
>> at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
>> at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
>> at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
>> at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>> at org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>> at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
>> at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>> at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>