You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Spico Florin <sp...@gmail.com> on 2016/04/19 13:25:08 UTC
Re: Storm 1.0.0 DRPC connection refused
Hi!
I suggest to run your topology with LocalCluster and LocalDRPC.
For me it seems that is related with some security added in the 1.0.0. Can
you check some security configuration parameters that you have to set up in
your storm.yaml?
at
org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64)
~[storm-core-1.0.0.jar:1.0.0]
at
org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56)
~[storm-core-1.0.0.jar:1.0.0]
I hope that these help.
Regards,
Florin
On Tue, Apr 19, 2016 at 11:04 AM, Victor Kovrizhkin <
vik.kovrizhkin@gmail.com> wrote:
> Please help!
>
> From: Victor Kovrizhkin
> Date: Monday, April 18, 2016 at 9:28 PM
> To: "user@storm.apache.org"
> Subject: Storm 1.0.0 DRPC connection refused
>
> Hi Good People!
>
> I’m trying to update my cluster running Storm 0.10.0 with DRPC to Storm
> 1.0.0. I’ve updated all machines with latest version of Storm, changed
> storm.yml configurations (e.g. nimbus.host -> nimbus.seeds), changed
> dependencies in topology. When I starting nimbus, supervisors, ui, drpc,
> logviewers everything looks good, I’m able to see UI, logs, configuration
> and submit topologies.
>
> But once I submit DRPC topology – it’s logs full of following exceptions:
>
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> org.apache.storm.thrift.transport.TTransportException:
> java.net.ConnectException: Connection refused
>
> at org.apache.storm.drpc.DRPCSpout.checkFutures(DRPCSpout.java:129)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at org.apache.storm.drpc.DRPCSpout.nextTuple(DRPCSpout.java:210)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.daemon.executor$fn__8158$fn__8173$fn__8204.invoke(executor.clj:647)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
> [storm-core-1.0.0.jar:1.0.0]
>
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
>
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> org.apache.storm.thrift.transport.TTransportException:
> java.net.ConnectException: Connection refused
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> ~[?:1.8.0_74]
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_74]
>
> at org.apache.storm.drpc.DRPCSpout.checkFutures(DRPCSpout.java:127)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> ... 5 more
>
> Caused by: java.lang.RuntimeException:
> org.apache.storm.thrift.transport.TTransportException:
> java.net.ConnectException: Connection refused
>
> at
> org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:40)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:101)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:88)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_74]
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[?:1.8.0_74]
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[?:1.8.0_74]
>
> ... 1 more
>
> Caused by: org.apache.storm.thrift.transport.TTransportException:
> java.net.ConnectException: Connection refused
>
> at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:226)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:40)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:101)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:88)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_74]
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[?:1.8.0_74]
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[?:1.8.0_74]
>
> ... 1 more
>
> Caused by: java.net.ConnectException: Connection refused
>
> at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_74]
>
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> ~[?:1.8.0_74]
>
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204)
> ~[?:1.8.0_74]
>
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> ~[?:1.8.0_74]
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_74]
>
> at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_74]
>
> at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:221)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:40)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:101)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:88)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_74]
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[?:1.8.0_74]
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[?:1.8.0_74]
>
> ... 1 more
>
> 2016-04-18 18:10:19.579 o.a.s.util [ERROR] Async loop died!
>
> It seems DRPCSpout is unable to connect to DRPC server in some reasons.
>
> - Everything is fine with Storm 0.10.0
> - Supervisor’s configuration is correct – there is valid
> ‘drpc.servers’ entry.
> - Zookeeper and Nimbus are running
> - I’m able to telnet DRPC server 3772, 3774, 3773 ports from any
> supervisor.
> - DRPC server is up and running, it is able to handle request (with
> timeout of course), here is the log:
>
> 2016-04-18 18:00:20.444 o.a.s.d.drpc [INFO] Starting drpc server for storm
> version '1.0.0'
>
> 2016-04-18 18:00:20.619 o.a.s.d.drpc [INFO] Starting Distributed RPC
> servers...
>
> 2016-04-18 18:00:20.692 o.a.s.s.o.e.j.s.Server [INFO] jetty-7.x.y-SNAPSHOT
>
> 2016-04-18 18:00:20.717 o.a.s.s.o.e.j.s.h.ContextHandler [INFO] started
> o.a.s.s.o.e.j.s.ServletContextHandler{/,null}
>
> 2016-04-18 18:00:20.728 o.a.s.s.o.e.j.s.AbstractConnector [INFO] Started
> SelectChannelConnector@0.0.0.0:3774
>
> 2016-04-18 18:00:20.729 o.a.s.d.m.MetricsUtils [INFO] Using statistics
> reporter
> plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
>
> 2016-04-18 18:00:20.730 o.a.s.d.m.r.JmxPreparableReporter [INFO]
> Preparing...
>
> 2016-04-18 18:00:20.737 o.a.s.d.common [INFO] Started statistics report
> plugin...
>
> 2016-04-18 18:09:49.790 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer
> [ERROR] Read an invalid frame size of -786435. Are you using
> TFramedTransport on the client side?
>
> 2016-04-18 18:26:10.413 o.a.s.d.drpc [INFO] Starting drpc server for storm
> version '1.0.0'
>
> 2016-04-18 18:26:10.743 o.a.s.d.drpc [INFO] Starting Distributed RPC
> servers...
>
> 2016-04-18 18:26:10.852 o.a.s.s.o.e.j.s.Server [INFO] jetty-7.x.y-SNAPSHOT
>
> 2016-04-18 18:26:10.896 o.a.s.s.o.e.j.s.h.ContextHandler [INFO] started
> o.a.s.s.o.e.j.s.ServletContextHandler{/,null}
>
> 2016-04-18 18:26:10.921 o.a.s.s.o.e.j.s.AbstractConnector [INFO] Started
> SelectChannelConnector@0.0.0.0:3774
>
> 2016-04-18 18:26:10.922 o.a.s.d.m.MetricsUtils [INFO] Using statistics
> reporter
> plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
>
> 2016-04-18 18:26:10.924 o.a.s.d.m.r.JmxPreparableReporter [INFO]
> Preparing...
>
> 2016-04-18 18:26:10.935 o.a.s.d.common [INFO] Started statistics report
> plugin...
>
> Please help
> Sincerely,
> Viktor
>
Re: Storm 1.0.0 DRPC connection refused
Posted by Victor Kovrizhkin <vi...@gmail.com>.
Hi,
Thanks a lot for response!
I’ll check security settings, but I don’t configure anything specific in my storm.yml, so I guess default.yml entries are used.
My configuration is following:
storm.zookeeper.servers:
- {{ZOOKEEPER_HOST}}
storm.zookeeper.port: {{ZOOKEEPER_PORT}}
nimbus.seeds: [{{NIMBUS_HOST}}]
nimbus.thrift.port: {{NIMBUS_PORT}}
nimbus.childopts: "-Xms2g -Xmx2g -agentpath:/yjp/bin/linux-x86-64/libyjpagent.so=disableexceptiontelemetry,delay=10000"
storm.local.dir: "/var/run/storm”
supervisor.slots.ports:
- 6700
supervisor.childopts: "-Xmx2g"
worker.childopts: "-Xms8g -Xmx8g -agentpath:/yjp/bin/linux-x86-64/libyjpagent.so=disableexceptiontelemetry,delay=10000"
drpc.servers:
- {{DRPC_HOST}}
drpc.port: 3772
drpc.http.port: 3774
drpc.request.timeout.secs: 600
drpc.childopts: "-Xms2g -Xmx2g -agentpath:/yjp/bin/linux-x86-64/libyjpagent.so=disableexceptiontelemetry,delay=10000"
drpc.queue.size: 1024
drpc.worker.threads: 81
drpc.invocations.threads: 81
Thank you very much!
Victor
From: Spico Florin
Reply-To: <us...@storm.apache.org>
Date: Tuesday, April 19, 2016 at 2:25 PM
To: <us...@storm.apache.org>
Subject: Re: Storm 1.0.0 DRPC connection refused
Hi!
I suggest to run your topology with LocalCluster and LocalDRPC.
For me it seems that is related with some security added in the 1.0.0. Can you check some security configuration parameters that you have to set up in your storm.yaml?
at org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56) ~[storm-core-1.0.0.jar:1.0.0]
I hope that these help.
Regards,
Florin
On Tue, Apr 19, 2016 at 11:04 AM, Victor Kovrizhkin <vi...@gmail.com> wrote:
Please help!
From: Victor Kovrizhkin
Date: Monday, April 18, 2016 at 9:28 PM
To: "user@storm.apache.org"
Subject: Storm 1.0.0 DRPC connection refused
Hi Good People!
I’m trying to update my cluster running Storm 0.10.0 with DRPC to Storm 1.0.0. I’ve updated all machines with latest version of Storm, changed storm.yml configurations (e.g. nimbus.host -> nimbus.seeds), changed dependencies in topology. When I starting nimbus, supervisors, ui, drpc, logviewers everything looks good, I’m able to see UI, logs, configuration and submit topologies.
But once I submit DRPC topology – it’s logs full of following exceptions:
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.storm.drpc.DRPCSpout.checkFutures(DRPCSpout.java:129) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCSpout.nextTuple(DRPCSpout.java:210) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.daemon.executor$fn__8158$fn__8173$fn__8204.invoke(executor.clj:647) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) [storm-core-1.0.0.jar:1.0.0]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_74]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_74]
at org.apache.storm.drpc.DRPCSpout.checkFutures(DRPCSpout.java:127) ~[storm-core-1.0.0.jar:1.0.0]
... 5 more
Caused by: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:40) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:101) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:88) ~[storm-core-1.0.0.jar:1.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_74]
... 1 more
Caused by: org.apache.storm.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:226) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:40) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:101) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:88) ~[storm-core-1.0.0.jar:1.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_74]
... 1 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_74]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_74]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204) ~[?:1.8.0_74]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_74]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_74]
at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_74]
at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:221) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:40) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:101) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:88) ~[storm-core-1.0.0.jar:1.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_74]
... 1 more
2016-04-18 18:10:19.579 o.a.s.util [ERROR] Async loop died!
It seems DRPCSpout is unable to connect to DRPC server in some reasons.
Everything is fine with Storm 0.10.0
Supervisor’s configuration is correct – there is valid ‘drpc.servers’ entry.
Zookeeper and Nimbus are running
I’m able to telnet DRPC server 3772, 3774, 3773 ports from any supervisor.
DRPC server is up and running, it is able to handle request (with timeout of course), here is the log:
2016-04-18 18:00:20.444 o.a.s.d.drpc [INFO] Starting drpc server for storm version '1.0.0'
2016-04-18 18:00:20.619 o.a.s.d.drpc [INFO] Starting Distributed RPC servers...
2016-04-18 18:00:20.692 o.a.s.s.o.e.j.s.Server [INFO] jetty-7.x.y-SNAPSHOT
2016-04-18 18:00:20.717 o.a.s.s.o.e.j.s.h.ContextHandler [INFO] started o.a.s.s.o.e.j.s.ServletContextHandler{/,null}
2016-04-18 18:00:20.728 o.a.s.s.o.e.j.s.AbstractConnector [INFO] Started SelectChannelConnector@0.0.0.0:3774
2016-04-18 18:00:20.729 o.a.s.d.m.MetricsUtils [INFO] Using statistics reporter plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
2016-04-18 18:00:20.730 o.a.s.d.m.r.JmxPreparableReporter [INFO] Preparing...
2016-04-18 18:00:20.737 o.a.s.d.common [INFO] Started statistics report plugin...
2016-04-18 18:09:49.790 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer [ERROR] Read an invalid frame size of -786435. Are you using TFramedTransport on the client side?
2016-04-18 18:26:10.413 o.a.s.d.drpc [INFO] Starting drpc server for storm version '1.0.0'
2016-04-18 18:26:10.743 o.a.s.d.drpc [INFO] Starting Distributed RPC servers...
2016-04-18 18:26:10.852 o.a.s.s.o.e.j.s.Server [INFO] jetty-7.x.y-SNAPSHOT
2016-04-18 18:26:10.896 o.a.s.s.o.e.j.s.h.ContextHandler [INFO] started o.a.s.s.o.e.j.s.ServletContextHandler{/,null}
2016-04-18 18:26:10.921 o.a.s.s.o.e.j.s.AbstractConnector [INFO] Started SelectChannelConnector@0.0.0.0:3774
2016-04-18 18:26:10.922 o.a.s.d.m.MetricsUtils [INFO] Using statistics reporter plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
2016-04-18 18:26:10.924 o.a.s.d.m.r.JmxPreparableReporter [INFO] Preparing...
2016-04-18 18:26:10.935 o.a.s.d.common [INFO] Started statistics report plugin...
Please help
Sincerely,
Viktor
Re: Storm 1.0.0 DRPC connection refused
Posted by Spico Florin <sp...@gmail.com>.
Hello!
I found also a post with similliar error that you have. Perhaps you get
some clues.
http://mail-archives.apache.org/mod_mbox/storm-user/201603.mbox/%3C0DD9AA99-8504-43C9-B3A8-6196DEF07877@viaplay.com%3E
On Tue, Apr 19, 2016 at 2:25 PM, Spico Florin <sp...@gmail.com> wrote:
> Hi!
> I suggest to run your topology with LocalCluster and LocalDRPC.
> For me it seems that is related with some security added in the 1.0.0. Can
> you check some security configuration parameters that you have to set up in
> your storm.yaml?
>
> at
> org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64)
> ~[storm-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56)
> ~[storm-core-1.0.0.jar:1.0.0]
>
>
>
> I hope that these help.
>
> Regards,
>
> Florin
>
>
> On Tue, Apr 19, 2016 at 11:04 AM, Victor Kovrizhkin <
> vik.kovrizhkin@gmail.com> wrote:
>
>> Please help!
>>
>> From: Victor Kovrizhkin
>> Date: Monday, April 18, 2016 at 9:28 PM
>> To: "user@storm.apache.org"
>> Subject: Storm 1.0.0 DRPC connection refused
>>
>> Hi Good People!
>>
>> I’m trying to update my cluster running Storm 0.10.0 with DRPC to Storm
>> 1.0.0. I’ve updated all machines with latest version of Storm, changed
>> storm.yml configurations (e.g. nimbus.host -> nimbus.seeds), changed
>> dependencies in topology. When I starting nimbus, supervisors, ui, drpc,
>> logviewers everything looks good, I’m able to see UI, logs, configuration
>> and submit topologies.
>>
>> But once I submit DRPC topology – it’s logs full of following exceptions:
>>
>> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
>> java.lang.RuntimeException:
>> org.apache.storm.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused
>>
>> at org.apache.storm.drpc.DRPCSpout.checkFutures(DRPCSpout.java:129)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at org.apache.storm.drpc.DRPCSpout.nextTuple(DRPCSpout.java:210)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.daemon.executor$fn__8158$fn__8173$fn__8204.invoke(executor.clj:647)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>> [storm-core-1.0.0.jar:1.0.0]
>>
>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
>>
>> Caused by: java.util.concurrent.ExecutionException:
>> java.lang.RuntimeException:
>> org.apache.storm.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused
>>
>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> ~[?:1.8.0_74]
>>
>> at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_74]
>>
>> at org.apache.storm.drpc.DRPCSpout.checkFutures(DRPCSpout.java:127)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> ... 5 more
>>
>> Caused by: java.lang.RuntimeException:
>> org.apache.storm.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused
>>
>> at
>> org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:40)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:101)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:88)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_74]
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> ~[?:1.8.0_74]
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> ~[?:1.8.0_74]
>>
>> ... 1 more
>>
>> Caused by: org.apache.storm.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused
>>
>> at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:226)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:40)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:101)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:88)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_74]
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> ~[?:1.8.0_74]
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> ~[?:1.8.0_74]
>>
>> ... 1 more
>>
>> Caused by: java.net.ConnectException: Connection refused
>>
>> at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_74]
>>
>> at
>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>> ~[?:1.8.0_74]
>>
>> at
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204)
>> ~[?:1.8.0_74]
>>
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>> ~[?:1.8.0_74]
>>
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> ~[?:1.8.0_74]
>>
>> at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_74]
>>
>> at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:221)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at
>> org.apache.storm.drpc.DRPCInvocationsClient.<init>(DRPCInvocationsClient.java:40)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:101)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at org.apache.storm.drpc.DRPCSpout$Adder.call(DRPCSpout.java:88)
>> ~[storm-core-1.0.0.jar:1.0.0]
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_74]
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> ~[?:1.8.0_74]
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> ~[?:1.8.0_74]
>>
>> ... 1 more
>>
>> 2016-04-18 18:10:19.579 o.a.s.util [ERROR] Async loop died!
>>
>> It seems DRPCSpout is unable to connect to DRPC server in some reasons.
>>
>> - Everything is fine with Storm 0.10.0
>> - Supervisor’s configuration is correct – there is valid
>> ‘drpc.servers’ entry.
>> - Zookeeper and Nimbus are running
>> - I’m able to telnet DRPC server 3772, 3774, 3773 ports from any
>> supervisor.
>> - DRPC server is up and running, it is able to handle request (with
>> timeout of course), here is the log:
>>
>> 2016-04-18 18:00:20.444 o.a.s.d.drpc [INFO] Starting drpc server for
>> storm version '1.0.0'
>>
>> 2016-04-18 18:00:20.619 o.a.s.d.drpc [INFO] Starting Distributed RPC
>> servers...
>>
>> 2016-04-18 18:00:20.692 o.a.s.s.o.e.j.s.Server [INFO] jetty-7.x.y-SNAPSHOT
>>
>> 2016-04-18 18:00:20.717 o.a.s.s.o.e.j.s.h.ContextHandler [INFO] started
>> o.a.s.s.o.e.j.s.ServletContextHandler{/,null}
>>
>> 2016-04-18 18:00:20.728 o.a.s.s.o.e.j.s.AbstractConnector [INFO] Started
>> SelectChannelConnector@0.0.0.0:3774
>>
>> 2016-04-18 18:00:20.729 o.a.s.d.m.MetricsUtils [INFO] Using statistics
>> reporter
>> plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
>>
>> 2016-04-18 18:00:20.730 o.a.s.d.m.r.JmxPreparableReporter [INFO]
>> Preparing...
>>
>> 2016-04-18 18:00:20.737 o.a.s.d.common [INFO] Started statistics report
>> plugin...
>>
>> 2016-04-18 18:09:49.790 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer
>> [ERROR] Read an invalid frame size of -786435. Are you using
>> TFramedTransport on the client side?
>>
>> 2016-04-18 18:26:10.413 o.a.s.d.drpc [INFO] Starting drpc server for
>> storm version '1.0.0'
>>
>> 2016-04-18 18:26:10.743 o.a.s.d.drpc [INFO] Starting Distributed RPC
>> servers...
>>
>> 2016-04-18 18:26:10.852 o.a.s.s.o.e.j.s.Server [INFO] jetty-7.x.y-SNAPSHOT
>>
>> 2016-04-18 18:26:10.896 o.a.s.s.o.e.j.s.h.ContextHandler [INFO] started
>> o.a.s.s.o.e.j.s.ServletContextHandler{/,null}
>>
>> 2016-04-18 18:26:10.921 o.a.s.s.o.e.j.s.AbstractConnector [INFO] Started
>> SelectChannelConnector@0.0.0.0:3774
>>
>> 2016-04-18 18:26:10.922 o.a.s.d.m.MetricsUtils [INFO] Using statistics
>> reporter
>> plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
>>
>> 2016-04-18 18:26:10.924 o.a.s.d.m.r.JmxPreparableReporter [INFO]
>> Preparing...
>>
>> 2016-04-18 18:26:10.935 o.a.s.d.common [INFO] Started statistics report
>> plugin...
>>
>> Please help
>> Sincerely,
>> Viktor
>>
>
>