You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Kristian Øllegaard <kr...@livesystems.info> on 2014/07/08 13:56:12 UTC
Storm hostnames and AWS
Hi everyone,
I’m currently setting up a storm cluster on AWS, but I have some problems with how storm supervisor connects to the peer nodes. On AWS I’m running storm in an auto scaling group, meaning that if one of the nodes go down, it automatically re-spawns another node with the same image. On AWS any node can disappear at any time, therefore it’s important to run a setup similar to this. All nodes have a hostname like ip-XX-XX-XX-XX representing their IP. However, they are not able to connect to the others hostnames, e.g. hostname ip-10-0-0-1 will not know about the hostname ip-10-0-0-2, although it could access 10.0.0.2. This is resulting in errors like this on the supervisors:
2014-07-08 11:43:05 b.s.d.executor [INFO] Loaded executor tasks count-bolt:[3 3]
2014-07-08 11:43:05 b.s.d.executor [INFO] Finished loading executor count-bolt:[3 3]
2014-07-08 11:43:05 b.s.d.executor [INFO] Loading executor word-spout:[5 5]
2014-07-08 11:43:05 b.s.d.executor [INFO] Loaded executor tasks word-spout:[5 5]
2014-07-08 11:43:05 b.s.d.executor [INFO] Preparing bolt count-bolt:(3)
2014-07-08 11:43:05 b.s.d.executor [INFO] Finished loading executor word-spout:[5 5]
2014-07-08 11:43:05 b.s.d.executor [INFO] Opening spout word-spout:(5)
2014-07-08 11:43:05 b.s.d.executor [INFO] Loading executor __system:[-1 -1]
2014-07-08 11:43:05 b.s.d.executor [INFO] Loaded executor tasks __system:[-1 -1]
2014-07-08 11:43:05 b.s.d.executor [INFO] Finished loading executor __system:[-1 -1]
2014-07-08 11:43:05 b.s.u.ShellProcess [INFO] Storm multilang serializer: backtype.storm.multilang.JsonSerializer
2014-07-08 11:43:05 b.s.u.ShellProcess [INFO] Storm multilang serializer: backtype.storm.multilang.JsonSerializer
2014-07-08 11:43:05 b.s.d.executor [INFO] Preparing bolt __system:(-1)
2014-07-08 11:43:05 b.s.m.n.Client [INFO] Reconnect started for Netty-Client-ip-10-0-11-36:6703... [4]
2014-07-08 11:43:05 b.s.m.n.StormClientErrorHandler [INFO] Connection failed Netty-Client-ip-10-0-11-36:6703
java.nio.channels.UnresolvedAddressException: null
at sun.nio.ch.Net.checkAddress(Net.java:127) ~[na:1.7.0_55]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:644) ~[na:1.7.0_55]
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:140) ~[netty-3.2.2.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:103) ~[netty-3.2.2.Final.jar:na]
at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:60) ~[netty-3.2.2.Final.jar:na]
at org.jboss.netty.channel.Channels.connect(Channels.java:541) [netty-3.2.2.Final.jar:na]
at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:218) [netty-3.2.2.Final.jar:na]
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:227) [netty-3.2.2.Final.jar:na]
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:188) [netty-3.2.2.Final.jar:na]
at backtype.storm.messaging.netty.Client.connect(Client.java:147) [storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.messaging.netty.Client.access$000(Client.java:42) [storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.messaging.netty.Client$1.run(Client.java:104) [storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_55]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_55]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_55]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) [na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55]
I’m a bit unsure how to proceed. Is it possible to form to register the IP instead of the hostname somehow? Or is there another way to do this?
Kind regards,
Kristian
Re: Storm hostnames and AWS
Posted by Spico Florin <sp...@gmail.com>.
Hello!
I'm interesting in this subject too. Can you please tell me if you solved
or not? If yes, can you share the solution?
Thanks.
Florin
On Tue, Jul 8, 2014 at 2:56 PM, Kristian Øllegaard <
kristian@livesystems.info> wrote:
> Hi everyone,
>
> I’m currently setting up a storm cluster on AWS, but I have some problems
> with how storm supervisor connects to the peer nodes. On AWS I’m running
> storm in an auto scaling group, meaning that if one of the nodes go down,
> it automatically re-spawns another node with the same image. On AWS any
> node can disappear at any time, therefore it’s important to run a setup
> similar to this. All nodes have a hostname like ip-XX-XX-XX-XX representing
> their IP. However, they are not able to connect to the others hostnames,
> e.g. hostname ip-10-0-0-1 will not know about the hostname ip-10-0-0-2,
> although it could access 10.0.0.2. This is resulting in errors like this on
> the supervisors:
>
> 2014-07-08 11:43:05 b.s.d.executor [INFO] Loaded executor tasks
> count-bolt:[3 3]
> 2014-07-08 11:43:05 b.s.d.executor [INFO] Finished loading executor
> count-bolt:[3 3]
> 2014-07-08 11:43:05 b.s.d.executor [INFO] Loading executor word-spout:[5 5]
> 2014-07-08 11:43:05 b.s.d.executor [INFO] Loaded executor tasks
> word-spout:[5 5]
> 2014-07-08 11:43:05 b.s.d.executor [INFO] Preparing bolt count-bolt:(3)
> 2014-07-08 11:43:05 b.s.d.executor [INFO] Finished loading executor
> word-spout:[5 5]
> 2014-07-08 11:43:05 b.s.d.executor [INFO] Opening spout word-spout:(5)
> 2014-07-08 11:43:05 b.s.d.executor [INFO] Loading executor __system:[-1 -1]
> 2014-07-08 11:43:05 b.s.d.executor [INFO] Loaded executor tasks
> __system:[-1 -1]
> 2014-07-08 11:43:05 b.s.d.executor [INFO] Finished loading executor
> __system:[-1 -1]
> 2014-07-08 11:43:05 b.s.u.ShellProcess [INFO] Storm multilang serializer:
> backtype.storm.multilang.JsonSerializer
> 2014-07-08 11:43:05 b.s.u.ShellProcess [INFO] Storm multilang serializer:
> backtype.storm.multilang.JsonSerializer
> 2014-07-08 11:43:05 b.s.d.executor [INFO] Preparing bolt __system:(-1)
> 2014-07-08 11:43:05 b.s.m.n.Client [INFO] Reconnect started for
> Netty-Client-ip-10-0-11-36:6703... [4]
> 2014-07-08 11:43:05 b.s.m.n.StormClientErrorHandler [INFO] Connection
> failed Netty-Client-ip-10-0-11-36:6703
> java.nio.channels.UnresolvedAddressException: null
> at sun.nio.ch.Net.checkAddress(Net.java:127) ~[na:1.7.0_55]
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:644)
> ~[na:1.7.0_55]
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:140)
> ~[netty-3.2.2.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:103)
> ~[netty-3.2.2.Final.jar:na]
> at
> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:60)
> ~[netty-3.2.2.Final.jar:na]
> at org.jboss.netty.channel.Channels.connect(Channels.java:541)
> [netty-3.2.2.Final.jar:na]
> at
> org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:218)
> [netty-3.2.2.Final.jar:na]
> at
> org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:227)
> [netty-3.2.2.Final.jar:na]
> at
> org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:188)
> [netty-3.2.2.Final.jar:na]
> at backtype.storm.messaging.netty.Client.connect(Client.java:147)
> [storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.messaging.netty.Client.access$000(Client.java:42)
> [storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.messaging.netty.Client$1.run(Client.java:104)
> [storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> [na:1.7.0_55]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_55]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> [na:1.7.0_55]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> [na:1.7.0_55]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_55]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_55]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55]
>
> I’m a bit unsure how to proceed. Is it possible to form to register the IP
> instead of the hostname somehow? Or is there another way to do this?
>
> Kind regards,
>
> Kristian
>