You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Ryan Ebanks <ry...@gmail.com> on 2014/10/15 21:59:17 UTC
Supervisors fail to start up
I have been running a storm-0.8.2 cluster for over a year now. Last night
AWS restarted the supervisor machines. I have tried to restart the
supervisor processes manually, but upon start up I receive this error
message in the logs.
014-10-15 19:48:04 supervisor [ERROR] Error on initialization of server
mk-supervisor
java.net.UnknownHostException: domU-12-31-39-09-D2-92:
domU-12-31-39-09-D2-92
at java.net.InetAddress.getLocalHost(InetAddress.java:1454)
at backtype.storm.util$local_hostname.invoke(util.clj:153)
at
backtype.storm.daemon.supervisor$supervisor_data.invoke(supervisor.clj:181)
at
backtype.storm.daemon.supervisor$fn__4729$exec_fn__1200__auto____4730.invoke(supervisor.clj:331)
at clojure.lang.AFn.applyToHelper(AFn.java:167)
at clojure.lang.AFn.applyTo(AFn.java:151)
at clojure.core$apply.invoke(core.clj:601)
at
backtype.storm.daemon.supervisor$fn__4729$mk_supervisor__4754.doInvoke(supervisor.clj:327)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at backtype.storm.daemon.supervisor$_launch.invoke(supervisor.clj:477)
at backtype.storm.daemon.supervisor$_main.invoke(supervisor.clj:506)
at clojure.lang.AFn.applyToHelper(AFn.java:159)
at clojure.lang.AFn.applyTo(AFn.java:151)
at backtype.storm.daemon.supervisor.main(Unknown Source)
I am not a clojure expert, but it looks like on line 215 of
backtype.storm.daemon.supervisor.clj, that it is possible to set the
localhost name in a config file.
215 :my-hostname (if (contains? conf STORM-LOCAL-HOSTNAME)
216 (conf STORM-LOCAL-HOSTNAME)
217 (local-hostname))
Is this possible? What file do I need to set this setting in? What is the
correct key for this setting?
Or am I way off base and need to do something else to get my workers to
restart?
Thanks,
Ryan Ebanks
Re: Supervisors fail to start up
Posted by Ryan Ebanks <ry...@gmail.com>.
Thanks,
Setting the "storm.local.hostname" key in the storm.yaml file did fix the
issue.
@itai, I did have the same issue with my zookeeper cluster but I had
already fixed that by setting the hosts in the zoo.cfg file. I just did
know what the correct key for the supervisors were. Thank you for the help.
-Ryan
On Thu, Oct 16, 2014 at 2:14 AM, Itai Frenkel <It...@forter.com> wrote:
> Hi Vikas,
>
>
> (I'm newbie)
>
>
> IMO you should inject "storm.local.hostname:" in conf/storm.yaml in the
> script that starts supervisor.
>
> Make sure you inject the FULL name given by "hostname -f" since on ec2 the
> shorter name is not guaranteed to work (and Java by default gets the short
> name). We had the same issue with the zookeeper cluster. Since you are
> probably running a zookeeper cluster, peek at that devops code, a similar
> solution is probably there.
>
>
> Regards,
>
> Itai
> ------------------------------
> *From:* Vikas Agarwal <vi...@infoobjects.com>
> *Sent:* Thursday, October 16, 2014 8:04 AM
> *To:* user@storm.apache.org
> *Subject:* Re: Supervisors fail to start up
>
> Are you using EC2 private IPs anywhere in your configuration? EC2
> internal IP changes after restart.
>
> On Thu, Oct 16, 2014 at 1:29 AM, Ryan Ebanks <ry...@gmail.com> wrote:
>
>> I have been running a storm-0.8.2 cluster for over a year now. Last
>> night AWS restarted the supervisor machines. I have tried to restart the
>> supervisor processes manually, but upon start up I receive this error
>> message in the logs.
>>
>> 014-10-15 19:48:04 supervisor [ERROR] Error on initialization of server
>> mk-supervisor
>>
>> java.net.UnknownHostException: domU-12-31-39-09-D2-92:
>> domU-12-31-39-09-D2-92
>>
>> at java.net.InetAddress.getLocalHost(InetAddress.java:1454)
>>
>> at backtype.storm.util$local_hostname.invoke(util.clj:153)
>>
>> at
>> backtype.storm.daemon.supervisor$supervisor_data.invoke(supervisor.clj:181)
>>
>> at
>> backtype.storm.daemon.supervisor$fn__4729$exec_fn__1200__auto____4730.invoke(supervisor.clj:331)
>>
>> at clojure.lang.AFn.applyToHelper(AFn.java:167)
>>
>> at clojure.lang.AFn.applyTo(AFn.java:151)
>>
>> at clojure.core$apply.invoke(core.clj:601)
>>
>> at
>> backtype.storm.daemon.supervisor$fn__4729$mk_supervisor__4754.doInvoke(supervisor.clj:327)
>>
>> at clojure.lang.RestFn.invoke(RestFn.java:436)
>>
>> at backtype.storm.daemon.supervisor$_launch.invoke(supervisor.clj:477)
>>
>> at backtype.storm.daemon.supervisor$_main.invoke(supervisor.clj:506)
>>
>> at clojure.lang.AFn.applyToHelper(AFn.java:159)
>>
>> at clojure.lang.AFn.applyTo(AFn.java:151)
>>
>> at backtype.storm.daemon.supervisor.main(Unknown Source)
>>
>>
>> I am not a clojure expert, but it looks like on line 215 of
>> backtype.storm.daemon.supervisor.clj, that it is possible to set the
>> localhost name in a config file.
>>
>> 215 :my-hostname (if (contains? conf STORM-LOCAL-HOSTNAME)
>> 216 (conf STORM-LOCAL-HOSTNAME)
>> 217 (local-hostname))
>>
>>
>> Is this possible? What file do I need to set this setting in? What is
>> the correct key for this setting?
>>
>> Or am I way off base and need to do something else to get my workers to
>> restart?
>>
>> Thanks,
>>
>> Ryan Ebanks
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
>
Re: Supervisors fail to start up
Posted by Itai Frenkel <It...@forter.com>.
Hi Vikas,
(I'm newbie)
IMO you should inject "storm.local.hostname:" in conf/storm.yaml in the script that starts supervisor.
Make sure you inject the FULL name given by "hostname -f" since on ec2 the shorter name is not guaranteed to work (and Java by default gets the short name). We had the same issue with the zookeeper cluster. Since you are probably running a zookeeper cluster, peek at that devops code, a similar solution is probably there.
Regards,
Itai
________________________________
From: Vikas Agarwal <vi...@infoobjects.com>
Sent: Thursday, October 16, 2014 8:04 AM
To: user@storm.apache.org
Subject: Re: Supervisors fail to start up
Are you using EC2 private IPs anywhere in your configuration? EC2 internal IP changes after restart.
On Thu, Oct 16, 2014 at 1:29 AM, Ryan Ebanks <ry...@gmail.com>> wrote:
I have been running a storm-0.8.2 cluster for over a year now. Last night AWS restarted the supervisor machines. I have tried to restart the supervisor processes manually, but upon start up I receive this error message in the logs.
014-10-15 19:48:04 supervisor [ERROR] Error on initialization of server mk-supervisor
java.net.UnknownHostException: domU-12-31-39-09-D2-92: domU-12-31-39-09-D2-92
at java.net.InetAddress.getLocalHost(InetAddress.java:1454)
at backtype.storm.util$local_hostname.invoke(util.clj:153)
at backtype.storm.daemon.supervisor$supervisor_data.invoke(supervisor.clj:181)
at backtype.storm.daemon.supervisor$fn__4729$exec_fn__1200__auto____4730.invoke(supervisor.clj:331)
at clojure.lang.AFn.applyToHelper(AFn.java:167)
at clojure.lang.AFn.applyTo(AFn.java:151)
at clojure.core$apply.invoke(core.clj:601)
at backtype.storm.daemon.supervisor$fn__4729$mk_supervisor__4754.doInvoke(supervisor.clj:327)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at backtype.storm.daemon.supervisor$_launch.invoke(supervisor.clj:477)
at backtype.storm.daemon.supervisor$_main.invoke(supervisor.clj:506)
at clojure.lang.AFn.applyToHelper(AFn.java:159)
at clojure.lang.AFn.applyTo(AFn.java:151)
at backtype.storm.daemon.supervisor.main(Unknown Source)
I am not a clojure expert, but it looks like on line 215 of backtype.storm.daemon.supervisor.clj, that it is possible to set the localhost name in a config file.
215 :my-hostname (if (contains? conf STORM-LOCAL-HOSTNAME)
216 (conf STORM-LOCAL-HOSTNAME)
217 (local-hostname))
Is this possible? What file do I need to set this setting in? What is the correct key for this setting?
Or am I way off base and need to do something else to get my workers to restart?
Thanks,
Ryan Ebanks
--
Regards,
Vikas Agarwal
91 - 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Re: Supervisors fail to start up
Posted by Vikas Agarwal <vi...@infoobjects.com>.
Are you using EC2 private IPs anywhere in your configuration? EC2 internal
IP changes after restart.
On Thu, Oct 16, 2014 at 1:29 AM, Ryan Ebanks <ry...@gmail.com> wrote:
> I have been running a storm-0.8.2 cluster for over a year now. Last night
> AWS restarted the supervisor machines. I have tried to restart the
> supervisor processes manually, but upon start up I receive this error
> message in the logs.
>
> 014-10-15 19:48:04 supervisor [ERROR] Error on initialization of server
> mk-supervisor
>
> java.net.UnknownHostException: domU-12-31-39-09-D2-92:
> domU-12-31-39-09-D2-92
>
> at java.net.InetAddress.getLocalHost(InetAddress.java:1454)
>
> at backtype.storm.util$local_hostname.invoke(util.clj:153)
>
> at
> backtype.storm.daemon.supervisor$supervisor_data.invoke(supervisor.clj:181)
>
> at
> backtype.storm.daemon.supervisor$fn__4729$exec_fn__1200__auto____4730.invoke(supervisor.clj:331)
>
> at clojure.lang.AFn.applyToHelper(AFn.java:167)
>
> at clojure.lang.AFn.applyTo(AFn.java:151)
>
> at clojure.core$apply.invoke(core.clj:601)
>
> at
> backtype.storm.daemon.supervisor$fn__4729$mk_supervisor__4754.doInvoke(supervisor.clj:327)
>
> at clojure.lang.RestFn.invoke(RestFn.java:436)
>
> at backtype.storm.daemon.supervisor$_launch.invoke(supervisor.clj:477)
>
> at backtype.storm.daemon.supervisor$_main.invoke(supervisor.clj:506)
>
> at clojure.lang.AFn.applyToHelper(AFn.java:159)
>
> at clojure.lang.AFn.applyTo(AFn.java:151)
>
> at backtype.storm.daemon.supervisor.main(Unknown Source)
>
>
> I am not a clojure expert, but it looks like on line 215 of
> backtype.storm.daemon.supervisor.clj, that it is possible to set the
> localhost name in a config file.
>
> 215 :my-hostname (if (contains? conf STORM-LOCAL-HOSTNAME)
> 216 (conf STORM-LOCAL-HOSTNAME)
> 217 (local-hostname))
>
>
> Is this possible? What file do I need to set this setting in? What is
> the correct key for this setting?
>
> Or am I way off base and need to do something else to get my workers to
> restart?
>
> Thanks,
>
> Ryan Ebanks
>
>
>
>
>
>
>
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax