You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Justin Workman <ju...@gmail.com> on 2015/03/25 20:53:44 UTC
Supervisors flapping after killing topology
We are currently running 7 node storm cluster, 1 nimbus and 6 supervisor
nodes all running storm 0.9.2, running 3 topologies. Any time we kill a
running topology the supervisors across all nodes start flapping and we end
up in a mess. To clean this up we end up killing all running topologies,
shutdown the supervisors, cleanup the storm/storm-local directories on all
supervisor nodes, restart the supervisor processes then restart the
topologies.
Has anyone experienced this issues, or have any ideas on how to resolve it.
Log snippet we see in the supervisor logs when this happens...
2015-03-25 11:28:13 b.s.d.supervisor [INFO] Shutting down
4d971d4b-a208-4758-a55e-3e8b34d7531f:ce049d5c-fd4c-499c-ad8d-ef1d8f2b992b
2015-03-25 11:28:13 b.s.event [ERROR] Error when processing event
java.io.IOException: . doesn't exist.
at
org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:157)
~[commons-exec-1.1.jar:1.1]
at
org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147)
~[commons-exec-1.1.jar:1.1]
at backtype.storm.util$exec_command_BANG_.invoke(util.clj:378)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.util$ensure_process_killed_BANG_.invoke(util.clj:394)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.supervisor$shutdown_worker.invoke(supervisor.clj:175)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:240)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.applyToHelper(AFn.java:161)
~[clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
at clojure.core$partial$fn__4190.doInvoke(core.clj:2396)
~[clojure-1.5.1.jar:na]
at clojure.lang.RestFn.invoke(RestFn.java:397)
~[clojure-1.5.1.jar:na]
at backtype.storm.event$event_manager$fn__2378.invoke(event.clj:39)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
2015-03-25 11:28:13 b.s.util [INFO] Halting process: ("Error when
processing an event")
There does not be any thing corresponding to this in the worker logs.'
Ideas??
Thanks
Justin
Re: Supervisors flapping after killing topology
Posted by Justin Workman <ju...@gmail.com>.
This happens in all our environments (dev,test,stage and prod) every time we kill even one topology. This did not happen with the previous version of storm.
It almost seems as though when we shutdown a topology, the storm-local working directory is cleaned up, however the topology information is not cleaned up from zookeeper. So when the supervisor checks to see what it should be running, it finds a stale topology in zookeeper and not a corresponding worker directory on the file system, hence the Java io error for file not find. This then sends the supervisor into a restart loop. Shutting everything down and restarting seems to clean things up.
I don't have hard evidence to support that theory, but the pattern seems to point in that direction
Thanks
Justin
Sent from my iPhone
> On Mar 26, 2015, at 9:36 AM, Bobby Evans <ev...@yahoo-inc.com> wrote:
>
> I'm not sure if it is a bug in storm, or if your system is somehow messed up. I tracked down the exception being thrown and it indicates that someone deleted the current working directory that you are running the supervisor in. I'm not sure how that happens, but I have never seen it happen on any of my clusters, which leads me to believe that it is something that happened with your system, and not storm itself.
>
> - Bobby
>
>
>
> On Wednesday, March 25, 2015 4:54 PM, Justin Workman <ju...@gmail.com> wrote:
>
>
> Sounds like your storm-local is pointed to tmp and you are going through the same process as us.
>
> Is this a bug, or is there a better solution?
>
> Sent from my iPhone
>
>> On Mar 25, 2015, at 2:41 PM, Andres Gomez Ferrer <ag...@redborder.net> wrote:
>>
>> I stop all my topologies, stop all storm nodes, remove /tmp/storm, start all again!
>>
>> Andrés Gómez
>> Developer
>> redborder.net / agomez@redborder.net
>> Phone: +34 955 60 11 60
>>
>> <0e6e8de_1.png>
>>
>> <square-twitter-20.png> <square-google-plus-20.png> <square-linkedin-20.png>
>>
>>
>> Piénsalo antes de imprimir este mensaje
>>
>> Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción.
>>
>> This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. If you have received this email in error, please notify the sender and delete it from your system.
>>
>>
>> En 25 de marzo de 2015 en 21:31:50, Justin Workman (justinjworkman@gmail.com) escrito:
>
> During each topology kill/restart? Where do you put that hook?
>
> Or was it a one time remove of /tmp/storm and future topology restarts were fine.
>
> Sent from my iPhone
>
> On Mar 25, 2015, at 1:59 PM, Andres Gomez Ferrer <ag...@redborder.net> wrote:
>
>> I solved it removing all /tmp/storm/*
>>
>> Regards,
>>
>> Andrés Gómez
>> Developer
>> redborder.net / agomez@redborder.net
>> Phone: +34 955 60 11 60
>>
>> <0e6e8de_1.png>
>>
>> <square-twitter-20.png> <square-google-plus-20.png> <square-linkedin-20.png>
>>
>> Piénsalo antes de imprimir este mensaje
>>
>> Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción.
>>
>> This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. If you have received this email in error, please notify the sender and delete it from your system.
>>
>>
>> En 25 de marzo de 2015 en 20:54:58, Justin Workman (justinjworkman@gmail.com) escrito:
>>> We are currently running 7 node storm cluster, 1 nimbus and 6 supervisor nodes all running storm 0.9.2, running 3 topologies. Any time we kill a running topology the supervisors across all nodes start flapping and we end up in a mess. To clean this up we end up killing all running topologies, shutdown the supervisors, cleanup the storm/storm-local directories on all supervisor nodes, restart the supervisor processes then restart the topologies.
>>>
>>> Has anyone experienced this issues, or have any ideas on how to resolve it.
>>>
>>> Log snippet we see in the supervisor logs when this happens...
>>>
>>> 2015-03-25 11:28:13 b.s.d.supervisor [INFO] Shutting down 4d971d4b-a208-4758-a55e-3e8b34d7531f:ce049d5c-fd4c-499c-ad8d-ef1d8f2b992b
>>> 2015-03-25 11:28:13 b.s.event [ERROR] Error when processing event
>>> java.io.IOException: . doesn't exist.
>>> at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:157) ~[commons-exec-1.1.jar:1.1]
>>> at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147) ~[commons-exec-1.1.jar:1.1]
>>> at backtype.storm.util$exec_command_BANG_.invoke(util.clj:378) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at backtype.storm.util$ensure_process_killed_BANG_.invoke(util.clj:394) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at backtype.storm.daemon.supervisor$shutdown_worker.invoke(supervisor.clj:175) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:240) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at clojure.lang.AFn.applyToHelper(AFn.java:161) ~[clojure-1.5.1.jar:na]
>>> at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
>>> at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
>>> at clojure.core$partial$fn__4190.doInvoke(core.clj:2396) ~[clojure-1.5.1.jar:na]
>>> at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.5.1.jar:na]
>>> at backtype.storm.event$event_manager$fn__2378.invoke(event.clj:39) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>> at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
>>> at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
>>> 2015-03-25 11:28:13 b.s.util [INFO] Halting process: ("Error when processing an event")
>>>
>>>
>>> There does not be any thing corresponding to this in the worker logs.'
>>>
>>> Ideas??
>>>
>>> Thanks
>>> Justin
>
>
Re: Supervisors flapping after killing topology
Posted by Bobby Evans <ev...@yahoo-inc.com>.
I'm not sure if it is a bug in storm, or if your system is somehow messed up. I tracked down the exception being thrown and it indicates that someone deleted the current working directory that you are running the supervisor in. I'm not sure how that happens, but I have never seen it happen on any of my clusters, which leads me to believe that it is something that happened with your system, and not storm itself.
- Bobby
On Wednesday, March 25, 2015 4:54 PM, Justin Workman <ju...@gmail.com> wrote:
Sounds like your storm-local is pointed to tmp and you are going through the same process as us.
Is this a bug, or is there a better solution?
Sent from my iPhone
On Mar 25, 2015, at 2:41 PM, Andres Gomez Ferrer <ag...@redborder.net> wrote:
#yiv0938392330 body{font-family:Helvetica, Arial;font-size:13px;}I stop all my topologies, stop all storm nodes, remove /tmp/storm, start all again!
Andrés Gómez
Developer redborder.net / agomez@redborder.netPhone: +34 955 60 11 60
<0e6e8de_1.png>
<square-twitter-20.png> <square-google-plus-20.png> <square-linkedin-20.png>
Piénsalo antes de imprimir este mensaje Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción. This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. If you have received this email in error, please notify the sender and delete it from your system.
En 25 de marzo de 2015 en 21:31:50, Justin Workman (justinjworkman@gmail.com) escrito:
During each topology kill/restart? Where do you put thathook?
Or was it a one time remove of /tmp/storm and futuretopology restarts were fine.
Sent from my iPhone
On Mar 25, 2015, at 1:59 PM, Andres Gomez Ferrer <ag...@redborder.net>wrote:
I solved it removing all /tmp/storm/*
Regards,
Andrés Gómez
Developer redborder.net / agomez@redborder.netPhone: +34955 60 11 60
<0e6e8de_1.png>
<square-twitter-20.png> <square-google-plus-20.png> <square-linkedin-20.png>
Piénsalo antes deimprimir este mensaje Este correoelectrónico, incluidos sus anexos, se dirige exclusivamente a sudestinatario. Contiene información CONFIDENCIAL cuya divulgaciónestá prohibida por la ley o puede estar sometida a secretoprofesional. Si ha recibido este mensaje por error, le rogamos noslo comunique inmediatamente y proceda a sudestrucción. This email, includingattachments, is intended exclusively for its addressee. It containsinformation that is CONFIDENTIAL whose disclosure is prohibited bylaw and may be covered by legal privilege. If you havereceived this email in error, please notify the sender and deleteit from your system.
En 25 de marzo de 2015 en 20:54:58, JustinWorkman (justinjworkman@gmail.com)escrito:
We are currently running 7 node storm cluster,1 nimbus and 6 supervisor nodes all running storm 0.9.2, running 3topologies. Any time we kill a running topology the supervisorsacross all nodes start flapping and we end up in a mess. To cleanthis up we end up killing all running topologies, shutdown thesupervisors, cleanup the storm/storm-local directories on allsupervisor nodes, restart the supervisor processes then restart thetopologies.
Has anyone experienced this issues, or have any ideas onhow to resolve it.
Log snippet we see in the supervisor logs when thishappens...
2015-03-25 11:28:13 b.s.d.supervisor [INFO] Shuttingdown4d971d4b-a208-4758-a55e-3e8b34d7531f:ce049d5c-fd4c-499c-ad8d-ef1d8f2b992b2015-03-25 11:28:13 b.s.event [ERROR] Error whenprocessing eventjava.io.IOException: . doesn't exist. atorg.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:157)~[commons-exec-1.1.jar:1.1] atorg.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147)~[commons-exec-1.1.jar:1.1] atbacktype.storm.util$exec_command_BANG_.invoke(util.clj:378)~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] atbacktype.storm.util$ensure_process_killed_BANG_.invoke(util.clj:394)~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] atbacktype.storm.daemon.supervisor$shutdown_worker.invoke(supervisor.clj:175)~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] atbacktype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:240)~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] atclojure.lang.AFn.applyToHelper(AFn.java:161)~[clojure-1.5.1.jar:na] atclojure.lang.AFn.applyTo(AFn.java:151)~[clojure-1.5.1.jar:na] atclojure.core$apply.invoke(core.clj:619)~[clojure-1.5.1.jar:na] atclojure.core$partial$fn__4190.doInvoke(core.clj:2396)~[clojure-1.5.1.jar:na] atclojure.lang.RestFn.invoke(RestFn.java:397)~[clojure-1.5.1.jar:na] atbacktype.storm.event$event_manager$fn__2378.invoke(event.clj:39)~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] atclojure.lang.AFn.run(AFn.java:24)~[clojure-1.5.1.jar:na] atjava.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]2015-03-25 11:28:13 b.s.util [INFO] Halting process:("Error when processing an event")
There does not be any thing corresponding to this in theworker logs.'
Ideas??
ThanksJustin
Re: Supervisors flapping after killing topology
Posted by Justin Workman <ju...@gmail.com>.
Sounds like your storm-local is pointed to tmp and you are going through the same process as us.
Is this a bug, or is there a better solution?
Sent from my iPhone
> On Mar 25, 2015, at 2:41 PM, Andres Gomez Ferrer <ag...@redborder.net> wrote:
>
> I stop all my topologies, stop all storm nodes, remove /tmp/storm, start all again!
>
> Andrés Gómez
> Developer
> redborder.net / agomez@redborder.net
> Phone: +34 955 60 11 60
>
> <0e6e8de_1.png>
>
> <square-twitter-20.png> <square-google-plus-20.png> <square-linkedin-20.png>
>
> Piénsalo antes de imprimir este mensaje
>
> Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción.
>
> This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. If you have received this email in error, please notify the sender and delete it from your system.
>
>
> En 25 de marzo de 2015 en 21:31:50, Justin Workman (justinjworkman@gmail.com) escrito:
>
>> During each topology kill/restart? Where do you put that hook?
>>
>> Or was it a one time remove of /tmp/storm and future topology restarts were fine.
>>
>> Sent from my iPhone
>>
>> On Mar 25, 2015, at 1:59 PM, Andres Gomez Ferrer <ag...@redborder.net> wrote:
>>
>>> I solved it removing all /tmp/storm/*
>>>
>>> Regards,
>>>
>>> Andrés Gómez
>>> Developer
>>> redborder.net / agomez@redborder.net
>>> Phone: +34 955 60 11 60
>>>
>>> <0e6e8de_1.png>
>>>
>>> <square-twitter-20.png> <square-google-plus-20.png> <square-linkedin-20.png>
>>>
>>> Piénsalo antes de imprimir este mensaje
>>>
>>> Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción.
>>>
>>> This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. If you have received this email in error, please notify the sender and delete it from your system.
>>>
>>>
>>> En 25 de marzo de 2015 en 20:54:58, Justin Workman (justinjworkman@gmail.com) escrito:
>>>
>>>> We are currently running 7 node storm cluster, 1 nimbus and 6 supervisor nodes all running storm 0.9.2, running 3 topologies. Any time we kill a running topology the supervisors across all nodes start flapping and we end up in a mess. To clean this up we end up killing all running topologies, shutdown the supervisors, cleanup the storm/storm-local directories on all supervisor nodes, restart the supervisor processes then restart the topologies.
>>>>
>>>> Has anyone experienced this issues, or have any ideas on how to resolve it.
>>>>
>>>> Log snippet we see in the supervisor logs when this happens...
>>>>
>>>> 2015-03-25 11:28:13 b.s.d.supervisor [INFO] Shutting down 4d971d4b-a208-4758-a55e-3e8b34d7531f:ce049d5c-fd4c-499c-ad8d-ef1d8f2b992b
>>>> 2015-03-25 11:28:13 b.s.event [ERROR] Error when processing event
>>>> java.io.IOException: . doesn't exist.
>>>> at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:157) ~[commons-exec-1.1.jar:1.1]
>>>> at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147) ~[commons-exec-1.1.jar:1.1]
>>>> at backtype.storm.util$exec_command_BANG_.invoke(util.clj:378) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at backtype.storm.util$ensure_process_killed_BANG_.invoke(util.clj:394) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at backtype.storm.daemon.supervisor$shutdown_worker.invoke(supervisor.clj:175) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:240) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at clojure.lang.AFn.applyToHelper(AFn.java:161) ~[clojure-1.5.1.jar:na]
>>>> at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
>>>> at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
>>>> at clojure.core$partial$fn__4190.doInvoke(core.clj:2396) ~[clojure-1.5.1.jar:na]
>>>> at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.5.1.jar:na]
>>>> at backtype.storm.event$event_manager$fn__2378.invoke(event.clj:39) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>>> at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
>>>> at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
>>>> 2015-03-25 11:28:13 b.s.util [INFO] Halting process: ("Error when processing an event")
>>>>
>>>>
>>>> There does not be any thing corresponding to this in the worker logs.'
>>>>
>>>> Ideas??
>>>>
>>>> Thanks
>>>> Justin
Re: Supervisors flapping after killing topology
Posted by Andres Gomez Ferrer <ag...@redborder.net>.
I stop all my topologies, stop all storm nodes, remove /tmp/storm, start all again!
Andrés Gómez
Developer
redborder.net / agomez@redborder.net
Phone: +34 955 60 11 60
Piénsalo antes de imprimir este mensaje
Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción.
This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. If you have received this email in error, please notify the sender and delete it from your system.
En 25 de marzo de 2015 en 21:31:50, Justin Workman (justinjworkman@gmail.com) escrito:
During each topology kill/restart? Where do you put that hook?
Or was it a one time remove of /tmp/storm and future topology restarts were fine.
Sent from my iPhone
On Mar 25, 2015, at 1:59 PM, Andres Gomez Ferrer <ag...@redborder.net> wrote:
I solved it removing all /tmp/storm/*
Regards,
Andrés Gómez
Developer
redborder.net / agomez@redborder.net
Phone: +34 955 60 11 60
<0e6e8de_1.png>
<square-twitter-20.png> <square-google-plus-20.png> <square-linkedin-20.png>
Piénsalo antes de imprimir este mensaje
Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción.
This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. If you have received this email in error, please notify the sender and delete it from your system.
En 25 de marzo de 2015 en 20:54:58, Justin Workman (justinjworkman@gmail.com) escrito:
We are currently running 7 node storm cluster, 1 nimbus and 6 supervisor nodes all running storm 0.9.2, running 3 topologies. Any time we kill a running topology the supervisors across all nodes start flapping and we end up in a mess. To clean this up we end up killing all running topologies, shutdown the supervisors, cleanup the storm/storm-local directories on all supervisor nodes, restart the supervisor processes then restart the topologies.
Has anyone experienced this issues, or have any ideas on how to resolve it.
Log snippet we see in the supervisor logs when this happens...
2015-03-25 11:28:13 b.s.d.supervisor [INFO] Shutting down 4d971d4b-a208-4758-a55e-3e8b34d7531f:ce049d5c-fd4c-499c-ad8d-ef1d8f2b992b
2015-03-25 11:28:13 b.s.event [ERROR] Error when processing event
java.io.IOException: . doesn't exist.
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:157) ~[commons-exec-1.1.jar:1.1]
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147) ~[commons-exec-1.1.jar:1.1]
at backtype.storm.util$exec_command_BANG_.invoke(util.clj:378) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$ensure_process_killed_BANG_.invoke(util.clj:394) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.supervisor$shutdown_worker.invoke(supervisor.clj:175) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:240) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.applyToHelper(AFn.java:161) ~[clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
at clojure.core$partial$fn__4190.doInvoke(core.clj:2396) ~[clojure-1.5.1.jar:na]
at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.5.1.jar:na]
at backtype.storm.event$event_manager$fn__2378.invoke(event.clj:39) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
2015-03-25 11:28:13 b.s.util [INFO] Halting process: ("Error when processing an event")
There does not be any thing corresponding to this in the worker logs.'
Ideas??
Thanks
Justin
Re: Supervisors flapping after killing topology
Posted by Justin Workman <ju...@gmail.com>.
During each topology kill/restart? Where do you put that hook?
Or was it a one time remove of /tmp/storm and future topology restarts were fine.
Sent from my iPhone
> On Mar 25, 2015, at 1:59 PM, Andres Gomez Ferrer <ag...@redborder.net> wrote:
>
> I solved it removing all /tmp/storm/*
>
> Regards,
>
> Andrés Gómez
> Developer
> redborder.net / agomez@redborder.net
> Phone: +34 955 60 11 60
>
> <0e6e8de_1.png>
>
> <square-twitter-20.png> <square-google-plus-20.png> <square-linkedin-20.png>
>
> Piénsalo antes de imprimir este mensaje
>
> Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción.
>
> This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. If you have received this email in error, please notify the sender and delete it from your system.
>
>
> En 25 de marzo de 2015 en 20:54:58, Justin Workman (justinjworkman@gmail.com) escrito:
>
>> We are currently running 7 node storm cluster, 1 nimbus and 6 supervisor nodes all running storm 0.9.2, running 3 topologies. Any time we kill a running topology the supervisors across all nodes start flapping and we end up in a mess. To clean this up we end up killing all running topologies, shutdown the supervisors, cleanup the storm/storm-local directories on all supervisor nodes, restart the supervisor processes then restart the topologies.
>>
>> Has anyone experienced this issues, or have any ideas on how to resolve it.
>>
>> Log snippet we see in the supervisor logs when this happens...
>>
>> 2015-03-25 11:28:13 b.s.d.supervisor [INFO] Shutting down 4d971d4b-a208-4758-a55e-3e8b34d7531f:ce049d5c-fd4c-499c-ad8d-ef1d8f2b992b
>> 2015-03-25 11:28:13 b.s.event [ERROR] Error when processing event
>> java.io.IOException: . doesn't exist.
>> at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:157) ~[commons-exec-1.1.jar:1.1]
>> at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147) ~[commons-exec-1.1.jar:1.1]
>> at backtype.storm.util$exec_command_BANG_.invoke(util.clj:378) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at backtype.storm.util$ensure_process_killed_BANG_.invoke(util.clj:394) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at backtype.storm.daemon.supervisor$shutdown_worker.invoke(supervisor.clj:175) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:240) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at clojure.lang.AFn.applyToHelper(AFn.java:161) ~[clojure-1.5.1.jar:na]
>> at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
>> at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
>> at clojure.core$partial$fn__4190.doInvoke(core.clj:2396) ~[clojure-1.5.1.jar:na]
>> at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.5.1.jar:na]
>> at backtype.storm.event$event_manager$fn__2378.invoke(event.clj:39) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
>> at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
>> 2015-03-25 11:28:13 b.s.util [INFO] Halting process: ("Error when processing an event")
>>
>>
>> There does not be any thing corresponding to this in the worker logs.'
>>
>> Ideas??
>>
>> Thanks
>> Justin
Re: Supervisors flapping after killing topology
Posted by Andres Gomez Ferrer <ag...@redborder.net>.
I solved it removing all /tmp/storm/*
Regards,
Andrés Gómez
Developer
redborder.net / agomez@redborder.net
Phone: +34 955 60 11 60
Piénsalo antes de imprimir este mensaje
Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción.
This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. If you have received this email in error, please notify the sender and delete it from your system.
En 25 de marzo de 2015 en 20:54:58, Justin Workman (justinjworkman@gmail.com) escrito:
We are currently running 7 node storm cluster, 1 nimbus and 6 supervisor nodes all running storm 0.9.2, running 3 topologies. Any time we kill a running topology the supervisors across all nodes start flapping and we end up in a mess. To clean this up we end up killing all running topologies, shutdown the supervisors, cleanup the storm/storm-local directories on all supervisor nodes, restart the supervisor processes then restart the topologies.
Has anyone experienced this issues, or have any ideas on how to resolve it.
Log snippet we see in the supervisor logs when this happens...
2015-03-25 11:28:13 b.s.d.supervisor [INFO] Shutting down 4d971d4b-a208-4758-a55e-3e8b34d7531f:ce049d5c-fd4c-499c-ad8d-ef1d8f2b992b
2015-03-25 11:28:13 b.s.event [ERROR] Error when processing event
java.io.IOException: . doesn't exist.
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:157) ~[commons-exec-1.1.jar:1.1]
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147) ~[commons-exec-1.1.jar:1.1]
at backtype.storm.util$exec_command_BANG_.invoke(util.clj:378) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$ensure_process_killed_BANG_.invoke(util.clj:394) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.supervisor$shutdown_worker.invoke(supervisor.clj:175) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:240) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.applyToHelper(AFn.java:161) ~[clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
at clojure.core$partial$fn__4190.doInvoke(core.clj:2396) ~[clojure-1.5.1.jar:na]
at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.5.1.jar:na]
at backtype.storm.event$event_manager$fn__2378.invoke(event.clj:39) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
2015-03-25 11:28:13 b.s.util [INFO] Halting process: ("Error when processing an event")
There does not be any thing corresponding to this in the worker logs.'
Ideas??
Thanks
Justin