Problems Runnning Storm 1.0.1 procceses under Supervision with Monit 5.17

[Julián Bermejo Ferreiro | BEEVA <ju...@beeva.com>]

Hi!

I need your help.

I'm ugrading our Storm production cluster from 0.9.4 to 1.0.1 version.

All the 0.9.4 Storm procceses are running under supervision with
Monit-5.12.2. In this case, all the start-scripts that Monit use exec
NImbus, Supervisor and UI daemons in *background* (&).

For example, the start-supervisor.sh only contains:

/usr/bin/nohup ${STORM_HOME}/bin/storm supervisor &

This config is strongly probed, actually is running on our production
environment for almost 2 years.

When I try the same configuration with Storm 1.0.1 and Monit 5.17.1, I
think that the system doesn't run properly. Although all the start scripts
are able to set up Nimbus, UI and Supervisor by themselves, when they're
executed by Monit it crash. Monit is not able to raise none of them.

I have tried by modifying start-scripts in order to set up all the Storm
daemons in *foreground*, for example:

/usr/bin/nohup ${STORM_HOME}/bin/storm supervisor

In this case, Monit daemon is able to restart all of them but always in the
second round. During the first attempt, all of them die because of timeout.

These are Monit logs:

For 1st attempt:
error    : 'nimbus' failed to start (exit status -1) --
....../start-nimbus.sh: Program timed out .....
info     : 'nimbus' restart action failed

For 2nd attempt:
info     : 'nimbus' process is running after previous exec error (slow
starting or manually recovered?)

Although finally all procceses are running when I force failures (kill dash
nines), the machine remains a "defunct" procces due to the first failed
attempt.

Is this problem related with the use of the conf/storm.py file?? (Of course
I have python 2.6.6 and jdk 1.8 running on the machnies and all procceses
run OK without Monit)

Should I run NImbus, Supervisor and UI in background or in foreground?

I don't know why is this happening, but I would definetly like to run a
Storm 1.0.1 cluster as strong & stable as our Storm 0.9.4 cluster is.

Could you help me guys?

Thank you sooo much in advance!


*JULIÁN BERMEJO FERREIRO*
*Departamento de Tecnología *
*julian.bermejo@beeva.com <ju...@beeva.com>*
<http://www.beeva.com/>

Re: Problems Runnning Storm 1.0.1 procceses under Supervision with Monit 5.17

[Abhishek Agarwal <ab...@gmail.com>]

You should check the nimbus logs as they may reveal the actual problem.

On Thu, May 12, 2016 at 12:46 PM, Julián Bermejo Ferreiro | BEEVA <
julian.bermejo@beeva.com> wrote:

> Hi!
>
> I need your help.
>
> I'm ugrading our Storm production cluster from 0.9.4 to 1.0.1 version.
>
> All the 0.9.4 Storm procceses are running under supervision with
> Monit-5.12.2. In this case, all the start-scripts that Monit use exec
> NImbus, Supervisor and UI daemons in *background* (&).
>
> For example, the start-supervisor.sh only contains:
>
> /usr/bin/nohup ${STORM_HOME}/bin/storm supervisor &
>
> This config is strongly probed, actually is running on our production
> environment for almost 2 years.
>
> When I try the same configuration with Storm 1.0.1 and Monit 5.17.1, I
> think that the system doesn't run properly. Although all the start scripts
> are able to set up Nimbus, UI and Supervisor by themselves, when they're
> executed by Monit it crash. Monit is not able to raise none of them.
>
> I have tried by modifying start-scripts in order to set up all the Storm
> daemons in *foreground*, for example:
>
> /usr/bin/nohup ${STORM_HOME}/bin/storm supervisor
>
> In this case, Monit daemon is able to restart all of them but always in
> the second round. During the first attempt, all of them die because of
> timeout.
>
> These are Monit logs:
>
> For 1st attempt:
> error    : 'nimbus' failed to start (exit status -1) --
> ....../start-nimbus.sh: Program timed out .....
> info     : 'nimbus' restart action failed
>
> For 2nd attempt:
> info     : 'nimbus' process is running after previous exec error (slow
> starting or manually recovered?)
>
> Although finally all procceses are running when I force failures (kill
> dash nines), the machine remains a "defunct" procces due to the first
> failed attempt.
>
> Is this problem related with the use of the conf/storm.py file?? (Of
> course I have python 2.6.6 and jdk 1.8 running on the machnies and all
> procceses run OK without Monit)
>
> Should I run NImbus, Supervisor and UI in background or in foreground?
>
> I don't know why is this happening, but I would definetly like to run a
> Storm 1.0.1 cluster as strong & stable as our Storm 0.9.4 cluster is.
>
> Could you help me guys?
>
> Thank you sooo much in advance!
>
>
> *JULIÁN BERMEJO FERREIRO*
> *Departamento de Tecnología *
> *julian.bermejo@beeva.com <ju...@beeva.com>*
> <http://www.beeva.com/>
>
>
>
>


-- 
Regards,
Abhishek Agarwal