You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@reef.apache.org by Brian Cho <ch...@gmail.com> on 2014/09/22 11:49:51 UTC

Can a Driver react to a KILLED container?

Hi,

I’ve run into a (hopefully) rare case when my JVM uses more physical memory
than allowed by YARN, and thus gets killed by the YARN RM. I’d like my REEF
driver to react to this, but I haven’t found a hook to do this. Is there
such a hook? Should there be?

Judging from the logs, I’m going into YarnContainerManager#onContainerStatus
and getting a state of ReefServiceProtos.State.KILLED. This eventually
leads to EvaluatorManager#onResourceStatusMessage where a FINEST log is
output. Then KILLED doesn’t match any of the conditions and that seems to
be the end.

(BTW, I’ve set the heap slack to avoid this. I’m still a bit paranoid
though, and would like the application to adjust if this ever happens
again.)

Thanks,
Brian

Re: Can a Driver react to a KILLED container?

Posted by Brian Cho <ch...@gmail.com>.

Hi Markus,




Sorry for the delay. Yes, this fixes my case. As expected, with the default FailedEvaluator handler, this triggers a Driver shutdown. I’ve attached this part of the logs below.




Thanks!

-Brian




```


2014-09-27 09:51:06,980 WARNING reef.runtime.common.driver.DriverStatusManager.onError container_1411717774404_0002_01_000003-pool-9-thread-1 | Shutting 

down the Driver with an exception:

java.lang.RuntimeException: No handler bound for FailedEvaluator: FailedEvaluator{id='container_1411717774404_0002_01_000003'}

        at com.microsoft.reef.runtime.common.driver.defaults.DefaultEvaluatorFailureHandler.onNext(DefaultEvaluatorFailureHandler.java:34)

        at com.microsoft.reef.runtime.common.driver.defaults.DefaultEvaluatorFailureHandler.onNext(DefaultEvaluatorFailureHandler.java:26)        at com.microsoft.reef.runtime.common.utils.BroadCastEventHandler.onNext(BroadCastEventHandler.java:34)        at com.microsoft.reef.util.ExceptionHandlingEventHandler.onNext(ExceptionHandlingEventHandler.java:42)

        at com.microsoft.reef.runtime.common.utils.DispatchingEStage$1.onNext(DispatchingEStage.java:82)

        at com.microsoft.reef.runtime.common.utils.DispatchingEStage$1.onNext(DispatchingEStage.java:79)

        at com.microsoft.wake.impl.ThreadPoolStage$1.run(ThreadPoolStage.java:182)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:744)

Caused by: com.microsoft.reef.exception.EvaluatorKilledByResourceManagerException: Evaluator [container_1411717774404_0002_01_000003] is assumed to be in state [RUNNING]. But the resource manager reports it to be in state [KILLED]. This means that the Evaluator failed but wasn't able to send an error message back to the driver. Task [InMemoryTask-container_1411717774404_0002_01_000003] was running when the Evaluator crashed.

        at com.microsoft.reef.runtime.common.driver.evaluator.EvaluatorManager.onResourceStatusMessage(EvaluatorManager.java:493)

        at com.microsoft.reef.runtime.common.driver.resourcemanager.ResourceStatusHandler.onNext(ResourceStatusHandler.java:57)

        at com.microsoft.reef.runtime.common.driver.resourcemanager.ResourceStatusHandler.onNext(ResourceStatusHandler.java:34)

        at com.microsoft.reef.runtime.yarn.driver.REEFEventHandlers.onResourceStatus(REEFEventHandlers.java:81)

        at com.microsoft.reef.runtime.yarn.driver.YarnContainerManager.onContainerStatus(YarnContainerManager.java:442)

        at com.microsoft.reef.runtime.yarn.driver.YarnContainerManager.onContainersCompleted(YarnContainerManager.java:112)

        at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:303)

```

On Sat, Sep 27, 2014 at 2:17 AM, Markus Weimer <we...@apache.org> wrote:

> Hi Brian,
> did you find out whether this works for your case? If so, we can merge this
> today.
> Thanks,
> Markus
> On Tue, Sep 23, 2014 at 9:44 PM, Brian Cho <ch...@gmail.com> wrote:
>> Hi Markus,
>>
>>
>>
>>
>> Thank you for the update. Judging from the code, it looks good. (But it
>> doesn’t include the Diagnostics information.)
>>
>>
>>
>>
>> I’ll give the branch a spin, hopefully by end of day (in Korea).
>>
>>
>>
>>
>> -Brian
>>
>> On Wed, Sep 24, 2014 at 8:50 AM, Markus Weimer <we...@apache.org> wrote:
>>
>> > Hi,
>> > I looked into this today. Brian, can you please check that this does
>> indeed
>> > fix it?
>> > https://github.com/Microsoft-CISL/REEF/pull/1000
>> > Thanks,
>> > Markus
>> > On Mon, Sep 22, 2014 at 8:17 PM, Yingda Chen <yd...@gmail.com> wrote:
>> >> This looks like a bug. I can take this if Markus confirm.
>> >>
>> >> -Yingda
>> >>
>> >> On Mon, Sep 22, 2014 at 6:41 PM, Brian Cho <ch...@gmail.com> wrote:
>> >>
>> >> > Hi Markus,
>> >> >
>> >> > In this case I don’t get a FailedEvaluator at all. State.KILLED is
>> >> missing
>> >> > from the condition at EvaluatorManager:L466
>> >> >
>> >> >       } else if ((resourceStatusProto.getState() ==
>> >> > ReefServiceProtos.State.DONE ||
>> >> >           resourceStatusProto.getState() ==
>> >> > ReefServiceProtos.State.FAILED) &&
>> >> >           this.stateManager.isAllocatedOrSubmittedOrRunning()) {
>> >> >
>> >> > Yes it would be great to have the message from the RM as well. It
>> should
>> >> > already be available at the above method using
>> >> > resourceStatusProto.getDiagnostics()
>> >> >
>> >> > Thanks,
>> >> > Brian
>> >> > 
>> >> >
>> >> > On Tue, Sep 23, 2014 at 4:26 AM, Markus Weimer <we...@apache.org>
>> >> wrote:
>> >> >
>> >> > > Hi,
>> >> > >
>> >> > > you should get a `FailedEvaluator` in that case. However, we haven't
>> >> put
>> >> > > logic in place to join the message from the RM and the NM. I haven't
>> >> > looked
>> >> > > into this in a long while. Does it seem easy? If so, we should
>> create a
>> >> > > special `Exception` class that we use in
>> >> > > `FailedEvaluator.getEvaluatorException()`.
>> >> > >
>> >> > > Markus
>> >> > >
>> >> > > On Mon, Sep 22, 2014 at 2:49 AM, Brian Cho <ch...@gmail.com>
>> wrote:
>> >> > >
>> >> > > > Hi,
>> >> > > >
>> >> > > > I’ve run into a (hopefully) rare case when my JVM uses more
>> physical
>> >> > > memory
>> >> > > > than allowed by YARN, and thus gets killed by the YARN RM. I’d
>> like
>> >> my
>> >> > > REEF
>> >> > > > driver to react to this, but I haven’t found a hook to do this. Is
>> >> > there
>> >> > > > such a hook? Should there be?
>> >> > > >
>> >> > > > Judging from the logs, I’m going into
>> >> > > > YarnContainerManager#onContainerStatus
>> >> > > > and getting a state of ReefServiceProtos.State.KILLED. This
>> >> eventually
>> >> > > > leads to EvaluatorManager#onResourceStatusMessage where a FINEST
>> log
>> >> is
>> >> > > > output. Then KILLED doesn’t match any of the conditions and that
>> >> seems
>> >> > to
>> >> > > > be the end.
>> >> > > >
>> >> > > > (BTW, I’ve set the heap slack to avoid this. I’m still a bit
>> paranoid
>> >> > > > though, and would like the application to adjust if this ever
>> happens
>> >> > > > again.)
>> >> > > >
>> >> > > > Thanks,
>> >> > > > Brian
>> >> > > > 
>> >> > > >
>> >> > >
>> >> >
>> >>
>>

Re: Can a Driver react to a KILLED container?

Posted by Markus Weimer <we...@apache.org>.

Hi Brian,

did you find out whether this works for your case? If so, we can merge this
today.

Thanks,

Markus

On Tue, Sep 23, 2014 at 9:44 PM, Brian Cho <ch...@gmail.com> wrote:

> Hi Markus,
>
>
>
>
> Thank you for the update. Judging from the code, it looks good. (But it
> doesn’t include the Diagnostics information.)
>
>
>
>
> I’ll give the branch a spin, hopefully by end of day (in Korea).
>
>
>
>
> -Brian
>
> On Wed, Sep 24, 2014 at 8:50 AM, Markus Weimer <we...@apache.org> wrote:
>
> > Hi,
> > I looked into this today. Brian, can you please check that this does
> indeed
> > fix it?
> > https://github.com/Microsoft-CISL/REEF/pull/1000
> > Thanks,
> > Markus
> > On Mon, Sep 22, 2014 at 8:17 PM, Yingda Chen <yd...@gmail.com> wrote:
> >> This looks like a bug. I can take this if Markus confirm.
> >>
> >> -Yingda
> >>
> >> On Mon, Sep 22, 2014 at 6:41 PM, Brian Cho <ch...@gmail.com> wrote:
> >>
> >> > Hi Markus,
> >> >
> >> > In this case I don’t get a FailedEvaluator at all. State.KILLED is
> >> missing
> >> > from the condition at EvaluatorManager:L466
> >> >
> >> >       } else if ((resourceStatusProto.getState() ==
> >> > ReefServiceProtos.State.DONE ||
> >> >           resourceStatusProto.getState() ==
> >> > ReefServiceProtos.State.FAILED) &&
> >> >           this.stateManager.isAllocatedOrSubmittedOrRunning()) {
> >> >
> >> > Yes it would be great to have the message from the RM as well. It
> should
> >> > already be available at the above method using
> >> > resourceStatusProto.getDiagnostics()
> >> >
> >> > Thanks,
> >> > Brian
> >> > 
> >> >
> >> > On Tue, Sep 23, 2014 at 4:26 AM, Markus Weimer <we...@apache.org>
> >> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > you should get a `FailedEvaluator` in that case. However, we haven't
> >> put
> >> > > logic in place to join the message from the RM and the NM. I haven't
> >> > looked
> >> > > into this in a long while. Does it seem easy? If so, we should
> create a
> >> > > special `Exception` class that we use in
> >> > > `FailedEvaluator.getEvaluatorException()`.
> >> > >
> >> > > Markus
> >> > >
> >> > > On Mon, Sep 22, 2014 at 2:49 AM, Brian Cho <ch...@gmail.com>
> wrote:
> >> > >
> >> > > > Hi,
> >> > > >
> >> > > > I’ve run into a (hopefully) rare case when my JVM uses more
> physical
> >> > > memory
> >> > > > than allowed by YARN, and thus gets killed by the YARN RM. I’d
> like
> >> my
> >> > > REEF
> >> > > > driver to react to this, but I haven’t found a hook to do this. Is
> >> > there
> >> > > > such a hook? Should there be?
> >> > > >
> >> > > > Judging from the logs, I’m going into
> >> > > > YarnContainerManager#onContainerStatus
> >> > > > and getting a state of ReefServiceProtos.State.KILLED. This
> >> eventually
> >> > > > leads to EvaluatorManager#onResourceStatusMessage where a FINEST
> log
> >> is
> >> > > > output. Then KILLED doesn’t match any of the conditions and that
> >> seems
> >> > to
> >> > > > be the end.
> >> > > >
> >> > > > (BTW, I’ve set the heap slack to avoid this. I’m still a bit
> paranoid
> >> > > > though, and would like the application to adjust if this ever
> happens
> >> > > > again.)
> >> > > >
> >> > > > Thanks,
> >> > > > Brian
> >> > > > 
> >> > > >
> >> > >
> >> >
> >>
>

Re: Can a Driver react to a KILLED container?

Posted by Brian Cho <ch...@gmail.com>.

Hi Markus,




Thank you for the update. Judging from the code, it looks good. (But it doesn’t include the Diagnostics information.)




I’ll give the branch a spin, hopefully by end of day (in Korea).




-Brian

On Wed, Sep 24, 2014 at 8:50 AM, Markus Weimer <we...@apache.org> wrote:

> Hi,
> I looked into this today. Brian, can you please check that this does indeed
> fix it?
> https://github.com/Microsoft-CISL/REEF/pull/1000
> Thanks,
> Markus
> On Mon, Sep 22, 2014 at 8:17 PM, Yingda Chen <yd...@gmail.com> wrote:
>> This looks like a bug. I can take this if Markus confirm.
>>
>> -Yingda
>>
>> On Mon, Sep 22, 2014 at 6:41 PM, Brian Cho <ch...@gmail.com> wrote:
>>
>> > Hi Markus,
>> >
>> > In this case I don’t get a FailedEvaluator at all. State.KILLED is
>> missing
>> > from the condition at EvaluatorManager:L466
>> >
>> >       } else if ((resourceStatusProto.getState() ==
>> > ReefServiceProtos.State.DONE ||
>> >           resourceStatusProto.getState() ==
>> > ReefServiceProtos.State.FAILED) &&
>> >           this.stateManager.isAllocatedOrSubmittedOrRunning()) {
>> >
>> > Yes it would be great to have the message from the RM as well. It should
>> > already be available at the above method using
>> > resourceStatusProto.getDiagnostics()
>> >
>> > Thanks,
>> > Brian
>> > 
>> >
>> > On Tue, Sep 23, 2014 at 4:26 AM, Markus Weimer <we...@apache.org>
>> wrote:
>> >
>> > > Hi,
>> > >
>> > > you should get a `FailedEvaluator` in that case. However, we haven't
>> put
>> > > logic in place to join the message from the RM and the NM. I haven't
>> > looked
>> > > into this in a long while. Does it seem easy? If so, we should create a
>> > > special `Exception` class that we use in
>> > > `FailedEvaluator.getEvaluatorException()`.
>> > >
>> > > Markus
>> > >
>> > > On Mon, Sep 22, 2014 at 2:49 AM, Brian Cho <ch...@gmail.com> wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > I’ve run into a (hopefully) rare case when my JVM uses more physical
>> > > memory
>> > > > than allowed by YARN, and thus gets killed by the YARN RM. I’d like
>> my
>> > > REEF
>> > > > driver to react to this, but I haven’t found a hook to do this. Is
>> > there
>> > > > such a hook? Should there be?
>> > > >
>> > > > Judging from the logs, I’m going into
>> > > > YarnContainerManager#onContainerStatus
>> > > > and getting a state of ReefServiceProtos.State.KILLED. This
>> eventually
>> > > > leads to EvaluatorManager#onResourceStatusMessage where a FINEST log
>> is
>> > > > output. Then KILLED doesn’t match any of the conditions and that
>> seems
>> > to
>> > > > be the end.
>> > > >
>> > > > (BTW, I’ve set the heap slack to avoid this. I’m still a bit paranoid
>> > > > though, and would like the application to adjust if this ever happens
>> > > > again.)
>> > > >
>> > > > Thanks,
>> > > > Brian
>> > > > 
>> > > >
>> > >
>> >
>>

Re: Can a Driver react to a KILLED container?

Posted by Markus Weimer <we...@apache.org>.

Hi,

I looked into this today. Brian, can you please check that this does indeed
fix it?

https://github.com/Microsoft-CISL/REEF/pull/1000

Thanks,

Markus

On Mon, Sep 22, 2014 at 8:17 PM, Yingda Chen <yd...@gmail.com> wrote:

> This looks like a bug. I can take this if Markus confirm.
>
> -Yingda
>
> On Mon, Sep 22, 2014 at 6:41 PM, Brian Cho <ch...@gmail.com> wrote:
>
> > Hi Markus,
> >
> > In this case I don’t get a FailedEvaluator at all. State.KILLED is
> missing
> > from the condition at EvaluatorManager:L466
> >
> >       } else if ((resourceStatusProto.getState() ==
> > ReefServiceProtos.State.DONE ||
> >           resourceStatusProto.getState() ==
> > ReefServiceProtos.State.FAILED) &&
> >           this.stateManager.isAllocatedOrSubmittedOrRunning()) {
> >
> > Yes it would be great to have the message from the RM as well. It should
> > already be available at the above method using
> > resourceStatusProto.getDiagnostics()
> >
> > Thanks,
> > Brian
> > 
> >
> > On Tue, Sep 23, 2014 at 4:26 AM, Markus Weimer <we...@apache.org>
> wrote:
> >
> > > Hi,
> > >
> > > you should get a `FailedEvaluator` in that case. However, we haven't
> put
> > > logic in place to join the message from the RM and the NM. I haven't
> > looked
> > > into this in a long while. Does it seem easy? If so, we should create a
> > > special `Exception` class that we use in
> > > `FailedEvaluator.getEvaluatorException()`.
> > >
> > > Markus
> > >
> > > On Mon, Sep 22, 2014 at 2:49 AM, Brian Cho <ch...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I’ve run into a (hopefully) rare case when my JVM uses more physical
> > > memory
> > > > than allowed by YARN, and thus gets killed by the YARN RM. I’d like
> my
> > > REEF
> > > > driver to react to this, but I haven’t found a hook to do this. Is
> > there
> > > > such a hook? Should there be?
> > > >
> > > > Judging from the logs, I’m going into
> > > > YarnContainerManager#onContainerStatus
> > > > and getting a state of ReefServiceProtos.State.KILLED. This
> eventually
> > > > leads to EvaluatorManager#onResourceStatusMessage where a FINEST log
> is
> > > > output. Then KILLED doesn’t match any of the conditions and that
> seems
> > to
> > > > be the end.
> > > >
> > > > (BTW, I’ve set the heap slack to avoid this. I’m still a bit paranoid
> > > > though, and would like the application to adjust if this ever happens
> > > > again.)
> > > >
> > > > Thanks,
> > > > Brian
> > > > 
> > > >
> > >
> >
>

Re: Can a Driver react to a KILLED container?

Posted by Yingda Chen <yd...@gmail.com>.

This looks like a bug. I can take this if Markus confirm.

-Yingda

On Mon, Sep 22, 2014 at 6:41 PM, Brian Cho <ch...@gmail.com> wrote:

> Hi Markus,
>
> In this case I don’t get a FailedEvaluator at all. State.KILLED is missing
> from the condition at EvaluatorManager:L466
>
>       } else if ((resourceStatusProto.getState() ==
> ReefServiceProtos.State.DONE ||
>           resourceStatusProto.getState() ==
> ReefServiceProtos.State.FAILED) &&
>           this.stateManager.isAllocatedOrSubmittedOrRunning()) {
>
> Yes it would be great to have the message from the RM as well. It should
> already be available at the above method using
> resourceStatusProto.getDiagnostics()
>
> Thanks,
> Brian
> 
>
> On Tue, Sep 23, 2014 at 4:26 AM, Markus Weimer <we...@apache.org> wrote:
>
> > Hi,
> >
> > you should get a `FailedEvaluator` in that case. However, we haven't put
> > logic in place to join the message from the RM and the NM. I haven't
> looked
> > into this in a long while. Does it seem easy? If so, we should create a
> > special `Exception` class that we use in
> > `FailedEvaluator.getEvaluatorException()`.
> >
> > Markus
> >
> > On Mon, Sep 22, 2014 at 2:49 AM, Brian Cho <ch...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I’ve run into a (hopefully) rare case when my JVM uses more physical
> > memory
> > > than allowed by YARN, and thus gets killed by the YARN RM. I’d like my
> > REEF
> > > driver to react to this, but I haven’t found a hook to do this. Is
> there
> > > such a hook? Should there be?
> > >
> > > Judging from the logs, I’m going into
> > > YarnContainerManager#onContainerStatus
> > > and getting a state of ReefServiceProtos.State.KILLED. This eventually
> > > leads to EvaluatorManager#onResourceStatusMessage where a FINEST log is
> > > output. Then KILLED doesn’t match any of the conditions and that seems
> to
> > > be the end.
> > >
> > > (BTW, I’ve set the heap slack to avoid this. I’m still a bit paranoid
> > > though, and would like the application to adjust if this ever happens
> > > again.)
> > >
> > > Thanks,
> > > Brian
> > > 
> > >
> >
>

Re: Can a Driver react to a KILLED container?

Posted by Brian Cho <ch...@gmail.com>.

Hi Markus,

In this case I don’t get a FailedEvaluator at all. State.KILLED is missing
from the condition at EvaluatorManager:L466

      } else if ((resourceStatusProto.getState() ==
ReefServiceProtos.State.DONE ||
          resourceStatusProto.getState() == ReefServiceProtos.State.FAILED) &&
          this.stateManager.isAllocatedOrSubmittedOrRunning()) {

Yes it would be great to have the message from the RM as well. It should
already be available at the above method using
resourceStatusProto.getDiagnostics()

Thanks,
Brian


On Tue, Sep 23, 2014 at 4:26 AM, Markus Weimer <we...@apache.org> wrote:

> Hi,
>
> you should get a `FailedEvaluator` in that case. However, we haven't put
> logic in place to join the message from the RM and the NM. I haven't looked
> into this in a long while. Does it seem easy? If so, we should create a
> special `Exception` class that we use in
> `FailedEvaluator.getEvaluatorException()`.
>
> Markus
>
> On Mon, Sep 22, 2014 at 2:49 AM, Brian Cho <ch...@gmail.com> wrote:
>
> > Hi,
> >
> > I’ve run into a (hopefully) rare case when my JVM uses more physical
> memory
> > than allowed by YARN, and thus gets killed by the YARN RM. I’d like my
> REEF
> > driver to react to this, but I haven’t found a hook to do this. Is there
> > such a hook? Should there be?
> >
> > Judging from the logs, I’m going into
> > YarnContainerManager#onContainerStatus
> > and getting a state of ReefServiceProtos.State.KILLED. This eventually
> > leads to EvaluatorManager#onResourceStatusMessage where a FINEST log is
> > output. Then KILLED doesn’t match any of the conditions and that seems to
> > be the end.
> >
> > (BTW, I’ve set the heap slack to avoid this. I’m still a bit paranoid
> > though, and would like the application to adjust if this ever happens
> > again.)
> >
> > Thanks,
> > Brian
> > 
> >
>

Re: Can a Driver react to a KILLED container?

Posted by Markus Weimer <we...@apache.org>.

Hi,

you should get a `FailedEvaluator` in that case. However, we haven't put
logic in place to join the message from the RM and the NM. I haven't looked
into this in a long while. Does it seem easy? If so, we should create a
special `Exception` class that we use in
`FailedEvaluator.getEvaluatorException()`.

Markus

On Mon, Sep 22, 2014 at 2:49 AM, Brian Cho <ch...@gmail.com> wrote:

> Hi,
>
> I’ve run into a (hopefully) rare case when my JVM uses more physical memory
> than allowed by YARN, and thus gets killed by the YARN RM. I’d like my REEF
> driver to react to this, but I haven’t found a hook to do this. Is there
> such a hook? Should there be?
>
> Judging from the logs, I’m going into
> YarnContainerManager#onContainerStatus
> and getting a state of ReefServiceProtos.State.KILLED. This eventually
> leads to EvaluatorManager#onResourceStatusMessage where a FINEST log is
> output. Then KILLED doesn’t match any of the conditions and that seems to
> be the end.
>
> (BTW, I’ve set the heap slack to avoid this. I’m still a bit paranoid
> though, and would like the application to adjust if this ever happens
> again.)
>
> Thanks,
> Brian
> 
>