You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Markus Weimer <ma...@weimo.de> on 2017/09/04 01:59:52 UTC
Re: Null Exception on accessing FailedContexts for IFailedEvaluator
You should only ever see this if the Evaluator failed during startup.
Any Evaluator process that is running does have at least one context
inside. When did these errors occur? During startup or shutdown of an
Evaluator? -- Markus
On Sat, Sep 2, 2017 at 7:31 PM, Achuth Raghavendra
<ac...@microsoft.com.invalid> wrote:
> In the below scenario, when there is a FailedEvaluator, I see that FailedContexts list is empty. Why would this be empty ?.
>
>
> ERROR: Exception in Call_ClrSystemFailedEvaluator_OnNext
> Encountered error [System.InvalidOperationException: Sequence contains no elements
> at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
> at Microsoft.Cosmos.Table.TableServerResourceReportService.ApplicationMasterDriverImpl.OnNext(IFailedEvaluator value)
> at Org.Apache.REEF.Driver.Bridge.ClrSystemHandler`1.OnNext(T value)
> at Org.Apache.REEF.Driver.Bridge.ClrSystemHandlerWrapper.Call_ClrSystemFailedEvaluator_OnNext(UInt64 handle, IFailedEvaluatorClr2Java clr2Java)
> at Java_org_apache_reef_javabridge_NativeInterop_clrSystemFailedEvaluatorHandlerOnNext(JNIEnv_* env, _jclass* cls, Int64 handler, _jobject* jfailedEvaluator, _jobject* jlogger)]
> <C++> Information: 0 : 2017-09-02T01:24:56.9522856-07:00 0010
>
>
> public void OnNext(IFailedEvaluator value)
> {
> TraceLog.I().Info("Evaluator failed, containerId: {0}", value.Id);
> TraceLog.I().Assert(1 == value.FailedContexts.Count);
> var hostName = value.FailedContexts.First().EvaluatorDescriptor.NodeDescriptor.HostName;
>
>
> From the java code itseems like the code always adding the context. I am not able follow through the C# side of the code. Does anybody know that this not populated at C# side of the code ?.
>
> /**
> * @return a FailedContext for the case of an EvaluatorFailure.
> */
> public synchronized FailedContext getFailedContextForEvaluatorFailure() {
>
> final String id = this.getId();
> final Optional<String> description = Optional.empty();
> final Optional<byte[]> data = Optional.empty();
> final Optional<Throwable> cause = Optional.empty();
> final String message = "Evaluator Failure";
>
> final Optional<ActiveContext> parentContext = getParentId().isPresent() ?
> Optional.<ActiveContext>of(this.contextRepresenters.getContext(getParentId().get())) :
> Optional.<ActiveContext>empty();
>
> final String evaluatorID = getEvaluatorId();
>
> return new FailedContextImpl(
> id, message, description, cause, data, parentContext, this.evaluatorDescriptor, evaluatorID);
> }
>
RE: Null Exception on accessing FailedContexts for IFailedEvaluator
Posted by Achuth Raghavendra <ac...@microsoft.com.INVALID>.
This happened during the shutdown of an Evaluator.
From the logs:
INFO: AllocatedEvaluatorHandler.OnNext
Sep 02, 2017 12:10:05 AM org.apache.reef.javabridge.generic.JobDriver submitEvaluator
INFO: Allocated Evaluator: container_e11267_1503597252060_0003_06_000002, total running running 0
Sep 02, 2017 12:10:05 AM org.apache.reef.javabridge.AllocatedEvaluatorBridge getEvaluatorDescriptorString
INFO: allocated evaluator - serialized evaluator descriptor: IP=bn2phy040431118/100.96.45.70, Port=10131, HostName=bn2phy040431118, Memory=25600, Core=4, RuntimeName=Yarn
Sep 02, 2017 12:10:37 AM com.microsoft.hadoop.net.autopilot.AutopilotTopologyMapping loadPhysicalMachineMapping
INFO: Readed 1146540 physical mappings (removed 0) in 1316 ms
Sep 02, 2017 12:11:01 AM org.apache.reef.runtime.yarn.driver.EvaluatorSetupHelper getResources
WARNING: Failed to delete [D:\data\yarnnm\local\usercache\SYSTEM\appcache\application_1503597252060_0003\container_e11267_1503597252060_0003_06_000001\wd\.\reef\temp\reef-evaluator-2404254299525147894]
Sep 02, 2017 12:11:01 AM org.apache.reef.runtime.yarn.driver.YARNResourceLaunchHandler onNext
INFO: TIME: Run ResourceLaunchProto container_e11267_1503597252060_0003_06_000002 command: reef\global\Org.Apache.Reef.Evaluator.exe reef/local/evaluator.conf ><LOG_DIR>/evaluator.stdout 2><LOG_DIR>/evaluator.stderr resources: {reef/local=resource { scheme: "cosmos" host: "CosmosBajaNarwhal-Test-BN1" port: -1 file: "/vol1/TableServerResourceReport/REEF/2017_08_24_11_04_50_reef-job-3/reef-evaluator-1322461759056979079.jar" } size: 995 timestamp: 995 type: ARCHIVE visibility: APPLICATION, reef/global=resource { scheme: "cosmos" host: "CosmosBajaNarwhal-Test-BN1" port: -1 file: "/vol1/TableServerResourceReport/REEF/2017_08_24_11_04_50_reef-job-3/global.jar" } size: 47667020 timestamp: 47667020 type: ARCHIVE visibility: APPLICATION}
Sep 02, 2017 12:11:01 AM org.apache.reef.runtime.yarn.util.YarnTypes getContainerLaunchContext
INFO: Added tokens to container launch context
Sep 02, 2017 12:11:01 AM org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl run
INFO: Processing Event EventType: START_CONTAINER for Container container_e11267_1503597252060_0003_06_000002
Sep 02, 2017 12:11:01 AM org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy newProxy
INFO: Opening proxy : bn2phy040431118:10131
Sep 02, 2017 12:11:01 AM org.apache.hadoop.ipc.Client <init>
INFO: Connecting to server on port bn2phy040431118:10131
Sep 02, 2017 12:11:01 AM org.apache.hadoop.ipc.Client <init>
INFO: Active service behind a VIP: bn2phy040431118/100.96.45.70
Sep 02, 2017 12:11:02 AM org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl run
INFO: Processing Event EventType: QUERY_CONTAINER for Container container_e11267_1503597252060_0003_06_000002
Sep 02, 2017 12:11:02 AM org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy newProxy
Fails here
WARNING: Failed evaluator: container_e11267_1503597252060_0003_06_000002
org.apache.reef.exception.EvaluatorException: Evaluator [container_e11267_1503597252060_0003_06_000002] is assumed to be in state [SUBMITTED]. But the resource manager reports it to be in state [DONE]. This most likely means that the Evaluator suffered a failure before establishing a communications link to the driver.
at org.apache.reef.runtime.common.driver.evaluator.EvaluatorManager.onResourceStatusMessage(EvaluatorManager.java:684)
at org.apache.reef.runtime.common.driver.resourcemanager.ResourceStatusHandler.onNext(ResourceStatusHandler.java:73)
at org.apache.reef.runtime.common.driver.resourcemanager.ResourceStatusHandler.onNext(ResourceStatusHandler.java:38)
at org.apache.reef.runtime.yarn.driver.REEFEventHandlers.onResourceStatus(REEFEventHandlers.java:93)
at org.apache.reef.runtime.yarn.driver.YarnContainerManager.onContainerStatus(YarnContainerManager.java:479)
at org.apache.reef.runtime.yarn.driver.YarnContainerManager.onContainersCompleted(YarnContainerManager.java:144)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:300)
Sep 02, 2017 1:24:56 AM org.apache.reef.javabridge.generic.JobDriver handleFailedEvaluator
SEVERE: FailedEvaluator
Sep 02, 2017 1:24:56 AM org.apache.reef.runtime.common.driver.client.LoggingJobStatusHandler onNext
INFO: In-process JobStatus:
identifier: "TableServerResourceReportAM"
state: RUNNING
message: "Evaluator container_e11267_1503597252060_0003_06_000002 failed with message: Evaluator [container_e11267_1503597252060_0003_06_000002] is assumed to be in state [SUBMITTED]. But the resource manager reports it to be in state [DONE]. This most likely means that the Evaluator suffered a failure before establishing a communications link to the driver."
Sep 02, 2017 1:24:56 AM org.apache.reef.javabridge.generic.JobDriver handleFailedEvaluatorInCLR
INFO: CLR FailedEvaluator handler set, handling things with CLR handler.
Sep 02, 2017 1:24:56 AM org.apache.reef.javabridge.NativeBridge onError
SEVERE: Bridge received error from CLR: Exception in Call_ClrSystemFailedEvaluator_OnNext
Sequence contains no elements
at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
at Microsoft.Cosmos.Table.TableServerResourceReportService.ApplicationMasterDriverImpl.OnNext(IFailedEvaluator value)
at Org.Apache.REEF.Driver.Bridge.ClrSystemHandler`1.OnNext(T value)
at Org.Apache.REEF.Driver.Bridge.ClrSystemHandlerWrapper.Call_ClrSystemFailedEvaluator_OnNext(UInt64 handle, IFailedEvaluatorClr2Java clr2Java)
at Java_org_apache_reef_javabridge_NativeInterop_clrSystemFailedEvaluatorHandlerOnNext(JNIEnv_* env, _jclass* cls, Int64 handler, _jobject* jfailedEvaluator, _jobject* jlogger)
Inner Exception: null
Sep 02, 2017 1:24:56 AM org.apache.reef.runtime.common.driver.DriverStatusManager onError
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Sunday, September 3, 2017 7:00 PM
To: REEF Developers Mailinglist <de...@reef.apache.org>
Subject: Re: Null Exception on accessing FailedContexts for IFailedEvaluator
You should only ever see this if the Evaluator failed during startup.
Any Evaluator process that is running does have at least one context inside. When did these errors occur? During startup or shutdown of an Evaluator? -- Markus
On Sat, Sep 2, 2017 at 7:31 PM, Achuth Raghavendra <ac...@microsoft.com.invalid>> wrote:
> In the below scenario, when there is a FailedEvaluator, I see that FailedContexts list is empty. Why would this be empty ?.
>
>
> ERROR: Exception in Call_ClrSystemFailedEvaluator_OnNext
> Encountered error [System.InvalidOperationException: Sequence contains no elements
> at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
> at Microsoft.Cosmos.Table.TableServerResourceReportService.ApplicationMasterDriverImpl.OnNext(IFailedEvaluator value)
> at Org.Apache.REEF.Driver.Bridge.ClrSystemHandler`1.OnNext(T value)
> at Org.Apache.REEF.Driver.Bridge.ClrSystemHandlerWrapper.Call_ClrSystemFailedEvaluator_OnNext(UInt64 handle, IFailedEvaluatorClr2Java clr2Java)
> at
> Java_org_apache_reef_javabridge_NativeInterop_clrSystemFailedEvaluator
> HandlerOnNext(JNIEnv_* env, _jclass* cls, Int64 handler, _jobject*
> jfailedEvaluator, _jobject* jlogger)] <C++> Information: 0 :
> 2017-09-02T01:24:56.9522856-07:00 0010
>
>
> public void OnNext(IFailedEvaluator value)
> {
> TraceLog.I().Info("Evaluator failed, containerId: {0}", value.Id);
> TraceLog.I().Assert(1 == value.FailedContexts.Count);
> var hostName =
> value.FailedContexts.First().EvaluatorDescriptor.NodeDescriptor.HostNa
> me;
>
>
> From the java code itseems like the code always adding the context. I am not able follow through the C# side of the code. Does anybody know that this not populated at C# side of the code ?.
>
> /**
> * @return a FailedContext for the case of an EvaluatorFailure.
> */
> public synchronized FailedContext
> getFailedContextForEvaluatorFailure() {
>
> final String id = this.getId();
> final Optional<String> description = Optional.empty();
> final Optional<byte[]> data = Optional.empty();
> final Optional<Throwable> cause = Optional.empty();
> final String message = "Evaluator Failure";
>
> final Optional<ActiveContext> parentContext = getParentId().isPresent() ?
> Optional.<ActiveContext>of(this.contextRepresenters.getContext(getParentId().get())) :
> Optional.<ActiveContext>empty();
>
> final String evaluatorID = getEvaluatorId();
>
> return new FailedContextImpl(
> id, message, description, cause, data, parentContext,
> this.evaluatorDescriptor, evaluatorID); }
>
Re: Null Exception on accessing FailedContexts for IFailedEvaluator
Posted by Markus Weimer <ma...@weimo.de>.
It looks like the Evaluator startup failed. Do you have the logs from
the Evaluator? -- Markus
On Mon, Sep 4, 2017 at 1:08 PM, Achuth Raghavendra
<ac...@microsoft.com.invalid> wrote:
> Also if the context is different in different state, then we should have different call backs like
>
> OnNext(IFailedAllocationEvaluator)
> OnNext(IFailedRunningEvaluator)
> OnNext(IFailedClosingEvaluator)
>
> -----Original Message-----
> From: Markus Weimer [mailto:markus@weimo.de]
> Sent: Sunday, September 3, 2017 7:00 PM
> To: REEF Developers Mailinglist <de...@reef.apache.org>
> Subject: Re: Null Exception on accessing FailedContexts for IFailedEvaluator
>
> You should only ever see this if the Evaluator failed during startup.
> Any Evaluator process that is running does have at least one context inside. When did these errors occur? During startup or shutdown of an Evaluator? -- Markus
>
> On Sat, Sep 2, 2017 at 7:31 PM, Achuth Raghavendra <ac...@microsoft.com.invalid> wrote:
>> In the below scenario, when there is a FailedEvaluator, I see that FailedContexts list is empty. Why would this be empty ?.
>>
>>
>> ERROR: Exception in Call_ClrSystemFailedEvaluator_OnNext
>> Encountered error [System.InvalidOperationException: Sequence contains no elements
>> at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
>> at Microsoft.Cosmos.Table.TableServerResourceReportService.ApplicationMasterDriverImpl.OnNext(IFailedEvaluator value)
>> at Org.Apache.REEF.Driver.Bridge.ClrSystemHandler`1.OnNext(T value)
>> at Org.Apache.REEF.Driver.Bridge.ClrSystemHandlerWrapper.Call_ClrSystemFailedEvaluator_OnNext(UInt64 handle, IFailedEvaluatorClr2Java clr2Java)
>> at
>> Java_org_apache_reef_javabridge_NativeInterop_clrSystemFailedEvaluator
>> HandlerOnNext(JNIEnv_* env, _jclass* cls, Int64 handler, _jobject*
>> jfailedEvaluator, _jobject* jlogger)] <C++> Information: 0 :
>> 2017-09-02T01:24:56.9522856-07:00 0010
>>
>>
>> public void OnNext(IFailedEvaluator value)
>> {
>> TraceLog.I().Info("Evaluator failed, containerId: {0}", value.Id);
>> TraceLog.I().Assert(1 == value.FailedContexts.Count);
>> var hostName =
>> value.FailedContexts.First().EvaluatorDescriptor.NodeDescriptor.HostNa
>> me;
>>
>>
>> From the java code itseems like the code always adding the context. I am not able follow through the C# side of the code. Does anybody know that this not populated at C# side of the code ?.
>>
>> /**
>> * @return a FailedContext for the case of an EvaluatorFailure.
>> */
>> public synchronized FailedContext
>> getFailedContextForEvaluatorFailure() {
>>
>> final String id = this.getId();
>> final Optional<String> description = Optional.empty();
>> final Optional<byte[]> data = Optional.empty();
>> final Optional<Throwable> cause = Optional.empty();
>> final String message = "Evaluator Failure";
>>
>> final Optional<ActiveContext> parentContext = getParentId().isPresent() ?
>> Optional.<ActiveContext>of(this.contextRepresenters.getContext(getParentId().get())) :
>> Optional.<ActiveContext>empty();
>>
>> final String evaluatorID = getEvaluatorId();
>>
>> return new FailedContextImpl(
>> id, message, description, cause, data, parentContext,
>> this.evaluatorDescriptor, evaluatorID); }
>>
RE: Null Exception on accessing FailedContexts for IFailedEvaluator
Posted by Achuth Raghavendra <ac...@microsoft.com.INVALID>.
Also if the context is different in different state, then we should have different call backs like
OnNext(IFailedAllocationEvaluator)
OnNext(IFailedRunningEvaluator)
OnNext(IFailedClosingEvaluator)
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Sunday, September 3, 2017 7:00 PM
To: REEF Developers Mailinglist <de...@reef.apache.org>
Subject: Re: Null Exception on accessing FailedContexts for IFailedEvaluator
You should only ever see this if the Evaluator failed during startup.
Any Evaluator process that is running does have at least one context inside. When did these errors occur? During startup or shutdown of an Evaluator? -- Markus
On Sat, Sep 2, 2017 at 7:31 PM, Achuth Raghavendra <ac...@microsoft.com.invalid> wrote:
> In the below scenario, when there is a FailedEvaluator, I see that FailedContexts list is empty. Why would this be empty ?.
>
>
> ERROR: Exception in Call_ClrSystemFailedEvaluator_OnNext
> Encountered error [System.InvalidOperationException: Sequence contains no elements
> at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
> at Microsoft.Cosmos.Table.TableServerResourceReportService.ApplicationMasterDriverImpl.OnNext(IFailedEvaluator value)
> at Org.Apache.REEF.Driver.Bridge.ClrSystemHandler`1.OnNext(T value)
> at Org.Apache.REEF.Driver.Bridge.ClrSystemHandlerWrapper.Call_ClrSystemFailedEvaluator_OnNext(UInt64 handle, IFailedEvaluatorClr2Java clr2Java)
> at
> Java_org_apache_reef_javabridge_NativeInterop_clrSystemFailedEvaluator
> HandlerOnNext(JNIEnv_* env, _jclass* cls, Int64 handler, _jobject*
> jfailedEvaluator, _jobject* jlogger)] <C++> Information: 0 :
> 2017-09-02T01:24:56.9522856-07:00 0010
>
>
> public void OnNext(IFailedEvaluator value)
> {
> TraceLog.I().Info("Evaluator failed, containerId: {0}", value.Id);
> TraceLog.I().Assert(1 == value.FailedContexts.Count);
> var hostName =
> value.FailedContexts.First().EvaluatorDescriptor.NodeDescriptor.HostNa
> me;
>
>
> From the java code itseems like the code always adding the context. I am not able follow through the C# side of the code. Does anybody know that this not populated at C# side of the code ?.
>
> /**
> * @return a FailedContext for the case of an EvaluatorFailure.
> */
> public synchronized FailedContext
> getFailedContextForEvaluatorFailure() {
>
> final String id = this.getId();
> final Optional<String> description = Optional.empty();
> final Optional<byte[]> data = Optional.empty();
> final Optional<Throwable> cause = Optional.empty();
> final String message = "Evaluator Failure";
>
> final Optional<ActiveContext> parentContext = getParentId().isPresent() ?
> Optional.<ActiveContext>of(this.contextRepresenters.getContext(getParentId().get())) :
> Optional.<ActiveContext>empty();
>
> final String evaluatorID = getEvaluatorId();
>
> return new FailedContextImpl(
> id, message, description, cause, data, parentContext,
> this.evaluatorDescriptor, evaluatorID); }
>