You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Chris K Wensel <ch...@wensel.net> on 2014/04/05 22:16:54 UTC
NPE in DAGAppMaster
I'm know for a fact I don't have things reasonably configured, but as I sort through what I'm missing, i'm hitting a few walls.
for example, an NPE is being thrown in the DAGAppMaster (below).
first, if this is familiar, anyone have any insight?
second, since we can't flatten everything into a single jvm and let our debuggers lend some insight, I have noticed I can set things like TEZ_PROFILE_JVM_OPTS (which I've yet to benefit from) which should help.
is there an equivalent for launching the AM? i'm sure I'll find it shortly, but thought it useful to ask and see if there is any additional related advice.
fwiw, i'm on the latest commit.
++++++++
> git show-branch
[master] TEZ-1021. TezClient cannot connect to AM in a secure cluster when launched via Oozie. (hitesh)
++++++++
2014-04-05 12:47:40,490 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.dag.impl.TaskImpl: task_1396727252029_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED
2014-04-05 12:47:40,491 FATAL [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
java.lang.NullPointerException
at org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1670)
at org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1656)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:81)
at java.lang.Thread.run(Thread.java:722)
2014-04-05 12:47:40,495 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
--
Chris K Wensel
chris@concurrentinc.com
http://concurrentinc.com
Re: NPE in DAGAppMaster
Posted by Chris K Wensel <ch...@wensel.net>.
sent a link to the logs in another email.
ckw
On Apr 5, 2014, at 8:43 PM, Hitesh Shah <hi...@apache.org> wrote:
> Forgot to add:
>
> Or an issue in the state transition handling itself where we may be assuming that some data is always present. The debug logs should point to what the transition was and help track down the potential null param if this was the case.
>
> -- Hitesh
>
>
> On Sat, Apr 5, 2014 at 8:03 PM, Hitesh Shah <hi...@apache.org> wrote:
> Hi Chris,
>
> For the AM, you can use TEZ_AM_JAVA_OPTS.
>
> Also, it would be useful if you could provide logs with TEZ_AM_LOG_LEVEL set to DEBUG. Might help track this issue down. An NPE in the dispatcher is pretty serious - based on the trace, it seems to imply that a state machine event was being sent to a non existent task.
>
> — Hitesh
>
>
> On Sat, Apr 5, 2014 at 1:16 PM, Chris K Wensel <ch...@wensel.net> wrote:
>
> I'm know for a fact I don't have things reasonably configured, but as I sort through what I'm missing, i'm hitting a few walls.
>
> for example, an NPE is being thrown in the DAGAppMaster (below).
>
> first, if this is familiar, anyone have any insight?
>
> second, since we can't flatten everything into a single jvm and let our debuggers lend some insight, I have noticed I can set things like TEZ_PROFILE_JVM_OPTS (which I've yet to benefit from) which should help.
>
> is there an equivalent for launching the AM? i'm sure I'll find it shortly, but thought it useful to ask and see if there is any additional related advice.
>
> fwiw, i'm on the latest commit.
>
> ++++++++
>
> > git show-branch
> [master] TEZ-1021. TezClient cannot connect to AM in a secure cluster when launched via Oozie. (hitesh)
>
> ++++++++
>
> 2014-04-05 12:47:40,490 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.dag.impl.TaskImpl: task_1396727252029_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED
> 2014-04-05 12:47:40,491 FATAL [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
> java.lang.NullPointerException
> at org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1670)
> at org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1656)
> at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:81)
> at java.lang.Thread.run(Thread.java:722)
> 2014-04-05 12:47:40,495 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
>
> --
> Chris K Wensel
> chris@concurrentinc.com
> http://concurrentinc.com
>
>
>
--
Chris K Wensel
chris@concurrentinc.com
http://concurrentinc.com
Re: NPE in DAGAppMaster
Posted by Hitesh Shah <hi...@apache.org>.
Forgot to add:
Or an issue in the state transition handling itself where we may be
assuming that some data is always present. The debug logs should point to
what the transition was and help track down the potential null param if
this was the case.
-- Hitesh
On Sat, Apr 5, 2014 at 8:03 PM, Hitesh Shah <hi...@apache.org> wrote:
> Hi Chris,
>
> For the AM, you can use TEZ_AM_JAVA_OPTS.
>
> Also, it would be useful if you could provide logs with TEZ_AM_LOG_LEVEL
> set to DEBUG. Might help track this issue down. An NPE in the dispatcher is
> pretty serious - based on the trace, it seems to imply that a state machine
> event was being sent to a non existent task.
>
> -- Hitesh
>
>
> On Sat, Apr 5, 2014 at 1:16 PM, Chris K Wensel <ch...@wensel.net> wrote:
>
>>
>> I'm know for a fact I don't have things reasonably configured, but as I
>> sort through what I'm missing, i'm hitting a few walls.
>>
>> for example, an NPE is being thrown in the DAGAppMaster (below).
>>
>> first, if this is familiar, anyone have any insight?
>>
>> second, since we can't flatten everything into a single jvm and let our
>> debuggers lend some insight, I have noticed I can set things
>> like TEZ_PROFILE_JVM_OPTS (which I've yet to benefit from) which should
>> help.
>>
>> is there an equivalent for launching the AM? i'm sure I'll find it
>> shortly, but thought it useful to ask and see if there is any additional
>> related advice.
>>
>> fwiw, i'm on the latest commit.
>>
>> ++++++++
>>
>> > git show-branch
>> [master] TEZ-1021. TezClient cannot connect to AM in a secure cluster
>> when launched via Oozie. (hitesh)
>>
>> ++++++++
>>
>> 2014-04-05 12:47:40,490 INFO [AsyncDispatcher event handler]
>> org.apache.tez.dag.app.dag.impl.TaskImpl:
>> task_1396727252029_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED
>> 2014-04-05 12:47:40,491 FATAL [AsyncDispatcher event handler]
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
>> java.lang.NullPointerException
>> at
>> org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1670)
>> at
>> org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1656)
>> at
>> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
>> at
>> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:81)
>> at java.lang.Thread.run(Thread.java:722)
>> 2014-04-05 12:47:40,495 INFO [AsyncDispatcher event handler]
>> org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
>>
>> --
>> Chris K Wensel
>> chris@concurrentinc.com
>> http://concurrentinc.com
>>
>>
>
Re: NPE in DAGAppMaster
Posted by Hitesh Shah <hi...@apache.org>.
Hi Chris,
For the AM, you can use TEZ_AM_JAVA_OPTS.
Also, it would be useful if you could provide logs with TEZ_AM_LOG_LEVEL
set to DEBUG. Might help track this issue down. An NPE in the dispatcher is
pretty serious - based on the trace, it seems to imply that a state machine
event was being sent to a non existent task.
-- Hitesh
On Sat, Apr 5, 2014 at 1:16 PM, Chris K Wensel <ch...@wensel.net> wrote:
>
> I'm know for a fact I don't have things reasonably configured, but as I
> sort through what I'm missing, i'm hitting a few walls.
>
> for example, an NPE is being thrown in the DAGAppMaster (below).
>
> first, if this is familiar, anyone have any insight?
>
> second, since we can't flatten everything into a single jvm and let our
> debuggers lend some insight, I have noticed I can set things
> like TEZ_PROFILE_JVM_OPTS (which I've yet to benefit from) which should
> help.
>
> is there an equivalent for launching the AM? i'm sure I'll find it
> shortly, but thought it useful to ask and see if there is any additional
> related advice.
>
> fwiw, i'm on the latest commit.
>
> ++++++++
>
> > git show-branch
> [master] TEZ-1021. TezClient cannot connect to AM in a secure cluster when
> launched via Oozie. (hitesh)
>
> ++++++++
>
> 2014-04-05 12:47:40,490 INFO [AsyncDispatcher event handler]
> org.apache.tez.dag.app.dag.impl.TaskImpl:
> task_1396727252029_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED
> 2014-04-05 12:47:40,491 FATAL [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1670)
> at
> org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1656)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:81)
> at java.lang.Thread.run(Thread.java:722)
> 2014-04-05 12:47:40,495 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
>
> --
> Chris K Wensel
> chris@concurrentinc.com
> http://concurrentinc.com
>
>