You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Pierre Villard <pi...@gmail.com> on 2016/06/14 16:04:52 UTC
Spark Action NPE
Hi,
I am trying to run a Spark action (Oozie 4.2.0) and everything works fine
if my action is run in local mode. As soon as I switch to cluster mode with
yarn as master I get a NullPointerException from Oozie:
JOB[0000724-160602113052690-oozie-oozi-W]
ACTION[0000724-160602113052690-oozie-oozi-W@submit-spark-job] Error
starting action [submit-spark-job]. ErrorType [ERROR], ErrorCode
[NullPointerException], Message [NullPointerException: null]
org.apache.oozie.action.ActionExecutorException: NullPointerException: null
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:445)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1139)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1293)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
at org.apache.oozie.command.XCommand.call(XCommand.java:286)
at org.apache.oozie.command.XCommand.call(XCommand.java:356)
at
org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:434)
at
org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:76)
at org.apache.oozie.command.XCommand.call(XCommand.java:286)
at org.apache.oozie.command.XCommand.call(XCommand.java:356)
at
org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:280)
at
org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:61)
at org.apache.oozie.command.XCommand.call(XCommand.java:286)
at org.apache.oozie.command.XCommand.call(XCommand.java:356)
at
org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
at
org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
at org.apache.oozie.command.XCommand.call(XCommand.java:286)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at
org.apache.oozie.action.hadoop.SparkActionExecutor.setupActionConf(SparkActionExecutor.java:79)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1042)
... 20 more
I am wondering if this is somehow related to:
https://issues.apache.org/jira/browse/SPARK-7706
In any case, what is the cause? is there a workaround?
Thanks for your help,
Pierre
Re: Spark Action NPE
Posted by Amit Kumar <ak...@gmail.com>.
Unsubscribe
On Tue, Jun 14, 2016 at 9:34 PM, Pierre Villard <pierre.villard.fr@gmail.com
> wrote:
> Hi,
>
> I am trying to run a Spark action (Oozie 4.2.0) and everything works fine
> if my action is run in local mode. As soon as I switch to cluster mode with
> yarn as master I get a NullPointerException from Oozie:
>
> JOB[0000724-160602113052690-oozie-oozi-W]
> ACTION[0000724-160602113052690-oozie-oozi-W@submit-spark-job] Error
> starting action [submit-spark-job]. ErrorType [ERROR], ErrorCode
> [NullPointerException], Message [NullPointerException: null]
> org.apache.oozie.action.ActionExecutorException: NullPointerException: null
> at
>
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:445)
> at
>
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1139)
> at
>
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1293)
> at
>
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
> at
>
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
> at org.apache.oozie.command.XCommand.call(XCommand.java:286)
> at org.apache.oozie.command.XCommand.call(XCommand.java:356)
> at
> org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:434)
> at
> org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:76)
> at org.apache.oozie.command.XCommand.call(XCommand.java:286)
> at org.apache.oozie.command.XCommand.call(XCommand.java:356)
> at
>
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:280)
> at
>
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:61)
> at org.apache.oozie.command.XCommand.call(XCommand.java:286)
> at org.apache.oozie.command.XCommand.call(XCommand.java:356)
> at
>
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
> at
>
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
> at org.apache.oozie.command.XCommand.call(XCommand.java:286)
> at
>
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
>
> org.apache.oozie.action.hadoop.SparkActionExecutor.setupActionConf(SparkActionExecutor.java:79)
> at
>
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1042)
> ... 20 more
>
> I am wondering if this is somehow related to:
> https://issues.apache.org/jira/browse/SPARK-7706
>
> In any case, what is the cause? is there a workaround?
>
> Thanks for your help,
> Pierre
>
Re: Spark Action NPE
Posted by Nitin Kumar <nk...@gmail.com>.
Hi,
I also faced similar a problem while trying to deploy spark applications
through the spark action. I switched to the shell action as a workaround.
You can follow the link below for a more descriptive answer.
http://stackoverflow.com/questions/31450828/oozie-job-wont-run-if-using-pyspark-in-sparkaction/33094709#33094709
I am interested however to know how to achieve this through the spark
action.
Hoping that someone can site a good example.
Regards,
Nitin Kumar
On 14-Jun-2016 9:35 PM, "Pierre Villard" <pi...@gmail.com>
wrote:
> Hi,
>
> I am trying to run a Spark action (Oozie 4.2.0) and everything works fine
> if my action is run in local mode. As soon as I switch to cluster mode with
> yarn as master I get a NullPointerException from Oozie:
>
> JOB[0000724-160602113052690-oozie-oozi-W]
> ACTION[0000724-160602113052690-oozie-oozi-W@submit-spark-job] Error
> starting action [submit-spark-job]. ErrorType [ERROR], ErrorCode
> [NullPointerException], Message [NullPointerException: null]
> org.apache.oozie.action.ActionExecutorException: NullPointerException: null
> at
>
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:445)
> at
>
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1139)
> at
>
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1293)
> at
>
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
> at
>
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
> at org.apache.oozie.command.XCommand.call(XCommand.java:286)
> at org.apache.oozie.command.XCommand.call(XCommand.java:356)
> at
> org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:434)
> at
> org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:76)
> at org.apache.oozie.command.XCommand.call(XCommand.java:286)
> at org.apache.oozie.command.XCommand.call(XCommand.java:356)
> at
>
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:280)
> at
>
> org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:61)
> at org.apache.oozie.command.XCommand.call(XCommand.java:286)
> at org.apache.oozie.command.XCommand.call(XCommand.java:356)
> at
>
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
> at
>
> org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
> at org.apache.oozie.command.XCommand.call(XCommand.java:286)
> at
>
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
>
> org.apache.oozie.action.hadoop.SparkActionExecutor.setupActionConf(SparkActionExecutor.java:79)
> at
>
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1042)
> ... 20 more
>
> I am wondering if this is somehow related to:
> https://issues.apache.org/jira/browse/SPARK-7706
>
> In any case, what is the cause? is there a workaround?
>
> Thanks for your help,
> Pierre
>