You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Muqtafi Akhmad <mu...@traveloka.com> on 2015/06/01 09:13:05 UTC
Submit Spark App to Oozie
Hello everyone,
I'm new to Oozie, currently I'm trying to use Oozie to manage Spark
applications. I've tried to submit Spark application as Java action ( as
described in
http://stackoverflow.com/questions/29233487/launching-a-spark-program-using-oozie-workflow
) and the action return error
> Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1]
My questions are
1. How can I see / access the complete stack trace to know what exactly
happened in my application execution?
2. By default, Oozie shared lib directory provides libraries for Hive, Pig,
and Map-Reduce. If I want to run Spark job on Oozie, it might be better to
add Spark lib jars to Oozie's shared lib instead of copy them to app's lib
directory. How can I add Spark lib jars (including spark-core and its
dependencies) to Oozie's shared lib?
Thank you,
--
Muqtafi Akhmad
Software Engineer
Traveloka
Re: Submit Spark App to Oozie
Posted by Muqtafi Akhmad <mu...@traveloka.com>.
thank you Laurent!
But I think I'll go with the new Oozie version now that Oozie supports
Spark apps
On Sat, Jun 6, 2015 at 4:59 AM, Laurent H <la...@gmail.com> wrote:
> Maybe add the spark-site.xml (if this file exists, I am not a spark
> expert!) To the container!
>
> Le vendredi 5 juin 2015, Laurent H <la...@gmail.com> a écrit :
>
> > Hi,
> >
> > I can give you some ideas. Hope to be helpful :
> >
> > Add spark.job.queuename or smthg like this in your spark program or in
> > oozie configuration node (like mapred.job.queuename) you can add some
> spark
> > libs in your shell action with the # to put all of them in the same
> > container.
> >
> > Good luck!
> >
> > Le jeudi 4 juin 2015, Muqtafi Akhmad <muqtafi@traveloka.com
> > <javascript:_e(%7B%7D,'cvml','muqtafi@traveloka.com');>> a écrit :
> >
> >> dear all,
> >> Currently I am trying to submit Spark application using spark-submit
> (with
> >> Spark application copy exists in all YARN nodes) by writing it down in
> >> Oozie shell action, here is the workflow.xml :
> >> http://pastebin.com/kHVtmNex
> >>
> >> When submitting this job, I kept getting
> >> Error code : JA018
> >> Error message : Main class [org.apache.oozie.action.hadoop.ShellMain],
> >> exit
> >> code [1]
> >>
> >> What do those messages mean?
> >> Thank you,
> >>
> >>
> >>
> >> On Mon, Jun 1, 2015 at 7:45 PM, Jaydeep Vishwakarma <
> >> jaydeep.vishwakarma@inmobi.com> wrote:
> >>
> >> > I am not sure what to look. There is a detail log hyperlink option in
> >> > Launcher log on ui. You can try to search the exception over there.
> >> >
> >> > You can add any library as part of sharelib. You just need to push the
> >> lib
> >> > directory inside the sharelib directory of hdfs and update the
> >> sharelibs by
> >> > "oozie admin -sharelibupdate" command.
> >> >
> >> > Spark libs have to present in hdfs, It is independent to modes.
> >> >
> >> >
> >> > On Mon, Jun 1, 2015 at 3:24 PM, Muqtafi Akhmad <muqtafi@traveloka.com
> >
> >> > wrote:
> >> >
> >> > > Hello Jaydeep,
> >> > > Logs from
> >> > >
> >> > > > oozie job -info ...
> >> > >
> >> > > and
> >> > >
> >> > > > yarn logs -applicationId ...
> >> > >
> >> > > give the same explanation like the one I've got from Oozie web app.
> Is
> >> > > there any meaning for
> >> > >
> >> > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code
> [1]
> >> > >
> >> > > error?
> >> > >
> >> > > It is good that Spark action will be supported by Oozie, but given
> our
> >> > > current Oozie version, can developer add Spark (or other 'common')
> >> > > libraries to Oozie's shared libs? When you mention
> >> > >
> >> > > > It is only working as local mode.
> >> > >
> >> > > does it mean that we don't need to copy Spark libs and apps to HDFS?
> >> > > Thank you,
> >> > >
> >> > > On Mon, Jun 1, 2015 at 3:17 PM, Jaydeep Vishwakarma <
> >> > > jaydeep.vishwakarma@inmobi.com> wrote:
> >> > >
> >> > > > You can get more details for this error on Launcher logs by
> >> accessing
> >> > the
> >> > > > console url.
> >> > > > Spark action has been recently introduced in oozie. It will come
> in
> >> > > 4.2.0 ,
> >> > > > But there are some issue with spark action, It is only working as
> >> local
> >> > > > mode. Here is the jira for same.
> >> > > > https://issues.apache.org/jira/browse/OOZIE-2143.
> >> > > >
> >> > > > Regards,
> >> > > > Jaydeep
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, Jun 1, 2015 at 12:43 PM, Muqtafi Akhmad <
> >> muqtafi@traveloka.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > Hello everyone,
> >> > > > > I'm new to Oozie, currently I'm trying to use Oozie to manage
> >> Spark
> >> > > > > applications. I've tried to submit Spark application as Java
> >> action (
> >> > > as
> >> > > > > described in
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> http://stackoverflow.com/questions/29233487/launching-a-spark-program-using-oozie-workflow
> >> > > > > ) and the action return error
> >> > > > >
> >> > > > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit
> code
> >> [1]
> >> > > > >
> >> > > > > My questions are
> >> > > > > 1. How can I see / access the complete stack trace to know what
> >> > exactly
> >> > > > > happened in my application execution?
> >> > > > > 2. By default, Oozie shared lib directory provides libraries for
> >> > Hive,
> >> > > > Pig,
> >> > > > > and Map-Reduce. If I want to run Spark job on Oozie, it might be
> >> > better
> >> > > > to
> >> > > > > add Spark lib jars to Oozie's shared lib instead of copy them to
> >> > app's
> >> > > > lib
> >> > > > > directory. How can I add Spark lib jars (including spark-core
> and
> >> its
> >> > > > > dependencies) to Oozie's shared lib?
> >> > > > >
> >> > > > > Thank you,
> >> > > > >
> >> > > > > --
> >> > > > > Muqtafi Akhmad
> >> > > > > Software Engineer
> >> > > > > Traveloka
> >> > > > >
> >> > > >
> >> > > > --
> >> > > > _____________________________________________________________
> >> > > > The information contained in this communication is intended solely
> >> for
> >> > > the
> >> > > > use of the individual or entity to whom it is addressed and others
> >> > > > authorized to receive it. It may contain confidential or legally
> >> > > privileged
> >> > > > information. If you are not the intended recipient you are hereby
> >> > > notified
> >> > > > that any disclosure, copying, distribution or taking any action in
> >> > > reliance
> >> > > > on the contents of this information is strictly prohibited and may
> >> be
> >> > > > unlawful. If you have received this communication in error, please
> >> > notify
> >> > > > us immediately by responding to this email and then delete it from
> >> your
> >> > > > system. The firm is neither liable for the proper and complete
> >> > > transmission
> >> > > > of the information contained in this communication nor for any
> >> delay in
> >> > > its
> >> > > > receipt.
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Muqtafi Akhmad
> >> > > Software Engineer
> >> > > Traveloka
> >> > >
> >> >
> >> > --
> >> > _____________________________________________________________
> >> > The information contained in this communication is intended solely for
> >> the
> >> > use of the individual or entity to whom it is addressed and others
> >> > authorized to receive it. It may contain confidential or legally
> >> privileged
> >> > information. If you are not the intended recipient you are hereby
> >> notified
> >> > that any disclosure, copying, distribution or taking any action in
> >> reliance
> >> > on the contents of this information is strictly prohibited and may be
> >> > unlawful. If you have received this communication in error, please
> >> notify
> >> > us immediately by responding to this email and then delete it from
> your
> >> > system. The firm is neither liable for the proper and complete
> >> transmission
> >> > of the information contained in this communication nor for any delay
> in
> >> its
> >> > receipt.
> >> >
> >>
> >>
> >>
> >> --
> >> Muqtafi Akhmad
> >> Software Engineer
> >> Traveloka
> >>
> >
> >
> > --
> > --
> > Laurent HATIER - Consultant Big Data & Business Intelligence chez
> CapGemini
> > fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
> > <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
> >
> >
>
> --
> --
> Laurent HATIER - Consultant Big Data & Business Intelligence chez CapGemini
> fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
> <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
>
--
Muqtafi Akhmad
Software Engineer
Traveloka
Re: Submit Spark App to Oozie
Posted by Laurent H <la...@gmail.com>.
Maybe add the spark-site.xml (if this file exists, I am not a spark
expert!) To the container!
Le vendredi 5 juin 2015, Laurent H <la...@gmail.com> a écrit :
> Hi,
>
> I can give you some ideas. Hope to be helpful :
>
> Add spark.job.queuename or smthg like this in your spark program or in
> oozie configuration node (like mapred.job.queuename) you can add some spark
> libs in your shell action with the # to put all of them in the same
> container.
>
> Good luck!
>
> Le jeudi 4 juin 2015, Muqtafi Akhmad <muqtafi@traveloka.com
> <javascript:_e(%7B%7D,'cvml','muqtafi@traveloka.com');>> a écrit :
>
>> dear all,
>> Currently I am trying to submit Spark application using spark-submit (with
>> Spark application copy exists in all YARN nodes) by writing it down in
>> Oozie shell action, here is the workflow.xml :
>> http://pastebin.com/kHVtmNex
>>
>> When submitting this job, I kept getting
>> Error code : JA018
>> Error message : Main class [org.apache.oozie.action.hadoop.ShellMain],
>> exit
>> code [1]
>>
>> What do those messages mean?
>> Thank you,
>>
>>
>>
>> On Mon, Jun 1, 2015 at 7:45 PM, Jaydeep Vishwakarma <
>> jaydeep.vishwakarma@inmobi.com> wrote:
>>
>> > I am not sure what to look. There is a detail log hyperlink option in
>> > Launcher log on ui. You can try to search the exception over there.
>> >
>> > You can add any library as part of sharelib. You just need to push the
>> lib
>> > directory inside the sharelib directory of hdfs and update the
>> sharelibs by
>> > "oozie admin -sharelibupdate" command.
>> >
>> > Spark libs have to present in hdfs, It is independent to modes.
>> >
>> >
>> > On Mon, Jun 1, 2015 at 3:24 PM, Muqtafi Akhmad <mu...@traveloka.com>
>> > wrote:
>> >
>> > > Hello Jaydeep,
>> > > Logs from
>> > >
>> > > > oozie job -info ...
>> > >
>> > > and
>> > >
>> > > > yarn logs -applicationId ...
>> > >
>> > > give the same explanation like the one I've got from Oozie web app. Is
>> > > there any meaning for
>> > >
>> > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1]
>> > >
>> > > error?
>> > >
>> > > It is good that Spark action will be supported by Oozie, but given our
>> > > current Oozie version, can developer add Spark (or other 'common')
>> > > libraries to Oozie's shared libs? When you mention
>> > >
>> > > > It is only working as local mode.
>> > >
>> > > does it mean that we don't need to copy Spark libs and apps to HDFS?
>> > > Thank you,
>> > >
>> > > On Mon, Jun 1, 2015 at 3:17 PM, Jaydeep Vishwakarma <
>> > > jaydeep.vishwakarma@inmobi.com> wrote:
>> > >
>> > > > You can get more details for this error on Launcher logs by
>> accessing
>> > the
>> > > > console url.
>> > > > Spark action has been recently introduced in oozie. It will come in
>> > > 4.2.0 ,
>> > > > But there are some issue with spark action, It is only working as
>> local
>> > > > mode. Here is the jira for same.
>> > > > https://issues.apache.org/jira/browse/OOZIE-2143.
>> > > >
>> > > > Regards,
>> > > > Jaydeep
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Jun 1, 2015 at 12:43 PM, Muqtafi Akhmad <
>> muqtafi@traveloka.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > Hello everyone,
>> > > > > I'm new to Oozie, currently I'm trying to use Oozie to manage
>> Spark
>> > > > > applications. I've tried to submit Spark application as Java
>> action (
>> > > as
>> > > > > described in
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://stackoverflow.com/questions/29233487/launching-a-spark-program-using-oozie-workflow
>> > > > > ) and the action return error
>> > > > >
>> > > > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code
>> [1]
>> > > > >
>> > > > > My questions are
>> > > > > 1. How can I see / access the complete stack trace to know what
>> > exactly
>> > > > > happened in my application execution?
>> > > > > 2. By default, Oozie shared lib directory provides libraries for
>> > Hive,
>> > > > Pig,
>> > > > > and Map-Reduce. If I want to run Spark job on Oozie, it might be
>> > better
>> > > > to
>> > > > > add Spark lib jars to Oozie's shared lib instead of copy them to
>> > app's
>> > > > lib
>> > > > > directory. How can I add Spark lib jars (including spark-core and
>> its
>> > > > > dependencies) to Oozie's shared lib?
>> > > > >
>> > > > > Thank you,
>> > > > >
>> > > > > --
>> > > > > Muqtafi Akhmad
>> > > > > Software Engineer
>> > > > > Traveloka
>> > > > >
>> > > >
>> > > > --
>> > > > _____________________________________________________________
>> > > > The information contained in this communication is intended solely
>> for
>> > > the
>> > > > use of the individual or entity to whom it is addressed and others
>> > > > authorized to receive it. It may contain confidential or legally
>> > > privileged
>> > > > information. If you are not the intended recipient you are hereby
>> > > notified
>> > > > that any disclosure, copying, distribution or taking any action in
>> > > reliance
>> > > > on the contents of this information is strictly prohibited and may
>> be
>> > > > unlawful. If you have received this communication in error, please
>> > notify
>> > > > us immediately by responding to this email and then delete it from
>> your
>> > > > system. The firm is neither liable for the proper and complete
>> > > transmission
>> > > > of the information contained in this communication nor for any
>> delay in
>> > > its
>> > > > receipt.
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Muqtafi Akhmad
>> > > Software Engineer
>> > > Traveloka
>> > >
>> >
>> > --
>> > _____________________________________________________________
>> > The information contained in this communication is intended solely for
>> the
>> > use of the individual or entity to whom it is addressed and others
>> > authorized to receive it. It may contain confidential or legally
>> privileged
>> > information. If you are not the intended recipient you are hereby
>> notified
>> > that any disclosure, copying, distribution or taking any action in
>> reliance
>> > on the contents of this information is strictly prohibited and may be
>> > unlawful. If you have received this communication in error, please
>> notify
>> > us immediately by responding to this email and then delete it from your
>> > system. The firm is neither liable for the proper and complete
>> transmission
>> > of the information contained in this communication nor for any delay in
>> its
>> > receipt.
>> >
>>
>>
>>
>> --
>> Muqtafi Akhmad
>> Software Engineer
>> Traveloka
>>
>
>
> --
> --
> Laurent HATIER - Consultant Big Data & Business Intelligence chez CapGemini
> fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
> <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
>
>
--
--
Laurent HATIER - Consultant Big Data & Business Intelligence chez CapGemini
fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
<http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
Re: Submit Spark App to Oozie
Posted by Laurent H <la...@gmail.com>.
Hi,
I can give you some ideas. Hope to be helpful :
Add spark.job.queuename or smthg like this in your spark program or in
oozie configuration node (like mapred.job.queuename) you can add some spark
libs in your shell action with the # to put all of them in the same
container.
Good luck!
Le jeudi 4 juin 2015, Muqtafi Akhmad <mu...@traveloka.com> a écrit :
> dear all,
> Currently I am trying to submit Spark application using spark-submit (with
> Spark application copy exists in all YARN nodes) by writing it down in
> Oozie shell action, here is the workflow.xml :
> http://pastebin.com/kHVtmNex
>
> When submitting this job, I kept getting
> Error code : JA018
> Error message : Main class [org.apache.oozie.action.hadoop.ShellMain], exit
> code [1]
>
> What do those messages mean?
> Thank you,
>
>
>
> On Mon, Jun 1, 2015 at 7:45 PM, Jaydeep Vishwakarma <
> jaydeep.vishwakarma@inmobi.com <javascript:;>> wrote:
>
> > I am not sure what to look. There is a detail log hyperlink option in
> > Launcher log on ui. You can try to search the exception over there.
> >
> > You can add any library as part of sharelib. You just need to push the
> lib
> > directory inside the sharelib directory of hdfs and update the sharelibs
> by
> > "oozie admin -sharelibupdate" command.
> >
> > Spark libs have to present in hdfs, It is independent to modes.
> >
> >
> > On Mon, Jun 1, 2015 at 3:24 PM, Muqtafi Akhmad <muqtafi@traveloka.com
> <javascript:;>>
> > wrote:
> >
> > > Hello Jaydeep,
> > > Logs from
> > >
> > > > oozie job -info ...
> > >
> > > and
> > >
> > > > yarn logs -applicationId ...
> > >
> > > give the same explanation like the one I've got from Oozie web app. Is
> > > there any meaning for
> > >
> > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1]
> > >
> > > error?
> > >
> > > It is good that Spark action will be supported by Oozie, but given our
> > > current Oozie version, can developer add Spark (or other 'common')
> > > libraries to Oozie's shared libs? When you mention
> > >
> > > > It is only working as local mode.
> > >
> > > does it mean that we don't need to copy Spark libs and apps to HDFS?
> > > Thank you,
> > >
> > > On Mon, Jun 1, 2015 at 3:17 PM, Jaydeep Vishwakarma <
> > > jaydeep.vishwakarma@inmobi.com <javascript:;>> wrote:
> > >
> > > > You can get more details for this error on Launcher logs by accessing
> > the
> > > > console url.
> > > > Spark action has been recently introduced in oozie. It will come in
> > > 4.2.0 ,
> > > > But there are some issue with spark action, It is only working as
> local
> > > > mode. Here is the jira for same.
> > > > https://issues.apache.org/jira/browse/OOZIE-2143.
> > > >
> > > > Regards,
> > > > Jaydeep
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Jun 1, 2015 at 12:43 PM, Muqtafi Akhmad <
> muqtafi@traveloka.com <javascript:;>
> > >
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > > I'm new to Oozie, currently I'm trying to use Oozie to manage Spark
> > > > > applications. I've tried to submit Spark application as Java
> action (
> > > as
> > > > > described in
> > > > >
> > > > >
> > > >
> > >
> >
> http://stackoverflow.com/questions/29233487/launching-a-spark-program-using-oozie-workflow
> > > > > ) and the action return error
> > > > >
> > > > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code
> [1]
> > > > >
> > > > > My questions are
> > > > > 1. How can I see / access the complete stack trace to know what
> > exactly
> > > > > happened in my application execution?
> > > > > 2. By default, Oozie shared lib directory provides libraries for
> > Hive,
> > > > Pig,
> > > > > and Map-Reduce. If I want to run Spark job on Oozie, it might be
> > better
> > > > to
> > > > > add Spark lib jars to Oozie's shared lib instead of copy them to
> > app's
> > > > lib
> > > > > directory. How can I add Spark lib jars (including spark-core and
> its
> > > > > dependencies) to Oozie's shared lib?
> > > > >
> > > > > Thank you,
> > > > >
> > > > > --
> > > > > Muqtafi Akhmad
> > > > > Software Engineer
> > > > > Traveloka
> > > > >
> > > >
> > > > --
> > > > _____________________________________________________________
> > > > The information contained in this communication is intended solely
> for
> > > the
> > > > use of the individual or entity to whom it is addressed and others
> > > > authorized to receive it. It may contain confidential or legally
> > > privileged
> > > > information. If you are not the intended recipient you are hereby
> > > notified
> > > > that any disclosure, copying, distribution or taking any action in
> > > reliance
> > > > on the contents of this information is strictly prohibited and may be
> > > > unlawful. If you have received this communication in error, please
> > notify
> > > > us immediately by responding to this email and then delete it from
> your
> > > > system. The firm is neither liable for the proper and complete
> > > transmission
> > > > of the information contained in this communication nor for any delay
> in
> > > its
> > > > receipt.
> > > >
> > >
> > >
> > >
> > > --
> > > Muqtafi Akhmad
> > > Software Engineer
> > > Traveloka
> > >
> >
> > --
> > _____________________________________________________________
> > The information contained in this communication is intended solely for
> the
> > use of the individual or entity to whom it is addressed and others
> > authorized to receive it. It may contain confidential or legally
> privileged
> > information. If you are not the intended recipient you are hereby
> notified
> > that any disclosure, copying, distribution or taking any action in
> reliance
> > on the contents of this information is strictly prohibited and may be
> > unlawful. If you have received this communication in error, please notify
> > us immediately by responding to this email and then delete it from your
> > system. The firm is neither liable for the proper and complete
> transmission
> > of the information contained in this communication nor for any delay in
> its
> > receipt.
> >
>
>
>
> --
> Muqtafi Akhmad
> Software Engineer
> Traveloka
>
--
--
Laurent HATIER - Consultant Big Data & Business Intelligence chez CapGemini
fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
<http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
Re: Submit Spark App to Oozie
Posted by Muqtafi Akhmad <mu...@traveloka.com>.
dear all,
Currently I am trying to submit Spark application using spark-submit (with
Spark application copy exists in all YARN nodes) by writing it down in
Oozie shell action, here is the workflow.xml :
http://pastebin.com/kHVtmNex
When submitting this job, I kept getting
Error code : JA018
Error message : Main class [org.apache.oozie.action.hadoop.ShellMain], exit
code [1]
What do those messages mean?
Thank you,
On Mon, Jun 1, 2015 at 7:45 PM, Jaydeep Vishwakarma <
jaydeep.vishwakarma@inmobi.com> wrote:
> I am not sure what to look. There is a detail log hyperlink option in
> Launcher log on ui. You can try to search the exception over there.
>
> You can add any library as part of sharelib. You just need to push the lib
> directory inside the sharelib directory of hdfs and update the sharelibs by
> "oozie admin -sharelibupdate" command.
>
> Spark libs have to present in hdfs, It is independent to modes.
>
>
> On Mon, Jun 1, 2015 at 3:24 PM, Muqtafi Akhmad <mu...@traveloka.com>
> wrote:
>
> > Hello Jaydeep,
> > Logs from
> >
> > > oozie job -info ...
> >
> > and
> >
> > > yarn logs -applicationId ...
> >
> > give the same explanation like the one I've got from Oozie web app. Is
> > there any meaning for
> >
> > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1]
> >
> > error?
> >
> > It is good that Spark action will be supported by Oozie, but given our
> > current Oozie version, can developer add Spark (or other 'common')
> > libraries to Oozie's shared libs? When you mention
> >
> > > It is only working as local mode.
> >
> > does it mean that we don't need to copy Spark libs and apps to HDFS?
> > Thank you,
> >
> > On Mon, Jun 1, 2015 at 3:17 PM, Jaydeep Vishwakarma <
> > jaydeep.vishwakarma@inmobi.com> wrote:
> >
> > > You can get more details for this error on Launcher logs by accessing
> the
> > > console url.
> > > Spark action has been recently introduced in oozie. It will come in
> > 4.2.0 ,
> > > But there are some issue with spark action, It is only working as local
> > > mode. Here is the jira for same.
> > > https://issues.apache.org/jira/browse/OOZIE-2143.
> > >
> > > Regards,
> > > Jaydeep
> > >
> > >
> > >
> > >
> > > On Mon, Jun 1, 2015 at 12:43 PM, Muqtafi Akhmad <muqtafi@traveloka.com
> >
> > > wrote:
> > >
> > > > Hello everyone,
> > > > I'm new to Oozie, currently I'm trying to use Oozie to manage Spark
> > > > applications. I've tried to submit Spark application as Java action (
> > as
> > > > described in
> > > >
> > > >
> > >
> >
> http://stackoverflow.com/questions/29233487/launching-a-spark-program-using-oozie-workflow
> > > > ) and the action return error
> > > >
> > > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1]
> > > >
> > > > My questions are
> > > > 1. How can I see / access the complete stack trace to know what
> exactly
> > > > happened in my application execution?
> > > > 2. By default, Oozie shared lib directory provides libraries for
> Hive,
> > > Pig,
> > > > and Map-Reduce. If I want to run Spark job on Oozie, it might be
> better
> > > to
> > > > add Spark lib jars to Oozie's shared lib instead of copy them to
> app's
> > > lib
> > > > directory. How can I add Spark lib jars (including spark-core and its
> > > > dependencies) to Oozie's shared lib?
> > > >
> > > > Thank you,
> > > >
> > > > --
> > > > Muqtafi Akhmad
> > > > Software Engineer
> > > > Traveloka
> > > >
> > >
> > > --
> > > _____________________________________________________________
> > > The information contained in this communication is intended solely for
> > the
> > > use of the individual or entity to whom it is addressed and others
> > > authorized to receive it. It may contain confidential or legally
> > privileged
> > > information. If you are not the intended recipient you are hereby
> > notified
> > > that any disclosure, copying, distribution or taking any action in
> > reliance
> > > on the contents of this information is strictly prohibited and may be
> > > unlawful. If you have received this communication in error, please
> notify
> > > us immediately by responding to this email and then delete it from your
> > > system. The firm is neither liable for the proper and complete
> > transmission
> > > of the information contained in this communication nor for any delay in
> > its
> > > receipt.
> > >
> >
> >
> >
> > --
> > Muqtafi Akhmad
> > Software Engineer
> > Traveloka
> >
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
--
Muqtafi Akhmad
Software Engineer
Traveloka
Re: Submit Spark App to Oozie
Posted by Jaydeep Vishwakarma <ja...@inmobi.com>.
I am not sure what to look. There is a detail log hyperlink option in
Launcher log on ui. You can try to search the exception over there.
You can add any library as part of sharelib. You just need to push the lib
directory inside the sharelib directory of hdfs and update the sharelibs by
"oozie admin -sharelibupdate" command.
Spark libs have to present in hdfs, It is independent to modes.
On Mon, Jun 1, 2015 at 3:24 PM, Muqtafi Akhmad <mu...@traveloka.com>
wrote:
> Hello Jaydeep,
> Logs from
>
> > oozie job -info ...
>
> and
>
> > yarn logs -applicationId ...
>
> give the same explanation like the one I've got from Oozie web app. Is
> there any meaning for
>
> > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1]
>
> error?
>
> It is good that Spark action will be supported by Oozie, but given our
> current Oozie version, can developer add Spark (or other 'common')
> libraries to Oozie's shared libs? When you mention
>
> > It is only working as local mode.
>
> does it mean that we don't need to copy Spark libs and apps to HDFS?
> Thank you,
>
> On Mon, Jun 1, 2015 at 3:17 PM, Jaydeep Vishwakarma <
> jaydeep.vishwakarma@inmobi.com> wrote:
>
> > You can get more details for this error on Launcher logs by accessing the
> > console url.
> > Spark action has been recently introduced in oozie. It will come in
> 4.2.0 ,
> > But there are some issue with spark action, It is only working as local
> > mode. Here is the jira for same.
> > https://issues.apache.org/jira/browse/OOZIE-2143.
> >
> > Regards,
> > Jaydeep
> >
> >
> >
> >
> > On Mon, Jun 1, 2015 at 12:43 PM, Muqtafi Akhmad <mu...@traveloka.com>
> > wrote:
> >
> > > Hello everyone,
> > > I'm new to Oozie, currently I'm trying to use Oozie to manage Spark
> > > applications. I've tried to submit Spark application as Java action (
> as
> > > described in
> > >
> > >
> >
> http://stackoverflow.com/questions/29233487/launching-a-spark-program-using-oozie-workflow
> > > ) and the action return error
> > >
> > > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1]
> > >
> > > My questions are
> > > 1. How can I see / access the complete stack trace to know what exactly
> > > happened in my application execution?
> > > 2. By default, Oozie shared lib directory provides libraries for Hive,
> > Pig,
> > > and Map-Reduce. If I want to run Spark job on Oozie, it might be better
> > to
> > > add Spark lib jars to Oozie's shared lib instead of copy them to app's
> > lib
> > > directory. How can I add Spark lib jars (including spark-core and its
> > > dependencies) to Oozie's shared lib?
> > >
> > > Thank you,
> > >
> > > --
> > > Muqtafi Akhmad
> > > Software Engineer
> > > Traveloka
> > >
> >
> > --
> > _____________________________________________________________
> > The information contained in this communication is intended solely for
> the
> > use of the individual or entity to whom it is addressed and others
> > authorized to receive it. It may contain confidential or legally
> privileged
> > information. If you are not the intended recipient you are hereby
> notified
> > that any disclosure, copying, distribution or taking any action in
> reliance
> > on the contents of this information is strictly prohibited and may be
> > unlawful. If you have received this communication in error, please notify
> > us immediately by responding to this email and then delete it from your
> > system. The firm is neither liable for the proper and complete
> transmission
> > of the information contained in this communication nor for any delay in
> its
> > receipt.
> >
>
>
>
> --
> Muqtafi Akhmad
> Software Engineer
> Traveloka
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: Submit Spark App to Oozie
Posted by Muqtafi Akhmad <mu...@traveloka.com>.
Hello Jaydeep,
Logs from
> oozie job -info ...
and
> yarn logs -applicationId ...
give the same explanation like the one I've got from Oozie web app. Is
there any meaning for
> Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1]
error?
It is good that Spark action will be supported by Oozie, but given our
current Oozie version, can developer add Spark (or other 'common')
libraries to Oozie's shared libs? When you mention
> It is only working as local mode.
does it mean that we don't need to copy Spark libs and apps to HDFS?
Thank you,
On Mon, Jun 1, 2015 at 3:17 PM, Jaydeep Vishwakarma <
jaydeep.vishwakarma@inmobi.com> wrote:
> You can get more details for this error on Launcher logs by accessing the
> console url.
> Spark action has been recently introduced in oozie. It will come in 4.2.0 ,
> But there are some issue with spark action, It is only working as local
> mode. Here is the jira for same.
> https://issues.apache.org/jira/browse/OOZIE-2143.
>
> Regards,
> Jaydeep
>
>
>
>
> On Mon, Jun 1, 2015 at 12:43 PM, Muqtafi Akhmad <mu...@traveloka.com>
> wrote:
>
> > Hello everyone,
> > I'm new to Oozie, currently I'm trying to use Oozie to manage Spark
> > applications. I've tried to submit Spark application as Java action ( as
> > described in
> >
> >
> http://stackoverflow.com/questions/29233487/launching-a-spark-program-using-oozie-workflow
> > ) and the action return error
> >
> > > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1]
> >
> > My questions are
> > 1. How can I see / access the complete stack trace to know what exactly
> > happened in my application execution?
> > 2. By default, Oozie shared lib directory provides libraries for Hive,
> Pig,
> > and Map-Reduce. If I want to run Spark job on Oozie, it might be better
> to
> > add Spark lib jars to Oozie's shared lib instead of copy them to app's
> lib
> > directory. How can I add Spark lib jars (including spark-core and its
> > dependencies) to Oozie's shared lib?
> >
> > Thank you,
> >
> > --
> > Muqtafi Akhmad
> > Software Engineer
> > Traveloka
> >
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
--
Muqtafi Akhmad
Software Engineer
Traveloka
Re: Submit Spark App to Oozie
Posted by Jaydeep Vishwakarma <ja...@inmobi.com>.
You can get more details for this error on Launcher logs by accessing the
console url.
Spark action has been recently introduced in oozie. It will come in 4.2.0 ,
But there are some issue with spark action, It is only working as local
mode. Here is the jira for same.
https://issues.apache.org/jira/browse/OOZIE-2143.
Regards,
Jaydeep
On Mon, Jun 1, 2015 at 12:43 PM, Muqtafi Akhmad <mu...@traveloka.com>
wrote:
> Hello everyone,
> I'm new to Oozie, currently I'm trying to use Oozie to manage Spark
> applications. I've tried to submit Spark application as Java action ( as
> described in
>
> http://stackoverflow.com/questions/29233487/launching-a-spark-program-using-oozie-workflow
> ) and the action return error
>
> > Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1]
>
> My questions are
> 1. How can I see / access the complete stack trace to know what exactly
> happened in my application execution?
> 2. By default, Oozie shared lib directory provides libraries for Hive, Pig,
> and Map-Reduce. If I want to run Spark job on Oozie, it might be better to
> add Spark lib jars to Oozie's shared lib instead of copy them to app's lib
> directory. How can I add Spark lib jars (including spark-core and its
> dependencies) to Oozie's shared lib?
>
> Thank you,
>
> --
> Muqtafi Akhmad
> Software Engineer
> Traveloka
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.