You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by purna pradeep <pu...@gmail.com> on 2018/05/15 15:07:50 UTC
Spark 2.3 in oozie
Hello,
Does oozie supports spark 2.3? Or will it even care of the spark version
I want to use spark action
Thanks,
Purna
Re: Spark 2.3 in oozie
Posted by Peter Cseh <ge...@cloudera.com>.
Wow, that's great news!
Can I ask you to summarize the steps necessary to make this happen? It
would be good to see everything together - also, it would probably help
others as well.
Thank you for sharing your struggles - and solutions as well!
Peter
On Wed, May 16, 2018 at 10:49 PM, purna pradeep <pu...@gmail.com>
wrote:
> Thanks Peter!
>
> I’m able to run spark pi example on Kubernetes cluster from oozie after
> this change
>
> On Wed, May 16, 2018 at 10:27 AM Peter Cseh <ge...@cloudera.com> wrote:
>
> > The version of the xml schema has nothing to do with the version of the
> > component you're using.
> >
> > Thanks for verifying that -Dspark.scala.binary.verstion=2.11 is required
> > for compilation with Spark 2.3.0
> >
> > Oozie does not pull in Spark's Kubernetes artifact.
> > To make it part of the Oozie Spark sharelib you'll have to include the
> > spark-kubernetes.jar
> > <
> > https://search.maven.org/#artifactdetails%7Corg.apache.
> spark%7Cspark-kubernetes_2.11%7C2.3.0%7Cjar
> > >
> > in
> > the sharelib/spark/pom.xml as a compile-time dependency.
> >
> > gp
> >
> > On Tue, May 15, 2018 at 9:04 PM, purna pradeep <pu...@gmail.com>
> > wrote:
> >
> > > I’m able to compile successfully with after adding this override option
> > >
> > > -Dspark.scala.binary.version=2.11
> > >
> > > Dspark.version = 2.3.0
> > >
> > > But when I’m running a spark action with spark-pi example jar against
> > > Kubernetes master I’m getting below error in stderr log
> > >
> > >
> > > *Error:Could not load KUBERNETES classes.This copy of spark may not
> have
> > > been compiled with Kubernetes support*
> > >
> > > Below is my workflow.xml
> > >
> > > <*spark xmlns="uri:oozie:spark-action:1.0">*
> > >
> > > * <resource-manager>${resourceManager}</resource-manager>*
> > >
> > > * <name-node>${nameNode}</name-node>*
> > >
> > > * <master>k8s://<***.com></master>*
> > >
> > > * <name>Python-Spark-Pi</name>*
> > >
> > > * <jar>spark-examples_2.11-2.3.0.jar</jar>*
> > >
> > > * <spark-opts>--class org.apache.spark.examples.SparkPi
> --conf
> > > spark.executor.instances=2 --conf spark.kubernetes.namespace=spark
> --conf
> > > spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf
> > > spark.kubernetes.container.image=artifactory.cloud.
> > > capitalone.com/kubespark/spark-quantum:v2.3.0
> > > <http://artifactory.cloud.capitalone.com/kubespark/
> spark-quantum:v2.3.0>
> > > --conf
> > spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true
> > > <
> > http://spark.kubernetes.node.selector.node-role.kubernetes.
> io/worker=true
> > > >
> > > --conf
> > > spark.kubernetes.driver.label.application=is1-driver --conf
> > > spark.kubernetes.executor.label.application=is1-exec*utor
> > > local:///opt/spark/examples/jars/spark-examples_2.11-2.3.
> > > 0.jar</spark-opts>
> > >
> > > </spark>
> > >
> > >
> > > Is this because of uri:oozie:spark-action:1.0 in spark xml tag? Does it
> > > needs to be spark-action:2.0 as I’m using spark 2.3?
> > >
> > >
> > > Please suggest!
> > >
> > >
> > > On Tue, May 15, 2018 at 12:43 PM Peter Cseh <ge...@cloudera.com>
> > wrote:
> > >
> > > > I think the error is related to the Scala version being present in
> the
> > > > artifact name.
> > > > I'll take a look at this tomorrow.
> > > > Gp
> > > >
> > > > On Tue, May 15, 2018, 18:28 Artem Ervits <ar...@gmail.com>
> > wrote:
> > > >
> > > > > Did you run
> > > > > mvn clean install first on the parent directory?
> > > > >
> > > > > On Tue, May 15, 2018, 11:35 AM purna pradeep <
> > purna2pradeep@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks peter,
> > > > > >
> > > > > > I have tried changing Dspark.version to 2.3.0 and compiled oozie
> > I’m
> > > > > > getting below error from oozie examples
> > > > > >
> > > > > >
> > > > > > *ERROR] Failed to execute goal on project oozie-examples: Could
> not
> > > > > resolve
> > > > > > dependencies for project org.apache.oozie:oozie-
> examples:jar:5.0.0:
> > > > Could
> > > > > > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in
> > > > > resolution
> > > > > > *
> > > > > >
> > > > > > On Tue, May 15, 2018 at 11:14 AM Peter Cseh <
> gezapeti@cloudera.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Oozie has a spark-2 profile that is currently hard-coded to
> Spark
> > > > 2.1:
> > > > > > > https://github.com/apache/oozie/blob/master/pom.xml#L1983
> > > > > > > I'm sure if you overwrite the -Dspark.version and compile Oozie
> > > that
> > > > > way
> > > > > > it
> > > > > > > will work.
> > > > > > > gp
> > > > > > >
> > > > > > >
> > > > > > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep <
> > > > > purna2pradeep@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > Does oozie supports spark 2.3? Or will it even care of the
> > spark
> > > > > > version
> > > > > > > >
> > > > > > > > I want to use spark action
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Purna
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Peter Cseh *| Software Engineer
> > > > > > > cloudera.com <https://www.cloudera.com>
> > > > > > >
> > > > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > > > >
> > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> > > [image:
> > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> > [image:
> > > > > > Cloudera
> > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > > > > ------------------------------
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > *Peter Cseh *| Software Engineer
> > cloudera.com <https://www.cloudera.com>
> >
> > [image: Cloudera] <https://www.cloudera.com/>
> >
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera
> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > ------------------------------
> >
>
--
*Peter Cseh *| Software Engineer
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
Re: Spark 2.3 in oozie
Posted by purna pradeep <pu...@gmail.com>.
Thanks Peter!
I’m able to run spark pi example on Kubernetes cluster from oozie after
this change
On Wed, May 16, 2018 at 10:27 AM Peter Cseh <ge...@cloudera.com> wrote:
> The version of the xml schema has nothing to do with the version of the
> component you're using.
>
> Thanks for verifying that -Dspark.scala.binary.verstion=2.11 is required
> for compilation with Spark 2.3.0
>
> Oozie does not pull in Spark's Kubernetes artifact.
> To make it part of the Oozie Spark sharelib you'll have to include the
> spark-kubernetes.jar
> <
> https://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-kubernetes_2.11%7C2.3.0%7Cjar
> >
> in
> the sharelib/spark/pom.xml as a compile-time dependency.
>
> gp
>
> On Tue, May 15, 2018 at 9:04 PM, purna pradeep <pu...@gmail.com>
> wrote:
>
> > I’m able to compile successfully with after adding this override option
> >
> > -Dspark.scala.binary.version=2.11
> >
> > Dspark.version = 2.3.0
> >
> > But when I’m running a spark action with spark-pi example jar against
> > Kubernetes master I’m getting below error in stderr log
> >
> >
> > *Error:Could not load KUBERNETES classes.This copy of spark may not have
> > been compiled with Kubernetes support*
> >
> > Below is my workflow.xml
> >
> > <*spark xmlns="uri:oozie:spark-action:1.0">*
> >
> > * <resource-manager>${resourceManager}</resource-manager>*
> >
> > * <name-node>${nameNode}</name-node>*
> >
> > * <master>k8s://<***.com></master>*
> >
> > * <name>Python-Spark-Pi</name>*
> >
> > * <jar>spark-examples_2.11-2.3.0.jar</jar>*
> >
> > * <spark-opts>--class org.apache.spark.examples.SparkPi --conf
> > spark.executor.instances=2 --conf spark.kubernetes.namespace=spark --conf
> > spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf
> > spark.kubernetes.container.image=artifactory.cloud.
> > capitalone.com/kubespark/spark-quantum:v2.3.0
> > <http://artifactory.cloud.capitalone.com/kubespark/spark-quantum:v2.3.0>
> > --conf
> spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true
> > <
> http://spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true
> > >
> > --conf
> > spark.kubernetes.driver.label.application=is1-driver --conf
> > spark.kubernetes.executor.label.application=is1-exec*utor
> > local:///opt/spark/examples/jars/spark-examples_2.11-2.3.
> > 0.jar</spark-opts>
> >
> > </spark>
> >
> >
> > Is this because of uri:oozie:spark-action:1.0 in spark xml tag? Does it
> > needs to be spark-action:2.0 as I’m using spark 2.3?
> >
> >
> > Please suggest!
> >
> >
> > On Tue, May 15, 2018 at 12:43 PM Peter Cseh <ge...@cloudera.com>
> wrote:
> >
> > > I think the error is related to the Scala version being present in the
> > > artifact name.
> > > I'll take a look at this tomorrow.
> > > Gp
> > >
> > > On Tue, May 15, 2018, 18:28 Artem Ervits <ar...@gmail.com>
> wrote:
> > >
> > > > Did you run
> > > > mvn clean install first on the parent directory?
> > > >
> > > > On Tue, May 15, 2018, 11:35 AM purna pradeep <
> purna2pradeep@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks peter,
> > > > >
> > > > > I have tried changing Dspark.version to 2.3.0 and compiled oozie
> I’m
> > > > > getting below error from oozie examples
> > > > >
> > > > >
> > > > > *ERROR] Failed to execute goal on project oozie-examples: Could not
> > > > resolve
> > > > > dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0:
> > > Could
> > > > > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in
> > > > resolution
> > > > > *
> > > > >
> > > > > On Tue, May 15, 2018 at 11:14 AM Peter Cseh <gezapeti@cloudera.com
> >
> > > > wrote:
> > > > >
> > > > > > Oozie has a spark-2 profile that is currently hard-coded to Spark
> > > 2.1:
> > > > > > https://github.com/apache/oozie/blob/master/pom.xml#L1983
> > > > > > I'm sure if you overwrite the -Dspark.version and compile Oozie
> > that
> > > > way
> > > > > it
> > > > > > will work.
> > > > > > gp
> > > > > >
> > > > > >
> > > > > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep <
> > > > purna2pradeep@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > Does oozie supports spark 2.3? Or will it even care of the
> spark
> > > > > version
> > > > > > >
> > > > > > > I want to use spark action
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Purna
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Peter Cseh *| Software Engineer
> > > > > > cloudera.com <https://www.cloudera.com>
> > > > > >
> > > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > > >
> > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> > [image:
> > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera>
> [image:
> > > > > Cloudera
> > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > > > ------------------------------
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> *Peter Cseh *| Software Engineer
> cloudera.com <https://www.cloudera.com>
>
> [image: Cloudera] <https://www.cloudera.com/>
>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
>
Re: Spark 2.3 in oozie
Posted by Peter Cseh <ge...@cloudera.com>.
The version of the xml schema has nothing to do with the version of the
component you're using.
Thanks for verifying that -Dspark.scala.binary.verstion=2.11 is required
for compilation with Spark 2.3.0
Oozie does not pull in Spark's Kubernetes artifact.
To make it part of the Oozie Spark sharelib you'll have to include the
spark-kubernetes.jar
<https://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-kubernetes_2.11%7C2.3.0%7Cjar>
in
the sharelib/spark/pom.xml as a compile-time dependency.
gp
On Tue, May 15, 2018 at 9:04 PM, purna pradeep <pu...@gmail.com>
wrote:
> I’m able to compile successfully with after adding this override option
>
> -Dspark.scala.binary.version=2.11
>
> Dspark.version = 2.3.0
>
> But when I’m running a spark action with spark-pi example jar against
> Kubernetes master I’m getting below error in stderr log
>
>
> *Error:Could not load KUBERNETES classes.This copy of spark may not have
> been compiled with Kubernetes support*
>
> Below is my workflow.xml
>
> <*spark xmlns="uri:oozie:spark-action:1.0">*
>
> * <resource-manager>${resourceManager}</resource-manager>*
>
> * <name-node>${nameNode}</name-node>*
>
> * <master>k8s://<***.com></master>*
>
> * <name>Python-Spark-Pi</name>*
>
> * <jar>spark-examples_2.11-2.3.0.jar</jar>*
>
> * <spark-opts>--class org.apache.spark.examples.SparkPi --conf
> spark.executor.instances=2 --conf spark.kubernetes.namespace=spark --conf
> spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf
> spark.kubernetes.container.image=artifactory.cloud.
> capitalone.com/kubespark/spark-quantum:v2.3.0
> <http://artifactory.cloud.capitalone.com/kubespark/spark-quantum:v2.3.0>
> --conf spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true
> <http://spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true
> >
> --conf
> spark.kubernetes.driver.label.application=is1-driver --conf
> spark.kubernetes.executor.label.application=is1-exec*utor
> local:///opt/spark/examples/jars/spark-examples_2.11-2.3.
> 0.jar</spark-opts>
>
> </spark>
>
>
> Is this because of uri:oozie:spark-action:1.0 in spark xml tag? Does it
> needs to be spark-action:2.0 as I’m using spark 2.3?
>
>
> Please suggest!
>
>
> On Tue, May 15, 2018 at 12:43 PM Peter Cseh <ge...@cloudera.com> wrote:
>
> > I think the error is related to the Scala version being present in the
> > artifact name.
> > I'll take a look at this tomorrow.
> > Gp
> >
> > On Tue, May 15, 2018, 18:28 Artem Ervits <ar...@gmail.com> wrote:
> >
> > > Did you run
> > > mvn clean install first on the parent directory?
> > >
> > > On Tue, May 15, 2018, 11:35 AM purna pradeep <pu...@gmail.com>
> > > wrote:
> > >
> > > > Thanks peter,
> > > >
> > > > I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m
> > > > getting below error from oozie examples
> > > >
> > > >
> > > > *ERROR] Failed to execute goal on project oozie-examples: Could not
> > > resolve
> > > > dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0:
> > Could
> > > > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in
> > > resolution
> > > > *
> > > >
> > > > On Tue, May 15, 2018 at 11:14 AM Peter Cseh <ge...@cloudera.com>
> > > wrote:
> > > >
> > > > > Oozie has a spark-2 profile that is currently hard-coded to Spark
> > 2.1:
> > > > > https://github.com/apache/oozie/blob/master/pom.xml#L1983
> > > > > I'm sure if you overwrite the -Dspark.version and compile Oozie
> that
> > > way
> > > > it
> > > > > will work.
> > > > > gp
> > > > >
> > > > >
> > > > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep <
> > > purna2pradeep@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > Does oozie supports spark 2.3? Or will it even care of the spark
> > > > version
> > > > > >
> > > > > > I want to use spark action
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Purna
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Peter Cseh *| Software Engineer
> > > > > cloudera.com <https://www.cloudera.com>
> > > > >
> > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > >
> > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> [image:
> > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > > Cloudera
> > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > > ------------------------------
> > > > >
> > > >
> > >
> >
>
--
*Peter Cseh *| Software Engineer
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
Re: Spark 2.3 in oozie
Posted by purna pradeep <pu...@gmail.com>.
I’m able to compile successfully with after adding this override option
-Dspark.scala.binary.version=2.11
Dspark.version = 2.3.0
But when I’m running a spark action with spark-pi example jar against
Kubernetes master I’m getting below error in stderr log
*Error:Could not load KUBERNETES classes.This copy of spark may not have
been compiled with Kubernetes support*
Below is my workflow.xml
<*spark xmlns="uri:oozie:spark-action:1.0">*
* <resource-manager>${resourceManager}</resource-manager>*
* <name-node>${nameNode}</name-node>*
* <master>k8s://<***.com></master>*
* <name>Python-Spark-Pi</name>*
* <jar>spark-examples_2.11-2.3.0.jar</jar>*
* <spark-opts>--class org.apache.spark.examples.SparkPi --conf
spark.executor.instances=2 --conf spark.kubernetes.namespace=spark --conf
spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf
spark.kubernetes.container.image=artifactory.cloud.capitalone.com/kubespark/spark-quantum:v2.3.0
<http://artifactory.cloud.capitalone.com/kubespark/spark-quantum:v2.3.0>
--conf spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true
<http://spark.kubernetes.node.selector.node-role.kubernetes.io/worker=true>
--conf
spark.kubernetes.driver.label.application=is1-driver --conf
spark.kubernetes.executor.label.application=is1-exec*utor
local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar</spark-opts>
</spark>
Is this because of uri:oozie:spark-action:1.0 in spark xml tag? Does it
needs to be spark-action:2.0 as I’m using spark 2.3?
Please suggest!
On Tue, May 15, 2018 at 12:43 PM Peter Cseh <ge...@cloudera.com> wrote:
> I think the error is related to the Scala version being present in the
> artifact name.
> I'll take a look at this tomorrow.
> Gp
>
> On Tue, May 15, 2018, 18:28 Artem Ervits <ar...@gmail.com> wrote:
>
> > Did you run
> > mvn clean install first on the parent directory?
> >
> > On Tue, May 15, 2018, 11:35 AM purna pradeep <pu...@gmail.com>
> > wrote:
> >
> > > Thanks peter,
> > >
> > > I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m
> > > getting below error from oozie examples
> > >
> > >
> > > *ERROR] Failed to execute goal on project oozie-examples: Could not
> > resolve
> > > dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0:
> Could
> > > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in
> > resolution
> > > *
> > >
> > > On Tue, May 15, 2018 at 11:14 AM Peter Cseh <ge...@cloudera.com>
> > wrote:
> > >
> > > > Oozie has a spark-2 profile that is currently hard-coded to Spark
> 2.1:
> > > > https://github.com/apache/oozie/blob/master/pom.xml#L1983
> > > > I'm sure if you overwrite the -Dspark.version and compile Oozie that
> > way
> > > it
> > > > will work.
> > > > gp
> > > >
> > > >
> > > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep <
> > purna2pradeep@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > Does oozie supports spark 2.3? Or will it even care of the spark
> > > version
> > > > >
> > > > > I want to use spark action
> > > > >
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Purna
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Peter Cseh *| Software Engineer
> > > > cloudera.com <https://www.cloudera.com>
> > > >
> > > > [image: Cloudera] <https://www.cloudera.com/>
> > > >
> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > Cloudera
> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > ------------------------------
> > > >
> > >
> >
>
Re: Spark 2.3 in oozie
Posted by Peter Cseh <ge...@cloudera.com>.
I think the error is related to the Scala version being present in the
artifact name.
I'll take a look at this tomorrow.
Gp
On Tue, May 15, 2018, 18:28 Artem Ervits <ar...@gmail.com> wrote:
> Did you run
> mvn clean install first on the parent directory?
>
> On Tue, May 15, 2018, 11:35 AM purna pradeep <pu...@gmail.com>
> wrote:
>
> > Thanks peter,
> >
> > I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m
> > getting below error from oozie examples
> >
> >
> > *ERROR] Failed to execute goal on project oozie-examples: Could not
> resolve
> > dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: Could
> > not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in
> resolution
> > *
> >
> > On Tue, May 15, 2018 at 11:14 AM Peter Cseh <ge...@cloudera.com>
> wrote:
> >
> > > Oozie has a spark-2 profile that is currently hard-coded to Spark 2.1:
> > > https://github.com/apache/oozie/blob/master/pom.xml#L1983
> > > I'm sure if you overwrite the -Dspark.version and compile Oozie that
> way
> > it
> > > will work.
> > > gp
> > >
> > >
> > > On Tue, May 15, 2018 at 5:07 PM, purna pradeep <
> purna2pradeep@gmail.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > Does oozie supports spark 2.3? Or will it even care of the spark
> > version
> > > >
> > > > I want to use spark action
> > > >
> > > >
> > > >
> > > > Thanks,
> > > > Purna
> > > >
> > >
> > >
> > >
> > > --
> > > *Peter Cseh *| Software Engineer
> > > cloudera.com <https://www.cloudera.com>
> > >
> > > [image: Cloudera] <https://www.cloudera.com/>
> > >
> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > Cloudera
> > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > ------------------------------
> > >
> >
>
Re: Spark 2.3 in oozie
Posted by Artem Ervits <ar...@gmail.com>.
Did you run
mvn clean install first on the parent directory?
On Tue, May 15, 2018, 11:35 AM purna pradeep <pu...@gmail.com>
wrote:
> Thanks peter,
>
> I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m
> getting below error from oozie examples
>
>
> *ERROR] Failed to execute goal on project oozie-examples: Could not resolve
> dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: Could
> not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in resolution
> *
>
> On Tue, May 15, 2018 at 11:14 AM Peter Cseh <ge...@cloudera.com> wrote:
>
> > Oozie has a spark-2 profile that is currently hard-coded to Spark 2.1:
> > https://github.com/apache/oozie/blob/master/pom.xml#L1983
> > I'm sure if you overwrite the -Dspark.version and compile Oozie that way
> it
> > will work.
> > gp
> >
> >
> > On Tue, May 15, 2018 at 5:07 PM, purna pradeep <pu...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > Does oozie supports spark 2.3? Or will it even care of the spark
> version
> > >
> > > I want to use spark action
> > >
> > >
> > >
> > > Thanks,
> > > Purna
> > >
> >
> >
> >
> > --
> > *Peter Cseh *| Software Engineer
> > cloudera.com <https://www.cloudera.com>
> >
> > [image: Cloudera] <https://www.cloudera.com/>
> >
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera
> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > ------------------------------
> >
>
Re: Spark 2.3 in oozie
Posted by purna pradeep <pu...@gmail.com>.
Thanks peter,
I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m
getting below error from oozie examples
*ERROR] Failed to execute goal on project oozie-examples: Could not resolve
dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: Could
not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in resolution *
On Tue, May 15, 2018 at 11:14 AM Peter Cseh <ge...@cloudera.com> wrote:
> Oozie has a spark-2 profile that is currently hard-coded to Spark 2.1:
> https://github.com/apache/oozie/blob/master/pom.xml#L1983
> I'm sure if you overwrite the -Dspark.version and compile Oozie that way it
> will work.
> gp
>
>
> On Tue, May 15, 2018 at 5:07 PM, purna pradeep <pu...@gmail.com>
> wrote:
>
> > Hello,
> >
> > Does oozie supports spark 2.3? Or will it even care of the spark version
> >
> > I want to use spark action
> >
> >
> >
> > Thanks,
> > Purna
> >
>
>
>
> --
> *Peter Cseh *| Software Engineer
> cloudera.com <https://www.cloudera.com>
>
> [image: Cloudera] <https://www.cloudera.com/>
>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
>
Re: Spark 2.3 in oozie
Posted by Peter Cseh <ge...@cloudera.com>.
Oozie has a spark-2 profile that is currently hard-coded to Spark 2.1:
https://github.com/apache/oozie/blob/master/pom.xml#L1983
I'm sure if you overwrite the -Dspark.version and compile Oozie that way it
will work.
gp
On Tue, May 15, 2018 at 5:07 PM, purna pradeep <pu...@gmail.com>
wrote:
> Hello,
>
> Does oozie supports spark 2.3? Or will it even care of the spark version
>
> I want to use spark action
>
>
>
> Thanks,
> Purna
>
--
*Peter Cseh *| Software Engineer
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------