You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by mdolgonos <mi...@citi.com> on 2016/04/22 15:57:46 UTC
Ignite Installation with Spark under CDH
I'm trying to install and integrate Ignite with Spark under CDH by following
the recommendation at
https://apacheignite-fs.readme.io/docs/installation-deployment for
Standalone Deployment. I modified the existing $SPARK_HOME/conf/spark-env.sh
as recommended but the following command still can't recognize the ignite
package
import org.apache.ignite.configuration._
Can somebody advise if I'm missing anything?
Thank you,
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Installation with Spark under CDH
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi Michael,
Ok, so it looks like the process didn't have enough heap.
Thank you for your inputs about CDH configuration. We will improve our
documentation based on this.
Vladimir
On Thu, Apr 28, 2016 at 5:15 PM, mdolgonos <mi...@citi.com>
wrote:
> Vladimir,
>
> I fixed this by changing the way I start Ignite based on a recommendation
> from another post here and the OOME has gone:
> ignite.sh -J-Xmx10g
> The data that I put in cache is about 1.5GB
>
> Thank you,
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457p4665.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Ignite Installation with Spark under CDH
Posted by mdolgonos <mi...@citi.com>.
Vladimir,
I fixed this by changing the way I start Ignite based on a recommendation
from another post here and the OOME has gone:
ignite.sh -J-Xmx10g
The data that I put in cache is about 1.5GB
Thank you,
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457p4665.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Installation with Spark under CDH
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi Michael,
Did you have a chance to analyze heap dump to understand what caused OOME?
As per *IgniteConfiguration*, it is made non-serialziable intentionally,
because we do not expect it to be passed over the wire. Could you please
provide a stack trace where you see it is being serialized?
Vladimir.
On Wed, Apr 27, 2016 at 10:31 PM, mdolgonos <mi...@citi.com>
wrote:
> Vladimir,
>
> Update - I think I solved the ClassNotFound exception. It looks like the
> Ignite installation document for Spark and CDH is outdated and doesn't
> contain complete information on integrating Ignite with Spark running in a
> 'Yarn' (cluster) mode on CDH which I have. This is what I have done and now
> am able to run my Spark program (however, with exception described below):
> went to
> Cloudera Manager -> YARN (MR2 Included) -> Configuration -> Service Wide ->
> Advanced -> Spark Client Advanced Configuration Snippet (Safety Valve) for
> spark-conf/spark-defaults.conf
> Added etc/ignite-fabric-1.5.0/libs/* to already present
>
> spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/jars/htrace-core-3.2.0-incubating.jar.
> The combined line looks like this:
>
> spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/jars/htrace-core-3.2.0-incubating.jar:/etc/ignite-fabric-1.5.0/libs/*
>
> However, after starting Ignite and submitting my Spark program (in another
> window) I see the following exception:
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> The only line of code where I put something into the Ignite cache where I'm
> trying to save RDD there:
> cacheIgRDD.savePairs(partRDD, true)
>
> One thing to note is it looks like IgniteConfigurationis not Serializable
> so
> I had to create MyIgniteConfiguration extends IgniteConfiguration with
> Serializable
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457p4624.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Ignite Installation with Spark under CDH
Posted by mdolgonos <mi...@citi.com>.
Vladimir,
Update - I think I solved the ClassNotFound exception. It looks like the
Ignite installation document for Spark and CDH is outdated and doesn't
contain complete information on integrating Ignite with Spark running in a
'Yarn' (cluster) mode on CDH which I have. This is what I have done and now
am able to run my Spark program (however, with exception described below):
went to
Cloudera Manager -> YARN (MR2 Included) -> Configuration -> Service Wide ->
Advanced -> Spark Client Advanced Configuration Snippet (Safety Valve) for
spark-conf/spark-defaults.conf
Added etc/ignite-fabric-1.5.0/libs/* to already present
spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/jars/htrace-core-3.2.0-incubating.jar.
The combined line looks like this:
spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/jars/htrace-core-3.2.0-incubating.jar:/etc/ignite-fabric-1.5.0/libs/*
However, after starting Ignite and submitting my Spark program (in another
window) I see the following exception:
java.lang.OutOfMemoryError: GC overhead limit exceeded
The only line of code where I put something into the Ignite cache where I'm
trying to save RDD there:
cacheIgRDD.savePairs(partRDD, true)
One thing to note is it looks like IgniteConfigurationis not Serializable so
I had to create MyIgniteConfiguration extends IgniteConfiguration with
Serializable
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457p4624.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Installation with Spark under CDH
Posted by mdolgonos <mi...@citi.com>.
Vladimir,
Unless I'm missing something there is no console to echo it to since in CDH
all services start by Cloudera Manager. The only console I have is the one I
execute spark-submit in but it doesn't have any Spark nor CDH related
properties.
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457p4611.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Installation with Spark under CDH
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi Michael,
I meant to echo to the console SPARK_CLASSPATH variable which you created
in *spark-env.sh* following recommendations from Ignite docs. We need to
ensure that JCache jar is included into it.
BTW, do you see Ignite jars in " java.class.path" property?
Vladimir.
On Wed, Apr 27, 2016 at 4:30 PM, mdolgonos <mi...@citi.com>
wrote:
> Vladimir,
>
> I don't see this property. I put the following code at the very beginning
> of
> my main() method:
> println("**** Props=" + System.getProperties())
> The only classpath related property I see is java.class.path which doesn't
> contain cache-api-1.0.0.jar despite my changing spark-env.sh as described
> in
> the documentation.
>
> Thank you
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457p4608.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Ignite Installation with Spark under CDH
Posted by mdolgonos <mi...@citi.com>.
Vladimir,
I don't see this property. I put the following code at the very beginning of
my main() method:
println("**** Props=" + System.getProperties())
The only classpath related property I see is java.class.path which doesn't
contain cache-api-1.0.0.jar despite my changing spark-env.sh as described in
the documentation.
Thank you
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457p4608.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Installation with Spark under CDH
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi Michael,
Could you please print resulting value of the SPARK_CLASSPATH variable?
Vladimir.
On Tue, Apr 26, 2016 at 8:55 PM, mdolgonos <mi...@citi.com>
wrote:
> Vladimir,
> I verified that the cache jar is in the Cloudera jars directory. All the
> cache packages are also included in the deployment jar-with-dependencies as
> I used
> <dependency>
> <groupId>org.apache.ignite</groupId>
> <artifactId>ignite-spark</artifactId>
> <version>${ignite.version}</version>
> <scope>compile</scope>
> </dependency>
> <dependency>
> <groupId>javax.cache</groupId>
> <artifactId>cache-api</artifactId>
> <version>1.0.0</version>
> <scope>compile</scope>
> </dependency>
> Not sure what else I can take a look at.
> Thank you again for your help.
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457p4558.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Ignite Installation with Spark under CDH
Posted by mdolgonos <mi...@citi.com>.
Vladimir,
I verified that the cache jar is in the Cloudera jars directory. All the
cache packages are also included in the deployment jar-with-dependencies as
I used
<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-spark</artifactId>
<version>${ignite.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>javax.cache</groupId>
<artifactId>cache-api</artifactId>
<version>1.0.0</version>
<scope>compile</scope>
</dependency>
Not sure what else I can take a look at.
Thank you again for your help.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457p4558.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Installation with Spark under CDH
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi,
This is pretty hard to say what is the root cause, especially in complex
deployments like CDH. Most probably you JAR is packaged incorrectly because
your application is able to load Ignite classes, but cannot load jcache
API.
Could you try to simply put cache-api-1.0.0.jar to all places and scripts
where you added Igntte JARs and/or your jar-with-dependencies?
Vladimir.
On Tue, Apr 26, 2016 at 12:43 AM, mdolgonos <mi...@citi.com>
wrote:
> Vladimir,
> There are 2 things that I'm experiencing so far:
> 1. I have added the following code to spark-env.sh in my CDH installation
> IGNITE_HOME=/etc/ignite-1.5.0
> IGNITE_LIBS="${IGNITE_HOME}/libs/*"
>
> for file in ${IGNITE_HOME}/libs/*
> do
> if [ -d ${file} ] && [ "${file}" != "${IGNITE_HOME}"/libs/optional ];
> then
> IGNITE_LIBS=${IGNITE_LIBS}:${file}/*
> fi
> done
>
> but Ignite jars are still not recognized by Spark after restarting Spark as
> well as the entire CDH
> . My location of CDH is the default one:
> /opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/etc/spark/conf.dist
> So I decided to compile my code into a jar-with-dependencies which led me
> to
> the second issue:
>
> 2. Looks like the jars were discovered by spark-submit, however, now I'm
> getting the following exception:
> Exception in thread "main" java.lang.NoClassDefFoundError:
> javax/cache/configuration/MutableConfiguration
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.marshallerSystemCache(IgnitionEx.java:2098)
> at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.initializeDefaultCacheConfiguration(IgnitionEx.java:1914)
> at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.initializeConfiguration(IgnitionEx.java:1899)
> at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1573)
> at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1547)
> at
> org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:534)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:515)
> at org.apache.ignite.Ignition.start(Ignition.java:322)
> at
> org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:153)
> at
> org.apache.ignite.spark.IgniteContext.<init>(IgniteContext.scala:62)
> at
> training.PnLMergerVectorIgnite$.main(PnLMergerVectorIgnite.scala:50)
> at training.PnLMergerVectorIgnite.main(PnLMergerVectorIgnite.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
>
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
> at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
> at
> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException:
> javax.cache.configuration.MutableConfiguration
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> I know that the class javax.cache.configuration.MutableConfiguration
> belongs
> to cache-api-1.0.0.jar and I see it present in my jar-with-dependencies
>
> Thank you for looking at this.
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457p4507.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Ignite Installation with Spark under CDH
Posted by mdolgonos <mi...@citi.com>.
Vladimir,
There are 2 things that I'm experiencing so far:
1. I have added the following code to spark-env.sh in my CDH installation
IGNITE_HOME=/etc/ignite-1.5.0
IGNITE_LIBS="${IGNITE_HOME}/libs/*"
for file in ${IGNITE_HOME}/libs/*
do
if [ -d ${file} ] && [ "${file}" != "${IGNITE_HOME}"/libs/optional ];
then
IGNITE_LIBS=${IGNITE_LIBS}:${file}/*
fi
done
but Ignite jars are still not recognized by Spark after restarting Spark as
well as the entire CDH
. My location of CDH is the default one:
/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/etc/spark/conf.dist
So I decided to compile my code into a jar-with-dependencies which led me to
the second issue:
2. Looks like the jars were discovered by spark-submit, however, now I'm
getting the following exception:
Exception in thread "main" java.lang.NoClassDefFoundError:
javax/cache/configuration/MutableConfiguration
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.marshallerSystemCache(IgnitionEx.java:2098)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.initializeDefaultCacheConfiguration(IgnitionEx.java:1914)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.initializeConfiguration(IgnitionEx.java:1899)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1573)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1547)
at
org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:534)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:515)
at org.apache.ignite.Ignition.start(Ignition.java:322)
at
org.apache.ignite.spark.IgniteContext.ignite(IgniteContext.scala:153)
at
org.apache.ignite.spark.IgniteContext.<init>(IgniteContext.scala:62)
at
training.PnLMergerVectorIgnite$.main(PnLMergerVectorIgnite.scala:50)
at training.PnLMergerVectorIgnite.main(PnLMergerVectorIgnite.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
javax.cache.configuration.MutableConfiguration
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
I know that the class javax.cache.configuration.MutableConfiguration belongs
to cache-api-1.0.0.jar and I see it present in my jar-with-dependencies
Thank you for looking at this.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457p4507.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Installation with Spark under CDH
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi,
Could you please clarify the exact problem? Do you see any exception?
Vladimir.
On Fri, Apr 22, 2016 at 4:57 PM, mdolgonos <mi...@citi.com>
wrote:
> I'm trying to install and integrate Ignite with Spark under CDH by
> following
> the recommendation at
> https://apacheignite-fs.readme.io/docs/installation-deployment for
> Standalone Deployment. I modified the existing
> $SPARK_HOME/conf/spark-env.sh
> as recommended but the following command still can't recognize the ignite
> package
> import org.apache.ignite.configuration._
> Can somebody advise if I'm missing anything?
> Thank you,
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-Installation-with-Spark-under-CDH-tp4457.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>