You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by roshan joe <im...@gmail.com> on 2017/11/03 13:01:07 UTC
Ignite with Spark on Yarn
Below are the steps I followed for ignite with spark on Yarn but I am stuck
after spending almost all day on this.
- copied apache-ignite-fabric-2.1.0-bin (tried 2.3.0 initially but the
it didnt work) to all the 4 slave nodes of Spark cluster and started ignite
with ./bin/ignite.sh. It shows as given below, after starting
*Topology snapshot [ver=10, servers=4, clients=0, CPUs=32, heap=4.0GB]*
- updated the spark-env.sh on all slave nodes with IGNITE_HOME and class
paths as below
IGNITE_HOME=/home/ec2-user/rchalil/apache-ignite-fabric-2.1.0-bin
IGNITE_LIBS="${IGNITE_HOME}/libs/*"
for file in ${IGNITE_HOME}/libs/*
do
if [ -d ${file} ] && [ "${file}" != "${IGNITE_HOME}"/libs/optional ];
then
IGNITE_LIBS=${IGNITE_LIBS}:${file}/*
fi
done
export spark.driver.extraClassPath=$IGNITE_LIBS
export spark.executor.extraClassPath=$IGNITE_LIBS
- updated the default config.xml as attached with the static ips
- On the Spark Master node (Only), added the - <property name=
"clientMode" value="true"/>
In the simple test code, i tried 2 things, both without success:
1. Added config/default-config.xml directly under the project and added the
below snippet.
val CONFIG = "config/default-config.xml"
val igniteContext = new IgniteContext(sparkContext, CONFIG, true)
val sharedRDD: IgniteRDD[Int, Int] = igniteContext.fromCache[Int,
Int]("sharedRDD")
sharedRDD.savePairs(sparkContext.parallelize(1 to 100000, 10).map(i => (i, i)))
sharedRDD.mapValues(x => (x * x))
During execution, it complains that - class
org.apache.ignite.IgniteCheckedException:
Spring XML configuration path is invalid: config/default-config.xml. Note
that this path should be either absolute or a relative local file system
path, relative to META-INF in classpath or valid URL to IGNITE_HOME.
2. Commented the above snippet and used the below:
val ic = new IgniteContext(sparkContext, () => new IgniteConfiguration())
val sharedRDD: IgniteRDD[Int, Int] = ic.fromCache("partitioned")
sharedRDD.savePairs(sparkContext.parallelize(1 to 100000, 10).map(i => (i, i)))
Now it complains that "it cannot find any IPs from multicast or IPFinder.
Not sure what am i doing not right. I also have a cluster.properties file
but I am not sure if it is needed for the yarn setup and I am not
explicitly calling it anywhere. Thank you.
Re: Ignite with Spark on Yarn
Posted by Denis Magda <dm...@apache.org>.
> On Nov 5, 2017, at 10:42 PM, roshan joe <im...@gmail.com> wrote:
>
> nested exception is java.io.FileNotFoundException: /home/ec2-user/apache-ignite-fabric-2.1.0-bin/config/default-config.xml (Permission denied)]
The Spark process could locate the configuration file but doesn’t have enough permissions to work with it - see permission denied message. Grant all the required permissions to the file and possible to parent directories.
—
Denis
Re: Ignite with Spark on Yarn
Posted by roshan joe <im...@gmail.com>.
I still couldn't get it working on Yarn with spark-submit. Looking at the
logs, the initial reading of default-config.xml succeeds and the ignite
node is started OK, but is failing when it starts the Task. Please see
below.
NB: I tried spark-submit from Spark Master node as well as the Spark client
nodes but seeing the same error in both the cases. Is there a way to get
this working on yarn with spark-submit?
17/11/06 06:06:26 main INFO XmlBeanDefinitionReader: Loading XML bean
definitions from URL
[file:/home/ec2-user/apache-ignite-fabric-2.1.0-bin/config/default-config.xml]
17/11/06 06:06:26 main INFO GenericApplicationContext: Refreshing
org.springframework.context.support.GenericApplicationContext@77b27b57:
startup date [Mon Nov 06 06:06:26 UTC 2017]; root of context hierarchy
[06:06:26] __________ ________________
[06:06:26] / _/ ___/ |/ / _/_ __/ __/
[06:06:26] _/ // (7 7 // / / / / _/
[06:06:26] /___/\___/_/|_/___/ /_/ /___/
[06:06:26]
[06:06:26] ver. 2.1.0#20170720-sha1:a6ca5c8a
[06:06:26] 2017 Copyright(C) Apache Software Foundation
[06:06:26]
[06:06:26] Ignite documentation: http://ignite.apache.org
[06:06:26]
[06:06:26] Quiet mode.
[06:06:26] ^-- Logging to file '/home/ec2-user/
apache-ignite-fabric-2.1.0-bin/work/log/ignite-bd37708a.0.log'
[06:06:26] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false
or "-v" to ignite.{sh|bat}
[06:06:26]
[06:06:26] OS: Linux 4.9.43-17.38.amzn1.x86_64 amd64
[06:06:26] VM information: Java(TM) SE Runtime Environment 1.8.0_60-b27
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.60-b23
[06:06:27] Configured plugins:
[06:06:27] ^-- None
[06:06:27]
[06:06:27] Message queue limit is set to 0 which may lead to potential
OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due
to message queues growth on sender and receiver sides.
[06:06:27] Security status [authentication=off, tls/ssl=off]
[06:06:27] REST protocols do not start on client node. To start the
protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system
property.
[06:06:28] Performance suggestions for grid (fix if possible)
[06:06:28] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[06:06:28] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM
options)
[06:06:28] ^-- Set max direct memory size if getting 'OOME: Direct buffer
memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
[06:06:28] ^-- Disable processing of calls to System.gc() (add
'-XX:+DisableExplicitGC' to JVM options)
[06:06:28] Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[06:06:28]
[06:06:28] To start Console Management & Monitoring run
ignitevisorcmd.{sh|bat}
[06:06:28]
*[06:06:28] Ignite node started OK (id=bd37708a)*
*[06:06:28] Topology snapshot [ver=112, servers=4, clients=2, CPUs=40,
heap=13.0GB]*
*17/11/06 06:06:29 dispatcher-event-loop-4 INFO TaskSetManager: Starting
task 1.0 in stage 0.0 (TID 1, ip-ç-162.us-west-2.compute.internal, executor
1, partition 1, PROCESS_LOCAL, 6495 bytes)*
*17/11/06 06:06:29 task-result-getter-0 WARN TaskSetManager: Lost task 0.0
in stage 0.0 (TID 0, ip-10-xx-xx-162.us-west-2.compute.internal, executor
1): class org.apache.ignite.IgniteCheckedException: Failed to instantiate
Spring XML application context
[springUrl=file:/home/ec2-user/apache-ignite-fabric-2.1.0-bin/config/default-config.xml,
err=IOException parsing XML document from URL
[file:/home/ec2-user/apache-ignite-fabric-2.1.0-bin/config/default-config.xml];
nested exception is java.io.FileNotFoundException:
/home/ec2-user/apache-ignite-fabric-2.1.0-bin/config/default-config.xml
(Permission denied)]*
On Fri, Nov 3, 2017 at 5:21 PM, roshan joe <im...@gmail.com> wrote:
> Thanks Alexey. I believe I have tried all the 4 methods you suggested
> below without any success.
>
> 1. Specified the absolute path of the default-config.xml using the
> file:/// as well as hdfs:///
>
> 2. passed the config file as file parameter to spark-submit as
> --files=file:///.../..../apache-ignite-fabric-2.1.0-
> bin/config/default-config.xml
>
> 3. exported IGNITE_HOME=/..../..../apache-ignite-fabric-2.1.0-bin/ in all
> the spark nodes and passed "config/default-config.xml" as relative path
>
> 4. created the config file under "src/resources/" but no luck.
>
> Please see the complete logs.
>
> Also, I have the default-config.xml file at the default location on all
> spark nodes. Do I need to pass it in the code then? If I do not pass it, ""it
> cannot find any IPs from multicast or IPFinder. "
>
>
> Thank you.
>
>
>
> On Fri, Nov 3, 2017 at 6:52 AM, Alexey Kukushkin <
> kukushkinalexey@gmail.com> wrote:
>
>> Hi,
>>
>> The problem is Ignite cannot find your configuration file.
>>
>> Ignite tries these 3 steps to find configuration file:
>>
>> 1. Try to resolve it as a URI
>> 2. If previous step fails, try to resolve it as $IGNITE_HOME + <the
>> path you specified>
>> 3. If previous step fails, try to resolve it in CLASSPATH.
>>
>> Thus, you have multiple options to specify config path:
>>
>> 1. Specify an absolute path
>> 2. Pass the config path as a command line parameter
>> 3. Create your path relative to $IGNITE_HOME
>> 4. If you use maven or gradle java plugin, then you can create file
>> under "resources/path-to-ignite-config.xml". On the "process
>> resources" phase maven will copy it under target/classes/path-to-ignite-config.xml"
>> and then it will go into JAR. Your app will find it by
>> "path-to-ignite-config.xml" since it will be in CLASSPATH.
>>
>>
>>
>
>
Re: Ignite with Spark on Yarn
Posted by roshan joe <im...@gmail.com>.
Thanks Alexey. I believe I have tried all the 4 methods you suggested below
without any success.
1. Specified the absolute path of the default-config.xml using the file:///
as well as hdfs:///
2. passed the config file as file parameter to spark-submit as
--files=file:///.../..../apache-ignite-fabric-2.1.0-bin/config/default-config.xml
3. exported IGNITE_HOME=/..../..../apache-ignite-fabric-2.1.0-bin/ in all
the spark nodes and passed "config/default-config.xml" as relative path
4. created the config file under "src/resources/" but no luck.
Please see the complete logs.
Also, I have the default-config.xml file at the default location on all
spark nodes. Do I need to pass it in the code then? If I do not pass it, ""it
cannot find any IPs from multicast or IPFinder. "
Thank you.
On Fri, Nov 3, 2017 at 6:52 AM, Alexey Kukushkin <ku...@gmail.com>
wrote:
> Hi,
>
> The problem is Ignite cannot find your configuration file.
>
> Ignite tries these 3 steps to find configuration file:
>
> 1. Try to resolve it as a URI
> 2. If previous step fails, try to resolve it as $IGNITE_HOME + <the
> path you specified>
> 3. If previous step fails, try to resolve it in CLASSPATH.
>
> Thus, you have multiple options to specify config path:
>
> 1. Specify an absolute path
> 2. Pass the config path as a command line parameter
> 3. Create your path relative to $IGNITE_HOME
> 4. If you use maven or gradle java plugin, then you can create file
> under "resources/path-to-ignite-config.xml". On the "process
> resources" phase maven will copy it under target/classes/path-to-ignite-config.xml"
> and then it will go into JAR. Your app will find it by
> "path-to-ignite-config.xml" since it will be in CLASSPATH.
>
>
>
Re: Ignite with Spark on Yarn
Posted by Alexey Kukushkin <ku...@gmail.com>.
Hi,
The problem is Ignite cannot find your configuration file.
Ignite tries these 3 steps to find configuration file:
1. Try to resolve it as a URI
2. If previous step fails, try to resolve it as $IGNITE_HOME + <the path
you specified>
3. If previous step fails, try to resolve it in CLASSPATH.
Thus, you have multiple options to specify config path:
1. Specify an absolute path
2. Pass the config path as a command line parameter
3. Create your path relative to $IGNITE_HOME
4. If you use maven or gradle java plugin, then you can create file
under "resources/path-to-ignite-config.xml". On the "process resources"
phase maven will copy it under target/classes/path-to-ignite-config.xml"
and then it will go into JAR. Your app will find it by
"path-to-ignite-config.xml" since it will be in CLASSPATH.