You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by prasanth <ma...@gmail.com> on 2016/08/19 06:24:09 UTC
ignition on yarn taking up all memory
I started ignite on yarn and then tried to submit a sample spark job. The
ignition job took all available memory and so spark job was in "accepted"
state forever. Spark job never had enough resources to run.
So, I copied the following file to hdfs path /tmp/ignite and gave
IGNITE_XML_CONFIG=/tmp/ignite/cache-settings.xml and then started ignite. It
still takes up all available cluster memory (which is less around 8.5 GB).
How can I manage the memory that Ignite takes up, so that I can start other
jobs?
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="grid.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="cacheConfiguration">
<list>
<bean
class="org.apache.ignite.configuration.CacheConfiguration">
<property name="startSize" value="#{2000 * 1024 *
1024}"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="1"/>
<property name="affinity" ref="cacheAffinity"/>
</bean>
</list>
</property>
</bean>
</beans>
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignition-on-yarn-taking-up-all-memory-tp7161.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: ignition on yarn taking up all memory
Posted by vkulichenko <va...@gmail.com>.
Hi,
Currently this is not possible out of the box. However, instead of providing
the XML file to IgniteContext, you can provide a function that will download
XML from HDFS and load IgniteConfiguration bean from it. Smth like this:
val ic = new IgniteContext(sc, () => {
// Download config XML from HDFS (you will have to write this code by
yourself).
Ignition.loadSpringBean[IgniteConfiguration("/path/to/downloaded/config.xml",
"ignite.cfg")
})
Note that the IgniteConfiguration bean ID has to be specified in the XML:
<bean <b>id="ignite.cfg"*
class="org.apache.ignite.configuration.IgniteConfiguration">
...
</bean>
This function will be invoked on all nodes where IgniteContext is created
(driver and executors) and will properly instantiate Ignite.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignition-on-yarn-taking-up-all-memory-tp7161p7178.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: ignition on yarn taking up all memory
Posted by prasanth <ma...@gmail.com>.
I had to copy config file to all boxes on cluster at same location. can we
use hdfs instead?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignition-on-yarn-taking-up-all-memory-tp7161p7177.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: ignition on yarn taking up all memory
Posted by prasanth <ma...@gmail.com>.
Nevermind, from https://github.com/apache/ignite/pull/740 , it looks like
parameters were taken off.
The following works:
val ic = new IgniteContext(sc, "config/default-config.xml")
Now, when I execute, the following line:
sharedRDD.filter(_._2 < 10).collect()
I get errors:
[Stage 0:> (0 + 2) /
1024]16/08/19 19:02:02 WARN scheduler.TaskSetManager: Lost task 0.0 in stage
0.0 (TID 0, ec2.internal): class org.apache.ignite.IgniteCheckedException:
Spring XML configuration path is invalid: config/default-config.xml. Note
that this path should be either absolute or a relative local file system
path, relative to META-INF in classpath or valid URL to IGNITE_HOME.
at
org.apache.ignite.internal.util.IgniteUtils.resolveSpringUrl(IgniteUtils.java:3604)
at
org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:678)
Do I have to put the default-config.xml on all nodes of the cluster? Can I
put it in HDFS and use
val ic = new IgniteContext(sc, "hdfs:/tmp/config/default-config.xml")
instead and get rid of the above error?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignition-on-yarn-taking-up-all-memory-tp7161p7174.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: ignition on yarn taking up all memory
Posted by prasanth <ma...@gmail.com>.
Cool. That worked. Now the memory is around 5GB.
In spark-shell, when I give
val ic = new IgniteContext[Integer, Integer](sc,
"config/default-config.xml")
I get,
error: org.apache.ignite.spark.IgniteContext does not take type parameters
Which means, something is wrong right?
When I remove the type parameters, I get
ic: org.apache.ignite.spark.IgniteContext =
org.apache.ignite.spark.IgniteContext@208ae393
however, when I try,
val sharedRDD = ic.fromCache("sample")
I get,
sharedRDD: org.apache.ignite.spark.IgniteRDD[Nothing,Nothing] = IgniteRDD[0]
at RDD at IgniteAbstractRDD.scala:32
and then, obviously,
sharedRDD.filter(_._2 < 10).collect()
<console>:41: error: value < is not a member of Nothing
sharedRDD.filter(_._2 < 10).collect()
How to fix this [Nothing, Nothing] problem?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignition-on-yarn-taking-up-all-memory-tp7161p7172.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: ignition on yarn taking up all memory
Posted by Nikolai Tikhonov <nt...@apache.org>.
Hi,
Ignite which running over YARN should occupies memory how set in
IGNITE_MEMORY_PER_NODE property. Could you please share your config file?
On Fri, Aug 19, 2016 at 9:24 AM, prasanth <ma...@gmail.com> wrote:
> I started ignite on yarn and then tried to submit a sample spark job. The
> ignition job took all available memory and so spark job was in "accepted"
> state forever. Spark job never had enough resources to run.
>
> So, I copied the following file to hdfs path /tmp/ignite and gave
> IGNITE_XML_CONFIG=/tmp/ignite/cache-settings.xml and then started ignite.
> It
> still takes up all available cluster memory (which is less around 8.5 GB).
>
> How can I manage the memory that Ignite takes up, so that I can start other
> jobs?
>
> <?xml version="1.0" encoding="UTF-8"?>
>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd">
>
> <bean id="grid.cfg"
> class="org.apache.ignite.configuration.IgniteConfiguration">
> <property name="cacheConfiguration">
> <list>
>
> <bean
> class="org.apache.ignite.configuration.CacheConfiguration">
>
> <property name="startSize" value="#{2000 * 1024 *
> 1024}"/>
> <property name="atomicityMode" value="ATOMIC"/>
> <property name="backups" value="1"/>
> <property name="affinity" ref="cacheAffinity"/>
> </bean>
> </list>
> </property>
> </bean>
> </beans>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/ignition-on-yarn-taking-up-all-memory-tp7161.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>