You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Murugar <gn...@gmail.com> on 2018/05/17 11:31:16 UTC
Ignite-yarn[Ignition] uses almost all memory and vcores available
in YARN cluster
Hi,
Ignite-yarn[Ignition] uses almost all memory and vcores available in YARN
cluster. The allocated memory to the Ignition server was 40GB and 16 VCores.
*When we checked the actual usage with yarn top command the Ignition server
is using 380 GB and 140 VCores!!! * When we checked the cache, it is having
only less number of records - 3000 records.
*Information about the application:*
We load data into Ignite Cache, process them periodically and delete
processed records periodically.
We are running Ignite-yarn in Cloudera cluster with 5 data nodes with below
cluster.properties
IGNITE_NODE_COUNT=4
IGNITE_RUN_CPU_PER_NODE=4
IGNITE_MEMORY_PER_NODE=10096
IGNITE_VERSION=2.3.0
IGNITE_PATH=/apache-ignite-fabric-2.3.0-bin.zip
IGNITE_RELEASES_DIR=/releases
IGNITE_WORKING_DIR=/work
IGNITE_XML_CONFIG=/ignite-config.xml
IGNITE_USERS_LIBS=/libs
IGNITE_LOCAL_WORK_DIR=/home/user
IGNITE_JVM_OPTS="-server -Xms10g -Xmx10g -XX:+AlwaysPreTouch -XX:+UseG1GC
-XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC"
Please find our Ignite-config.xml below
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util-2.0.xsd">
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="clientMode" value="false"/>
<property name="peerClassLoadingEnabled" value="true"/>
<property name="networkTimeout" value="100000"/>
<property name="networkSendRetryCount" value="5"/>
<property name="discoverySpi">
<bean
class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>ipaddress:47500</value>
</list>
</property>
</bean>
</property>
<property name="networkTimeout"
value="10000000"/>
<property name="joinTimeout"
value="10000000"/>
<property name="maxAckTimeout"
value="10000000"/>
<property name="reconnectCount" value="50"/>
<property name="socketTimeout"
value="10000000"/>
</bean>
</property>
</bean>
</beans>
Cache Configuration:
var userGridConf: CacheConfiguration[Long, User] = new
CacheConfiguration[Long, User](userCacheName)
userGridConf.setCacheMode(CacheMode.REPLICATED)
userGridConf.setEagerTtl(false)
userGridConf.setOnheapCacheEnabled(false)
userGridConf.setBackups(0)
userGridConf.setIndexedTypes(classOf[Long], classOf[User])
val cache = ignite.getOrCreateCache(userGridConf)
cache.size(CachePeekMode.OFFHEAP)
Could you please let me know what is wrong with the above configurations?
Thanks in Advance!!!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite-yarn[Ignition] uses almost all memory and vcores available
in YARN cluster
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Since we don't seem to have much experience in running YARN here, can you
please elaborate on this problem? Is the number of consumed resources wrong
or are resources actually get consumed by Ignite?
BTW, if you plan to store data inside Ignite, you should also set
IGNITE_MEMORY_OVERHEAD_PER_NODE, see discussion at
http://apache-ignite-developers.2346864.n4.nabble.com/YARN-parameters-and-default-values-td29899.html
Regards,
--
Ilya Kasnacheev
2018-05-17 14:31 GMT+03:00 Murugar <gn...@gmail.com>:
> Hi,
>
> Ignite-yarn[Ignition] uses almost all memory and vcores available in YARN
> cluster. The allocated memory to the Ignition server was 40GB and 16
> VCores.
>
> *When we checked the actual usage with yarn top command the Ignition server
> is using 380 GB and 140 VCores!!! * When we checked the cache, it is having
> only less number of records - 3000 records.
>
> *Information about the application:*
> We load data into Ignite Cache, process them periodically and delete
> processed records periodically.
>
> We are running Ignite-yarn in Cloudera cluster with 5 data nodes with below
> cluster.properties
>
> IGNITE_NODE_COUNT=4
> IGNITE_RUN_CPU_PER_NODE=4
> IGNITE_MEMORY_PER_NODE=10096
> IGNITE_VERSION=2.3.0
> IGNITE_PATH=/apache-ignite-fabric-2.3.0-bin.zip
>
> IGNITE_RELEASES_DIR=/releases
> IGNITE_WORKING_DIR=/work
> IGNITE_XML_CONFIG=/ignite-config.xml
> IGNITE_USERS_LIBS=/libs
> IGNITE_LOCAL_WORK_DIR=/home/user
> IGNITE_JVM_OPTS="-server -Xms10g -Xmx10g -XX:+AlwaysPreTouch -XX:+UseG1GC
> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC"
>
>
> Please find our Ignite-config.xml below
>
> <?xml version="1.0" encoding="UTF-8"?>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:util="http://www.springframework.org/schema/util"
> xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
> http://www.springframework.org/schema/util
> http://www.springframework.org/schema/util/spring-util-2.0.xsd">
> <bean class="org.apache.ignite.configuration.IgniteConfiguration">
> <property name="clientMode" value="false"/>
> <property name="peerClassLoadingEnabled" value="true"/>
> <property name="networkTimeout" value="100000"/>
> <property name="networkSendRetryCount" value="5"/>
> <property name="discoverySpi">
> <bean
> class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="ipFinder">
> <bean
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.
> TcpDiscoveryVmIpFinder">
> <property name="addresses">
> <list>
>
> <value>ipaddress:47500</value>
> </list>
> </property>
> </bean>
> </property>
> <property name="networkTimeout"
> value="10000000"/>
> <property name="joinTimeout"
> value="10000000"/>
> <property name="maxAckTimeout"
> value="10000000"/>
> <property name="reconnectCount"
> value="50"/>
> <property name="socketTimeout"
> value="10000000"/>
> </bean>
> </property>
> </bean>
> </beans>
>
> Cache Configuration:
>
> var userGridConf: CacheConfiguration[Long, User] = new
> CacheConfiguration[Long, User](userCacheName)
> userGridConf.setCacheMode(CacheMode.REPLICATED)
> userGridConf.setEagerTtl(false)
> userGridConf.setOnheapCacheEnabled(false)
> userGridConf.setBackups(0)
> userGridConf.setIndexedTypes(classOf[Long], classOf[User])
> val cache = ignite.getOrCreateCache(userGridConf)
> cache.size(CachePeekMode.OFFHEAP)
>
> Could you please let me know what is wrong with the above configurations?
>
> Thanks in Advance!!!
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>