You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Andrew Fung <af...@medallia.com> on 2018/06/21 21:32:46 UTC
running Apache Ignite in docker with cgroups
I can see from the docs for IgniteConfiguration that some properties
auto-size based on visible OS resources. In docker, "visible" ends up being
the host values, which will exceed any CPU/memory limits applied via
cgroups to the container.
I see on
IgniteConfiguration.DataStorageConfiguration.DataRegionConfiguration the
properties initialSize and maxSize, which I've set substantially lower than
the container memory allocation, which helped avoid being killed due to
exceeding memory limits, but I'm still seeing the oom reaper kick in
occasionally. Am I missing some other configuration value that controls
off-heap memory use?
I've set the DataRegionConfiguration initial/max size to 16GB, the JVM heap
to 8GB, and the container is currently allocated 32GB, which seems more
than enough for OS and other ancillary uses. In case relevant, I've enabled
persistence, and set the relevant CacheConfiguration to partitioned mode
with backups=2 and writeSynchronizationMode=full_sync. Running Ignite
2.5.0, Oracle JDK 8u172.
Thanks!
Andrew.
Re: running Apache Ignite in docker with cgroups
Posted by Andrew Fung <af...@medallia.com>.
One node per container. Configuration below, key values come from env vars.
IGNITE_DATA_REGION_MAX_SIZE_MB=16384
IGNITE_DATA_REGION_MAX_SIZE=$(( $PIPE_IGNITE_DATA_REGION_MAX_SIZE_MB * 1024
* 1024 )) # 17179869184
IGNITE_JVM_OPTS='-Xms8g -Xmx8g'
IGNITE_PERSISTENCE_ENABLED=true
IGNITE_TEST_CACHE_BACKUPS=2
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd"
>
<bean id="grid.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="dataStorageConfiguration">
<bean
class="org.apache.ignite.configuration.DataStorageConfiguration">
<property
name="defaultDataRegionConfiguration">
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name"
value="Default_Region"/>
<property
name="initialSize" value="#{new
Long(systemEnvironment['IGNITE_DATA_REGION_MAX_SIZE'])}"/>
<property name="maxSize"
value="#{new Long(systemEnvironment['IGNITE_DATA_REGION_MAX_SIZE'])}"/>
<property
name="persistenceEnabled" value="#{new
Boolean(systemEnvironment['IGNITE_PERSISTENCE_ENABLED'])}"/>
</bean>
</property>
<property name="concurrencyLevel"
value="#{new
Integer(systemEnvironment['IGNITE_DATA_REGION_CONCURRENCY_LEVEL'])}"/>
</bean>
</property>
<property name="discoverySpi">
<bean
class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="joinTimeout" value="60000"/>
<property name="ipFinder">
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses"
value="#{systemEnvironment['IGNITE_IP_LIST'].split(',')}"/>
</bean>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<bean
class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name"
value="#{systemEnvironment['IGNITE_TEST_CACHE']}"/>
<property name="cacheMode"
value="PARTITIONED"/>
<property name="backups" value="#{new
Integer(systemEnvironment['IGNITE_TEST_CACHE_BACKUPS'])}"/>
<property name="writeSynchronizationMode"
value="FULL_SYNC"/>
</bean>
</property>
</bean>
</beans>
On Fri, Jun 22, 2018 at 7:11 AM, aealexsandrov <ae...@gmail.com>
wrote:
> Hi,
>
> Could you please provide your configuration files? How many nodes did you
> start in your container?
>
> BR,
> Andrei
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
*Andrew Fung*
Engineering | ❖ Medallia
afung@medallia.com
https://lwn.net/2000/0824/a/esr-sharing.php3
Re: running Apache Ignite in docker with cgroups
Posted by aealexsandrov <ae...@gmail.com>.
Hi,
Could you please provide your configuration files? How many nodes did you
start in your container?
BR,
Andrei
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: running Apache Ignite in docker with cgroups
Posted by Andrew Fung <af...@medallia.com>.
Thanks! Wasn't aware of these!
On Fri, Jun 22, 2018 at 7:14 AM, David Wimsey <dw...@bandwidth.com> wrote:
> Are you enabling the extra flags required for the JVM work detect memory
> and work properly?
>
> Specifically adding the following options to the JVM options when starting
> ignite
>
> -XX:+UseCGroupMemoryLimitForHeap. The -XX:+UnlockExperimentalVMOptions
>
> See: https://blogs.oracle.com/java-platform-group/java-se-
> support-for-docker-cpu-and-memory-limits
>
>
> > On Jun 21, 2018, at 5:32 PM, Andrew Fung <af...@medallia.com> wrote:
> >
> > I can see from the docs for IgniteConfiguration that some properties
> auto-size based on visible OS resources. In docker, "visible" ends up being
> the host values, which will exceed any CPU/memory limits applied via
> cgroups to the container.
> >
> > I see on IgniteConfiguration.DataStorageConfiguration.DataRegionConfiguration
> the properties initialSize and maxSize, which I've set substantially lower
> than the container memory allocation, which helped avoid being killed due
> to exceeding memory limits, but I'm still seeing the oom reaper kick in
> occasionally. Am I missing some other configuration value that controls
> off-heap memory use?
> >
> > I've set the DataRegionConfiguration initial/max size to 16GB, the JVM
> heap to 8GB, and the container is currently allocated 32GB, which seems
> more than enough for OS and other ancillary uses. In case relevant, I've
> enabled persistence, and set the relevant CacheConfiguration to partitioned
> mode with backups=2 and writeSynchronizationMode=full_sync. Running
> Ignite 2.5.0, Oracle JDK 8u172.
> >
> > Thanks!
> > Andrew.
>
>
--
*Andrew Fung*
Engineering | ❖ Medallia
afung@medallia.com
https://lwn.net/2000/0824/a/esr-sharing.php3
Re: running Apache Ignite in docker with cgroups
Posted by David Wimsey <dw...@bandwidth.com>.
Are you enabling the extra flags required for the JVM work detect memory and work properly?
Specifically adding the following options to the JVM options when starting ignite
-XX:+UseCGroupMemoryLimitForHeap. The -XX:+UnlockExperimentalVMOptions
See: https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-memory-limits
> On Jun 21, 2018, at 5:32 PM, Andrew Fung <af...@medallia.com> wrote:
>
> I can see from the docs for IgniteConfiguration that some properties auto-size based on visible OS resources. In docker, "visible" ends up being the host values, which will exceed any CPU/memory limits applied via cgroups to the container.
>
> I see on IgniteConfiguration.DataStorageConfiguration.DataRegionConfiguration the properties initialSize and maxSize, which I've set substantially lower than the container memory allocation, which helped avoid being killed due to exceeding memory limits, but I'm still seeing the oom reaper kick in occasionally. Am I missing some other configuration value that controls off-heap memory use?
>
> I've set the DataRegionConfiguration initial/max size to 16GB, the JVM heap to 8GB, and the container is currently allocated 32GB, which seems more than enough for OS and other ancillary uses. In case relevant, I've enabled persistence, and set the relevant CacheConfiguration to partitioned mode with backups=2 and writeSynchronizationMode=full_sync. Running Ignite 2.5.0, Oracle JDK 8u172.
>
> Thanks!
> Andrew.