You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Devin Bost <de...@gmail.com> on 2020/11/13 04:25:44 UTC
maxSize - per node or per cluster? - only 2k/sec in 11 node dedicated cluster
We're trying to figure out how to get more throughput from our Ignite
cluster.
We have 11 dedicated Ignite VM nodes, each with 32 GB of RAM. Yet, we're
only writing at 2k/sec max, even when we parallelize the writes to Ignite.
We're using native persistence, but it just seems way slower than expected.
We have our cache maxSize set to 16 GB. Since the cluster has a total of
352 GB of RAM, should our maxSize be set per node, or per cluster? If it's
per cluster, then I'm thinking we'd want to increase maxSize to at least
300 GB. Looking for feedback on this.
We've already turned off swapping on the boxes.
What else can we tune to increase throughput? It seems like we should be
getting 100x the throughput at least, so I'm hoping something is just
misconfigured.
Devin G. Bost
Re: maxSize - per node or per cluster? - only 2k/sec in 11 node
dedicated cluster
Posted by Pavel Vinokurov <vi...@gmail.com>.
Devin,
Could you share the code block making puts.
Do you use thick or thin clients?
Thanks,
Pavel
сб, 14 нояб. 2020 г. в 06:29, Devin Bost <de...@gmail.com>:
> Hi Pavel,
>
> Thanks for your reply.
>
> Here's one of our configs:
>
>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:util="http://www.springframework.org/schema/util"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
> http://www.springframework.org/schema/util
> http://www.springframework.org/schema/util/spring-util.xsd">
> <bean id="ignite.cfg" class=
> "org.apache.ignite.configuration.IgniteConfiguration">
>
> <property name="cacheConfiguration">
> <list>
> <bean class="org.apache.ignite.configuration.CacheConfiguration">
> <property name="name" value="bot"/>
> <property name="atomicityMode" value="TRANSACTIONAL"/>
> <property name="backups" value="2"/>
> <property name="cacheMode" value="PARTITIONED"/>
> <property name="statisticsEnabled" value="true"/>
> </bean>
> <bean class="org.apache.ignite.configuration.CacheConfiguration">
> <property name="name" value="session"/>
> <property name="atomicityMode" value="TRANSACTIONAL"/>
> <property name="backups" value="2"/>
> <property name="cacheMode" value="PARTITIONED"/>
> <property name="statisticsEnabled" value="true"/>
> </bean>
> <bean class="org.apache.ignite.configuration.CacheConfiguration">
> <property name="name" value="placements"/>
> <property name="atomicityMode" value="ATOMIC"/>
> <property name="backups" value="2"/>
> <property name="eagerTtl" value="true"/>
> <property name="cacheMode" value="PARTITIONED"/>
> <property name="statisticsEnabled" value="true"/>
> <property name="expiryPolicyFactory">
> <bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method=
> "factoryOf">
> <constructor-arg>
> <bean class="javax.cache.expiry.Duration">
> <constructor-arg value="HOURS"/>
> <constructor-arg value="48"/>
> </bean>
> </constructor-arg>
> </bean>
> </property>
> </bean>
> </list>
> </property>
>
> <property name="authenticationEnabled" value="true"/>
> <property name="dataStorageConfiguration">
> <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
> <property name="defaultDataRegionConfiguration">
> <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property name="persistenceEnabled" value="true"/>
> <property name="maxSize" value="#{16L * 1024 * 1024 * 1024}"/>
> </bean>
> </property>
> </bean>
> </property>
>
> <property name="discoverySpi">
> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="ipFinder">
> <bean class=
> "org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder"
> >
> <property name="addresses" value="[list of hosts]"/>
> </bean>
> </property>
> </bean>
> </property>
>
> <property name="connectorConfiguration">
> <bean class="org.apache.ignite.configuration.ConnectorConfiguration">
> <property name="host" value="0.0.0.0"/>
> <property name="port" value="[port]"/>
> <property name="idleQueryCursorTimeout" value="1000"/>
> </bean>
> </property>
>
> <property name="clientConnectorConfiguration">
> <bean class="org.apache.ignite.configuration.ClientConnectorConfiguration"
> >
> <property name="maxOpenCursorsPerConnection" value="1200"/>
> </bean>
> </property>
>
> <property name="includeEventTypes">
> <util:constant static-field=
> "org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_EXPIRED"/>
> </property>
>
> </bean>
> </beans>
>
> There are about 20 clients, and I'm not seeing any threading in the client
> code; however, what's strange is that we're seeing a very large number of
> connections (thousands), and we can't figure out what's causing the large
> number of connections. It does seem that performance drops even further
> when the connections increases over 2,000, but even when connections are in
> the hundreds, throughput is still below 3,000 / sec at best.
> One of our dev's said that IgniteDataStreamer didn't support our use case,
> I think because we're doing a lot of updates.
>
> Devin G. Bost
>
>
> On Thu, Nov 12, 2020 at 10:04 PM Pavel Vinokurov <
> vinokurov.pasha@gmail.com> wrote:
>
>> Hi Devin,
>>
>> MaxSize is set per node for the specified data region.
>> A few clarifying questions:
>> What cache configuration are you using? The performance could depend on
>> the type of a cache and the number of backups.
>> How many clients and threads are writing to the cluster? Because it is a
>> possible bottleneck on the client side.
>> Have you tried IgniteDataStreamer[2]? It's recommended approach to load
>> the massive amount of data.
>>
>> You could refer to persistence tuning documentation[1].
>>
>> [1] https://ignite.apache.org/docs/latest/persistence/persistence-tuning
>> [2] https://ignite.apache.org/docs/latest/data-streaming#data-streaming
>> Thanks,
>> Pavel
>>
>> пт, 13 нояб. 2020 г. в 07:26, Devin Bost <de...@gmail.com>:
>>
>>> We're trying to figure out how to get more throughput from our Ignite
>>> cluster.
>>> We have 11 dedicated Ignite VM nodes, each with 32 GB of RAM. Yet, we're
>>> only writing at 2k/sec max, even when we parallelize the writes to Ignite.
>>> We're using native persistence, but it just seems way slower than
>>> expected.
>>>
>>> We have our cache maxSize set to 16 GB. Since the cluster has a total of
>>> 352 GB of RAM, should our maxSize be set per node, or per cluster? If it's
>>> per cluster, then I'm thinking we'd want to increase maxSize to at least
>>> 300 GB. Looking for feedback on this.
>>>
>>> We've already turned off swapping on the boxes.
>>>
>>> What else can we tune to increase throughput? It seems like we should be
>>> getting 100x the throughput at least, so I'm hoping something is just
>>> misconfigured.
>>>
>>> Devin G. Bost
>>>
>>
>>
>> --
>>
>> Regards
>>
>> Pavel Vinokurov
>>
>
--
Regards
Pavel Vinokurov
Re: maxSize - per node or per cluster? - only 2k/sec in 11 node
dedicated cluster
Posted by Devin Bost <de...@gmail.com>.
Hi Pavel,
Thanks for your reply.
Here's one of our configs:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<bean id="ignite.cfg" class=
"org.apache.ignite.configuration.IgniteConfiguration">
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="bot"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
<property name="backups" value="2"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="statisticsEnabled" value="true"/>
</bean>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="session"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
<property name="backups" value="2"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="statisticsEnabled" value="true"/>
</bean>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="placements"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="2"/>
<property name="eagerTtl" value="true"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="statisticsEnabled" value="true"/>
<property name="expiryPolicyFactory">
<bean class="javax.cache.expiry.CreatedExpiryPolicy" factory-method=
"factoryOf">
<constructor-arg>
<bean class="javax.cache.expiry.Duration">
<constructor-arg value="HOURS"/>
<constructor-arg value="48"/>
</bean>
</constructor-arg>
</bean>
</property>
</bean>
</list>
</property>
<property name="authenticationEnabled" value="true"/>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true"/>
<property name="maxSize" value="#{16L * 1024 * 1024 * 1024}"/>
</bean>
</property>
</bean>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class=
"org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder"
>
<property name="addresses" value="[list of hosts]"/>
</bean>
</property>
</bean>
</property>
<property name="connectorConfiguration">
<bean class="org.apache.ignite.configuration.ConnectorConfiguration">
<property name="host" value="0.0.0.0"/>
<property name="port" value="[port]"/>
<property name="idleQueryCursorTimeout" value="1000"/>
</bean>
</property>
<property name="clientConnectorConfiguration">
<bean class="org.apache.ignite.configuration.ClientConnectorConfiguration">
<property name="maxOpenCursorsPerConnection" value="1200"/>
</bean>
</property>
<property name="includeEventTypes">
<util:constant static-field=
"org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_EXPIRED"/>
</property>
</bean>
</beans>
There are about 20 clients, and I'm not seeing any threading in the client
code; however, what's strange is that we're seeing a very large number of
connections (thousands), and we can't figure out what's causing the large
number of connections. It does seem that performance drops even further
when the connections increases over 2,000, but even when connections are in
the hundreds, throughput is still below 3,000 / sec at best.
One of our dev's said that IgniteDataStreamer didn't support our use case,
I think because we're doing a lot of updates.
Devin G. Bost
On Thu, Nov 12, 2020 at 10:04 PM Pavel Vinokurov <vi...@gmail.com>
wrote:
> Hi Devin,
>
> MaxSize is set per node for the specified data region.
> A few clarifying questions:
> What cache configuration are you using? The performance could depend on
> the type of a cache and the number of backups.
> How many clients and threads are writing to the cluster? Because it is a
> possible bottleneck on the client side.
> Have you tried IgniteDataStreamer[2]? It's recommended approach to load
> the massive amount of data.
>
> You could refer to persistence tuning documentation[1].
>
> [1] https://ignite.apache.org/docs/latest/persistence/persistence-tuning
> [2] https://ignite.apache.org/docs/latest/data-streaming#data-streaming
> Thanks,
> Pavel
>
> пт, 13 нояб. 2020 г. в 07:26, Devin Bost <de...@gmail.com>:
>
>> We're trying to figure out how to get more throughput from our Ignite
>> cluster.
>> We have 11 dedicated Ignite VM nodes, each with 32 GB of RAM. Yet, we're
>> only writing at 2k/sec max, even when we parallelize the writes to Ignite.
>> We're using native persistence, but it just seems way slower than
>> expected.
>>
>> We have our cache maxSize set to 16 GB. Since the cluster has a total of
>> 352 GB of RAM, should our maxSize be set per node, or per cluster? If it's
>> per cluster, then I'm thinking we'd want to increase maxSize to at least
>> 300 GB. Looking for feedback on this.
>>
>> We've already turned off swapping on the boxes.
>>
>> What else can we tune to increase throughput? It seems like we should be
>> getting 100x the throughput at least, so I'm hoping something is just
>> misconfigured.
>>
>> Devin G. Bost
>>
>
>
> --
>
> Regards
>
> Pavel Vinokurov
>
Re: maxSize - per node or per cluster? - only 2k/sec in 11 node
dedicated cluster
Posted by Pavel Vinokurov <vi...@gmail.com>.
Hi Devin,
MaxSize is set per node for the specified data region.
A few clarifying questions:
What cache configuration are you using? The performance could depend on
the type of a cache and the number of backups.
How many clients and threads are writing to the cluster? Because it is a
possible bottleneck on the client side.
Have you tried IgniteDataStreamer[2]? It's recommended approach to load
the massive amount of data.
You could refer to persistence tuning documentation[1].
[1] https://ignite.apache.org/docs/latest/persistence/persistence-tuning
[2] https://ignite.apache.org/docs/latest/data-streaming#data-streaming
Thanks,
Pavel
пт, 13 нояб. 2020 г. в 07:26, Devin Bost <de...@gmail.com>:
> We're trying to figure out how to get more throughput from our Ignite
> cluster.
> We have 11 dedicated Ignite VM nodes, each with 32 GB of RAM. Yet, we're
> only writing at 2k/sec max, even when we parallelize the writes to Ignite.
> We're using native persistence, but it just seems way slower than expected.
>
> We have our cache maxSize set to 16 GB. Since the cluster has a total of
> 352 GB of RAM, should our maxSize be set per node, or per cluster? If it's
> per cluster, then I'm thinking we'd want to increase maxSize to at least
> 300 GB. Looking for feedback on this.
>
> We've already turned off swapping on the boxes.
>
> What else can we tune to increase throughput? It seems like we should be
> getting 100x the throughput at least, so I'm hoping something is just
> misconfigured.
>
> Devin G. Bost
>
--
Regards
Pavel Vinokurov