You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Ramzinator <ra...@gmail.com> on 2017/05/15 13:27:15 UTC
Performance issue with Replicated Cache among 7 nodes
Hi all,
I have an ignite cluster of 7 nodes and a cache that is replicated among all
of them configured as follows:
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="cacheName"/>
<property name="cacheMode" value="REPLICATED"/>
<property name="writeSynchronizationMode" value="FULL_SYNC"/>
<property name="copyOnRead" value="false"/>
<property name="backups" value="0"/>
</bean>
When inserting the first object (of size ~15mb) into the cache, the
insertion takes around *30 mins*.
No exception are thrown during this time, and the nodes were practically
idle. However we do witness logs such as:
2017-05-15 12:28:55.507 WARN [gridname,,,] 1 --- [gridname%]
.i.p.c.GridCachePartitionExchangeManager : Found long running cache
operations, dump IO statistics.
2017-05-15 12:28:55.509 WARN [gridname,,,] 1 --- [gridname%]
o.a.i.s.c.tcp.TcpCommunicationSpi : Communication SPI recovery
descriptors:
[key=ConnectionKey [nodeId=8d24da8d-38a9-455b-a295-e735bbb503e0, idx=0,
connCnt=-1], msgsSent=2, msgsAckedByRmt=2, msgsRcvd=2, lastAcked=2,
reserveCnt=5, descIdHash=420108385]
[key=ConnectionKey [nodeId=89c5e965-6351-4878-b78c-813ae159cc29, idx=0,
connCnt=-1], msgsSent=11, msgsAckedByRmt=11, msgsRcvd=9, lastAcked=9,
reserveCnt=4, descIdHash=510995069]
[key=ConnectionKey [nodeId=7fe63f13-d34d-4a75-8b99-233ed21f192c, idx=0,
connCnt=-1], msgsSent=7, msgsAckedByRmt=6, msgsRcvd=6, lastAcked=5,
reserveCnt=8, descIdHash=334727389]
[key=ConnectionKey [nodeId=ac0a6e04-8ea1-427e-989f-bec85f476bb1, idx=0,
connCnt=-1], msgsSent=32, msgsAckedByRmt=32, msgsRcvd=30, lastAcked=29,
reserveCnt=4, descIdHash=1989271882]
[key=ConnectionKey [nodeId=95ac3c7b-0445-492d-9bd5-0f22544b15f5, idx=0,
connCnt=-1], msgsSent=3, msgsAckedByRmt=1, msgsRcvd=3, lastAcked=1,
reserveCnt=8, descIdHash=330160523]
[key=ConnectionKey [nodeId=47f1b382-5acb-4882-b3d8-d5878857778f, idx=0,
connCnt=-1], msgsSent=9, msgsAckedByRmt=9, msgsRcvd=6, lastAcked=4,
reserveCnt=3, descIdHash=1024681641]
[key=ConnectionKey [nodeId=98855451-c7ca-4b40-bf50-c8ae17a88a61, idx=0,
connCnt=-1], msgsSent=3, msgsAckedByRmt=2, msgsRcvd=3, lastAcked=2,
reserveCnt=8, descIdHash=167941556]
[key=ConnectionKey [nodeId=5a079141-43ad-406c-820b-cebaf32c9729, idx=0,
connCnt=0], msgsSent=17, msgsAckedByRmt=17, msgsRcvd=14, lastAcked=13,
reserveCnt=1, descIdHash=87576905]
[key=ConnectionKey [nodeId=573f277b-8a2c-47cc-ab0f-33d389a6294f, idx=0,
connCnt=-1], msgsSent=12, msgsAckedByRmt=12, msgsRcvd=8, lastAcked=7,
reserveCnt=4, descIdHash=2094693141]
Also the insertion completes but after a very long time.
My question is why is the insertion taking so long?
Is ignite stuck in a transaction and waiting to timeout?
I'd appreciate any help/advice on how to proceed to speed up the insertion.
Note that the issue became noticeable when we scaled up the cluster from 3
ignite nodes to 7.
Thanks,
Ramzinator
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-issue-with-Replicated-Cache-among-7-nodes-tp12848.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Performance issue with Replicated Cache among 7 nodes
Posted by Ramzinator <ra...@gmail.com>.
Hi Andrey,
It turns out that we had increases our failureDetectionTimeout wayyy too
much. After setting it to something more sensible, the issue stopped.
Thanks for the info,
Rami
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-issue-with-Replicated-Cache-among-7-nodes-tp12848p12880.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Performance issue with Replicated Cache among 7 nodes
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Ramzinator,
1. There is no need to se number of backups when using replicated cache.
All data nodes will have full cache data copy.
2. For FULL_SYNC mode, Ignite will send updates to all nodes and waits for
they being finished.
3. Is it possible there is slow connection? How much time takes to transfer
15m between nodes?
4. Does second put will takes same time or faster?
Would you please share a repro?
On Mon, May 15, 2017 at 4:27 PM, Ramzinator <ra...@gmail.com> wrote:
> Hi all,
>
> I have an ignite cluster of 7 nodes and a cache that is replicated among
> all
> of them configured as follows:
> <bean class="org.apache.ignite.configuration.CacheConfiguration">
> <property name="name" value="cacheName"/>
> <property name="cacheMode" value="REPLICATED"/>
> <property name="writeSynchronizationMode" value="FULL_SYNC"/>
> <property name="copyOnRead" value="false"/>
> <property name="backups" value="0"/>
> </bean>
>
> When inserting the first object (of size ~15mb) into the cache, the
> insertion takes around *30 mins*.
>
> No exception are thrown during this time, and the nodes were practically
> idle. However we do witness logs such as:
>
> 2017-05-15 12:28:55.507 WARN [gridname,,,] 1 --- [gridname%]
> .i.p.c.GridCachePartitionExchangeManager : Found long running cache
> operations, dump IO statistics.
> 2017-05-15 12:28:55.509 WARN [gridname,,,] 1 --- [gridname%]
> o.a.i.s.c.tcp.TcpCommunicationSpi : Communication SPI recovery
> descriptors:
> [key=ConnectionKey [nodeId=8d24da8d-38a9-455b-a295-e735bbb503e0,
> idx=0,
> connCnt=-1], msgsSent=2, msgsAckedByRmt=2, msgsRcvd=2, lastAcked=2,
> reserveCnt=5, descIdHash=420108385]
> [key=ConnectionKey [nodeId=89c5e965-6351-4878-b78c-813ae159cc29,
> idx=0,
> connCnt=-1], msgsSent=11, msgsAckedByRmt=11, msgsRcvd=9, lastAcked=9,
> reserveCnt=4, descIdHash=510995069]
> [key=ConnectionKey [nodeId=7fe63f13-d34d-4a75-8b99-233ed21f192c,
> idx=0,
> connCnt=-1], msgsSent=7, msgsAckedByRmt=6, msgsRcvd=6, lastAcked=5,
> reserveCnt=8, descIdHash=334727389]
> [key=ConnectionKey [nodeId=ac0a6e04-8ea1-427e-989f-bec85f476bb1,
> idx=0,
> connCnt=-1], msgsSent=32, msgsAckedByRmt=32, msgsRcvd=30, lastAcked=29,
> reserveCnt=4, descIdHash=1989271882]
> [key=ConnectionKey [nodeId=95ac3c7b-0445-492d-9bd5-0f22544b15f5,
> idx=0,
> connCnt=-1], msgsSent=3, msgsAckedByRmt=1, msgsRcvd=3, lastAcked=1,
> reserveCnt=8, descIdHash=330160523]
> [key=ConnectionKey [nodeId=47f1b382-5acb-4882-b3d8-d5878857778f,
> idx=0,
> connCnt=-1], msgsSent=9, msgsAckedByRmt=9, msgsRcvd=6, lastAcked=4,
> reserveCnt=3, descIdHash=1024681641]
> [key=ConnectionKey [nodeId=98855451-c7ca-4b40-bf50-c8ae17a88a61,
> idx=0,
> connCnt=-1], msgsSent=3, msgsAckedByRmt=2, msgsRcvd=3, lastAcked=2,
> reserveCnt=8, descIdHash=167941556]
> [key=ConnectionKey [nodeId=5a079141-43ad-406c-820b-cebaf32c9729,
> idx=0,
> connCnt=0], msgsSent=17, msgsAckedByRmt=17, msgsRcvd=14, lastAcked=13,
> reserveCnt=1, descIdHash=87576905]
> [key=ConnectionKey [nodeId=573f277b-8a2c-47cc-ab0f-33d389a6294f,
> idx=0,
> connCnt=-1], msgsSent=12, msgsAckedByRmt=12, msgsRcvd=8, lastAcked=7,
> reserveCnt=4, descIdHash=2094693141]
>
> Also the insertion completes but after a very long time.
>
> My question is why is the insertion taking so long?
> Is ignite stuck in a transaction and waiting to timeout?
> I'd appreciate any help/advice on how to proceed to speed up the insertion.
> Note that the issue became noticeable when we scaled up the cluster from 3
> ignite nodes to 7.
>
> Thanks,
> Ramzinator
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Performance-issue-with-Replicated-Cache-among-7-
> nodes-tp12848.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov