You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "javastuff.sam@gmail.com" <ja...@gmail.com> on 2017/04/18 19:13:45 UTC
stdout - Message queue limit is set to 0, potential OOMEs
Recently moved to Ignite 1.9 and noticed below line on stdout -
/Message queue limit is set to 0 which may lead to potential OOMEs when
running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message
queues growth on sender and receiver sides./
Not sure if this existed in older version.
Can anybody explain what does it mean and how to avoid potential OOME?
Thanks,
-Sam
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/stdout-Message-queue-limit-is-set-to-0-potential-OOMEs-tp12048.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: stdout - Message queue limit is set to 0, potential OOMEs
Posted by "javastuff.sam@gmail.com" <ja...@gmail.com>.
In that case I will change my configuration to have <property
name="messageQueueLimit" value="1024"/> and change it if needed.
Out-of-box unlimited seems too aggressive with potential OOME risk.
Thanks,
-Sam
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/stdout-Message-queue-limit-is-set-to-0-potential-OOMEs-tp12048p12209.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: stdout - Message queue limit is set to 0, potential OOMEs
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Sam,
If you see messageQueueSize is often close to limit and low CPU pressure on
nodes,
then limit value increasing can be helpful.
On Fri, Apr 21, 2017 at 11:36 PM, javastuff.sam@gmail.com <
javastuff.sam@gmail.com> wrote:
> Added below to TcpCommunicationSpi and message went away, able to see the
> new
> value on JMX as well.
> /<property name="messageQueueLimit" value="1024"/>/
>
> However, how do I know what is the good or suitable value one need to set?
> I
> have more than 50 different PARTITIONED OFF-HEAP caches defined, most
> likely
> to have 2 to 4 nodes with 20GB each or more.
>
> in my opinion, potential OOME is big risk with default configuration.
> Everybody needs performance, but no one really need OOME.
>
> Thanks,
> -Sam
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/stdout-Message-queue-limit-is-set-to-
> 0-potential-OOMEs-tp12048p12165.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov
Re: stdout - Message queue limit is set to 0, potential OOMEs
Posted by "javastuff.sam@gmail.com" <ja...@gmail.com>.
Added below to TcpCommunicationSpi and message went away, able to see the new
value on JMX as well.
/<property name="messageQueueLimit" value="1024"/>/
However, how do I know what is the good or suitable value one need to set? I
have more than 50 different PARTITIONED OFF-HEAP caches defined, most likely
to have 2 to 4 nodes with 20GB each or more.
in my opinion, potential OOME is big risk with default configuration.
Everybody needs performance, but no one really need OOME.
Thanks,
-Sam
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/stdout-Message-queue-limit-is-set-to-0-potential-OOMEs-tp12048p12165.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: stdout - Message queue limit is set to 0, potential OOMEs
Posted by Andrey Mashenkov <an...@gmail.com>.
Actual 'outbound queue size' are printed in node mertics messages of
accessible via mbean ClusterLocalNodeMetricsMXBean.
On Wed, Apr 19, 2017 at 11:13 AM, javastuff.sam@gmail.com <
javastuff.sam@gmail.com> wrote:
> What do you mean by messages? I am not using Ignite messaging. Are these
> messages of rebalancing during topology change?
> How do I configure it to avoid potential OOME?
>
> Thanks,
> -Sam
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/stdout-Message-queue-limit-is-set-to-
> 0-potential-OOMEs-tp12048p12066.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov
Re: stdout - Message queue limit is set to 0, potential OOMEs
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Sam,
Before 1.9, default queue limit value was 1024.
This was changed in IGNITE-3220 [1]
[1] https://issues.apache.org/jira/browse/IGNITE-3220
On Wed, Apr 19, 2017 at 8:54 PM, javastuff.sam@gmail.com <
javastuff.sam@gmail.com> wrote:
> Thank you. Last question -
> Why it showing up with 1.9 and not earlier?
>
> Thanks,
> -Sam
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/stdout-Message-queue-limit-is-set-to-
> 0-potential-OOMEs-tp12048p12091.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov
Re: stdout - Message queue limit is set to 0, potential OOMEs
Posted by "javastuff.sam@gmail.com" <ja...@gmail.com>.
Thank you. Last question -
Why it showing up with 1.9 and not earlier?
Thanks,
-Sam
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/stdout-Message-queue-limit-is-set-to-0-potential-OOMEs-tp12048p12091.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: stdout - Message queue limit is set to 0, potential OOMEs
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Sam,
Ignite use messages for inter-node communication.
You have to configure TcpCommunicationSPI in IgniteConfiguretion. Method
TcpCommunicationSPI.setMessageQueueLimit() is what you need.
On Wed, Apr 19, 2017 at 11:13 AM, javastuff.sam@gmail.com <
javastuff.sam@gmail.com> wrote:
> What do you mean by messages? I am not using Ignite messaging. Are these
> messages of rebalancing during topology change?
> How do I configure it to avoid potential OOME?
>
> Thanks,
> -Sam
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/stdout-Message-queue-limit-is-set-to-
> 0-potential-OOMEs-tp12048p12066.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov
Re: stdout - Message queue limit is set to 0, potential OOMEs
Posted by "javastuff.sam@gmail.com" <ja...@gmail.com>.
What do you mean by messages? I am not using Ignite messaging. Are these
messages of rebalancing during topology change?
How do I configure it to avoid potential OOME?
Thanks,
-Sam
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/stdout-Message-queue-limit-is-set-to-0-potential-OOMEs-tp12048p12066.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: stdout - Message queue limit is set to 0, potential OOMEs
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Sam,
Slow node can cause OOME, as messages for slow node can make queue grows
infinitly.
Also, you can receive messages with higher rate than node able to process,
that also can results in OOME.
On Tue, Apr 18, 2017 at 10:13 PM, javastuff.sam@gmail.com <
javastuff.sam@gmail.com> wrote:
> Recently moved to Ignite 1.9 and noticed below line on stdout -
>
> /Message queue limit is set to 0 which may lead to potential OOMEs when
> running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message
> queues growth on sender and receiver sides./
>
> Not sure if this existed in older version.
>
> Can anybody explain what does it mean and how to avoid potential OOME?
>
> Thanks,
> -Sam
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/stdout-Message-queue-limit-is-set-to-
> 0-potential-OOMEs-tp12048.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov