You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Mikael <mi...@gmail.com> on 2022/01/11 17:59:13 UTC
Re: long JVM pauses
Hi!
There are no generic settings, each case is unique, it will require some
tuning to get it right, how much java heap do you use ? and what garbage
collector are you using ? how much data do you have ? persistence
enabled ? how do you use the caches, is data deleted often or do you
keep it around for longer time ?
Mikael
On 2022-01-11 18:23, satyajit.mandal@barclays.com wrote:
>
> Hi Team,
>
> We do see long JVM pauses and after that nodes in our cluster
> stops. This happens in every two weeks usually. What are the
> possible solutions to avoid long JVM pause. Can someone advise
> generic settings which is recommended in such cases?
>
> [23:20:37,824][WARNING][jvm-pause-detector-worker][IgniteKernal]
> Possible too long JVM pause: 2262 milliseconds.
>
> Regards
>
> Satyajit
>
>
> Restricted - Internal
>
> This message is for information purposes only. It is not a
> recommendation, advice, offer or solicitation to buy or sell a product
> or service, nor an official confirmation of any transaction. It is
> directed at persons who are professionals and is intended for the
> recipient(s) only. It is not directed at retail customers. This
> message is subject to the terms at:
> https://www.cib.barclays/disclosures/web-and-email-disclaimer.html.
>
> For important disclosures, please see:
> https://www.cib.barclays/disclosures/sales-and-trading-disclaimer.html
> regarding marketing commentary from Barclays Sales and/or Trading
> desks, who are active market participants;
> https://www.cib.barclays/disclosures/barclays-global-markets-disclosures.html
> regarding our standard terms for Barclays Corporate and Investment
> Bank where we trade with you in principal-to-principal wholesale
> markets transactions; and in respect to Barclays Research, including
> disclosures relating to specific issuers, see:
> http://publicresearch.barclays.com.
> __________________________________________________________________________________
>
> If you are incorporated or operating in Australia, read these
> important disclosures:
> https://www.cib.barclays/disclosures/important-disclosures-asia-pacific.html.
> __________________________________________________________________________________
> For more details about how we use personal information, see our
> privacy notice:
> https://www.cib.barclays/disclosures/personal-information-use.html.
> __________________________________________________________________________________
>
Re: long JVM pauses
Posted by Mikael <mi...@gmail.com>.
Hi!
Ok, it does not sound like very extreme, do you really need 10GB heap ?
as you said it takes two weeks before the problems start, it does sound
like you have something growing there pretty slow and it does have to do
some intense GC pass finally to clean all that up, using a smaller heap
might help there if you don't need the big heap (less garbage to cleanup
when it get full, less GC time, so it will GC more often but will not
lock up the JVM for such long time).
~2 seconds GC pause is not a huge disaster so I think you should be able
to solve it with some tweaking and/or change collector.
The G1 or ZGC collectors work pretty well these days but I don't know
what JVM you use.
Linux/Windows ? setting a large heap (in relation to ram) on a Linux
machine with default swappiness can cause problems, it often start
paging if you just fill up half the memory or so.
I am no export on GC so I hope someone with better knowledge can give
some ideas, but I think the best is to log the GC for a few days and see
what is going on with the heap, each application is unique so it is very
difficult to give any generic " do this" and all will be good tips, also
make sure it actually is GC pauses that is causing the problem so that
it isn't something else.
regards
Mikael
On 2022-01-12 05:52, satyajit.mandal@barclays.com wrote:
>
> Hi Mikael,
>
> We are using below settings and we have default off heap
> memory enabled. Heap size is 10 GB per node and we are
> running 4 nodes as part of cluster. Data size is ~25MB but
> it continuously updates and inserts records in every 5 mins
> and 15 mins. It’s a low latency application. We haven’t enabled
> persistence.
>
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=55555
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false -Xms10g -Xmx10g
> -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC
> -XX:+DisableExplicitGC -Djava.net.preferIPv4Stack=true
>
> Regards
>
> Satyajit
>
> *From:*Mikael <mi...@gmail.com>
> *Sent:* Tuesday, January 11, 2022 11:29 PM
> *To:* user@ignite.apache.org
> *Subject:* Re: long JVM pauses
>
> CAUTION: This email originated from outside our organisation -
> mikael.aronsson@gmail.com Do not click on links, open attachments, or
> respond unless you recognize the sender and can validate the content
> is safe.
>
> Hi!
>
> There are no generic settings, each case is unique, it will require
> some tuning to get it right, how much java heap do you use ? and what
> garbage collector are you using ? how much data do you have ?
> persistence enabled ? how do you use the caches, is data deleted often
> or do you keep it around for longer time ?
>
> Mikael
>
> On 2022-01-11 18:23, satyajit.mandal@barclays.com wrote:
>
> Hi Team,
>
> We do see long JVM pauses and after that nodes in our
> cluster stops. This happens in every two weeks usually.
> What are the possible solutions to avoid long JVM pause.
> Can someone advise generic settings which is recommended in
> such cases?
>
> [23:20:37,824][WARNING][jvm-pause-detector-worker][IgniteKernal]
> Possible too long JVM pause: 2262 milliseconds.
>
> Regards
>
> Satyajit
>
>
> Restricted - Internal
>
> This message is for information purposes only. It is not a
> recommendation, advice, offer or solicitation to buy or sell a
> product or service, nor an official confirmation of any
> transaction. It is directed at persons who are professionals and
> is intended for the recipient(s) only. It is not directed at
> retail customers. This message is subject to the terms at:
> https://www.cib.barclays/disclosures/web-and-email-disclaimer.html
> <https://clicktime.symantec.com/36Drmrd2f6GnrX5HMTwSVT36H4?u=https%3A%2F%2Fwww.cib.barclays%2Fdisclosures%2Fweb-and-email-disclaimer.html>.
>
>
> For important disclosures, please see:
> https://www.cib.barclays/disclosures/sales-and-trading-disclaimer.html
> <https://clicktime.symantec.com/3B4s9V77DYoyX9dN8ua84my6H4?u=https%3A%2F%2Fwww.cib.barclays%2Fdisclosures%2Fsales-and-trading-disclaimer.html>
> regarding marketing commentary from Barclays Sales and/or Trading
> desks, who are active market participants;
> https://www.cib.barclays/disclosures/barclays-global-markets-disclosures.html
> <https://clicktime.symantec.com/3JegRP7ErAjYtcC7TGJFKtS6H4?u=https%3A%2F%2Fwww.cib.barclays%2Fdisclosures%2Fbarclays-global-markets-disclosures.html>
> regarding our standard terms for Barclays Corporate and Investment
> Bank where we trade with you in principal-to-principal wholesale
> markets transactions; and in respect to Barclays Research,
> including disclosures relating to specific issuers, see:
> http://publicresearch.barclays.com
> <https://clicktime.symantec.com/3HZtwy1wDet5tLGDuUowvHv6H4?u=http%3A%2F%2Fpublicresearch.barclays.com>.
> __________________________________________________________________________________
>
> If you are incorporated or operating in Australia, read these
> important disclosures:
> https://www.cib.barclays/disclosures/important-disclosures-asia-pacific.html
> <https://clicktime.symantec.com/35pi9P5XPXZcYsC5PhKqM1K6H4?u=https%3A%2F%2Fwww.cib.barclays%2Fdisclosures%2Fimportant-disclosures-asia-pacific.html>.
> __________________________________________________________________________________
> For more details about how we use personal information, see our
> privacy notice:
> https://www.cib.barclays/disclosures/personal-information-use.html
> <https://clicktime.symantec.com/3H8MJpzhKqWY15E4QwLkCNv6H4?u=https%3A%2F%2Fwww.cib.barclays%2Fdisclosures%2Fpersonal-information-use.html>.
>
> __________________________________________________________________________________
>
> This message is for information purposes only. It is not a
> recommendation, advice, offer or solicitation to buy or sell a product
> or service, nor an official confirmation of any transaction. It is
> directed at persons who are professionals and is intended for the
> recipient(s) only. It is not directed at retail customers. This
> message is subject to the terms at:
> https://www.cib.barclays/disclosures/web-and-email-disclaimer.html.
>
> For important disclosures, please see:
> https://www.cib.barclays/disclosures/sales-and-trading-disclaimer.html
> regarding marketing commentary from Barclays Sales and/or Trading
> desks, who are active market participants;
> https://www.cib.barclays/disclosures/barclays-global-markets-disclosures.html
> regarding our standard terms for Barclays Corporate and Investment
> Bank where we trade with you in principal-to-principal wholesale
> markets transactions; and in respect to Barclays Research, including
> disclosures relating to specific issuers, see:
> http://publicresearch.barclays.com.
> __________________________________________________________________________________
>
> If you are incorporated or operating in Australia, read these
> important disclosures:
> https://www.cib.barclays/disclosures/important-disclosures-asia-pacific.html.
> __________________________________________________________________________________
> For more details about how we use personal information, see our
> privacy notice:
> https://www.cib.barclays/disclosures/personal-information-use.html.
> __________________________________________________________________________________
>
RE: long JVM pauses
Posted by sa...@barclays.com.
Hi Mikael,
We are using below settings and we have default off heap memory enabled. Heap size is 10 GB per node and we are running 4 nodes as part of cluster. Data size is ~25MB but it continuously updates and inserts records in every 5 mins and 15 mins. It’s a low latency application. We haven’t enabled persistence.
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=55555 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Xms10g -Xmx10g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -Djava.net.preferIPv4Stack=true
Regards
Satyajit
From: Mikael <mi...@gmail.com>
Sent: Tuesday, January 11, 2022 11:29 PM
To: user@ignite.apache.org
Subject: Re: long JVM pauses
CAUTION: This email originated from outside our organisation - mikael.aronsson@gmail.com<ma...@gmail.com> Do not click on links, open attachments, or respond unless you recognize the sender and can validate the content is safe.
Hi!
There are no generic settings, each case is unique, it will require some tuning to get it right, how much java heap do you use ? and what garbage collector are you using ? how much data do you have ? persistence enabled ? how do you use the caches, is data deleted often or do you keep it around for longer time ?
Mikael
On 2022-01-11 18:23, satyajit.mandal@barclays.com<ma...@barclays.com> wrote:
Hi Team,
We do see long JVM pauses and after that nodes in our cluster stops. This happens in every two weeks usually. What are the possible solutions to avoid long JVM pause. Can someone advise generic settings which is recommended in such cases?
[23:20:37,824][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible too long JVM pause: 2262 milliseconds.
Regards
Satyajit
Restricted - Internal
This message is for information purposes only. It is not a recommendation, advice, offer or solicitation to buy or sell a product or service, nor an official confirmation of any transaction. It is directed at persons who are professionals and is intended for the recipient(s) only. It is not directed at retail customers. This message is subject to the terms at: https://www.cib.barclays/disclosures/web-and-email-disclaimer.html<https://clicktime.symantec.com/36Drmrd2f6GnrX5HMTwSVT36H4?u=https%3A%2F%2Fwww.cib.barclays%2Fdisclosures%2Fweb-and-email-disclaimer.html>.
For important disclosures, please see: https://www.cib.barclays/disclosures/sales-and-trading-disclaimer.html<https://clicktime.symantec.com/3B4s9V77DYoyX9dN8ua84my6H4?u=https%3A%2F%2Fwww.cib.barclays%2Fdisclosures%2Fsales-and-trading-disclaimer.html> regarding marketing commentary from Barclays Sales and/or Trading desks, who are active market participants; https://www.cib.barclays/disclosures/barclays-global-markets-disclosures.html<https://clicktime.symantec.com/3JegRP7ErAjYtcC7TGJFKtS6H4?u=https%3A%2F%2Fwww.cib.barclays%2Fdisclosures%2Fbarclays-global-markets-disclosures.html> regarding our standard terms for Barclays Corporate and Investment Bank where we trade with you in principal-to-principal wholesale markets transactions; and in respect to Barclays Research, including disclosures relating to specific issuers, see: http://publicresearch.barclays.com<https://clicktime.symantec.com/3HZtwy1wDet5tLGDuUowvHv6H4?u=http%3A%2F%2Fpublicresearch.barclays.com>.
__________________________________________________________________________________
If you are incorporated or operating in Australia, read these important disclosures: https://www.cib.barclays/disclosures/important-disclosures-asia-pacific.html<https://clicktime.symantec.com/35pi9P5XPXZcYsC5PhKqM1K6H4?u=https%3A%2F%2Fwww.cib.barclays%2Fdisclosures%2Fimportant-disclosures-asia-pacific.html>.
__________________________________________________________________________________
For more details about how we use personal information, see our privacy notice: https://www.cib.barclays/disclosures/personal-information-use.html<https://clicktime.symantec.com/3H8MJpzhKqWY15E4QwLkCNv6H4?u=https%3A%2F%2Fwww.cib.barclays%2Fdisclosures%2Fpersonal-information-use.html>.
__________________________________________________________________________________
This message is for information purposes only. It is not a recommendation, advice, offer or solicitation to buy or sell a product or service, nor an official confirmation of any transaction. It is directed at persons who are professionals and is intended for the recipient(s) only. It is not directed at retail customers. This message is subject to the terms at: https://www.cib.barclays/disclosures/web-and-email-disclaimer.html.
For important disclosures, please see: https://www.cib.barclays/disclosures/sales-and-trading-disclaimer.html regarding marketing commentary from Barclays Sales and/or Trading desks, who are active market participants; https://www.cib.barclays/disclosures/barclays-global-markets-disclosures.html regarding our standard terms for Barclays Corporate and Investment Bank where we trade with you in principal-to-principal wholesale markets transactions; and in respect to Barclays Research, including disclosures relating to specific issuers, see: http://publicresearch.barclays.com.
__________________________________________________________________________________
If you are incorporated or operating in Australia, read these important disclosures: https://www.cib.barclays/disclosures/important-disclosures-asia-pacific.html.
__________________________________________________________________________________
For more details about how we use personal information, see our privacy notice: https://www.cib.barclays/disclosures/personal-information-use.html.
__________________________________________________________________________________