You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by "Kalwakuntla, Srinivas" <sr...@accenture.com.INVALID> on 2024/03/07 14:54:35 UTC
Performance Issue at Ignite Level
Dear Ignite Team,
Hope this email finds your well !
Reaching out to you, as currently we are facing performance issue with Apache Ignite. Our application is running on Azure Kubernetes (v1.27.1), which has 24 nodes of size "Standard_F64s_v2" (handling roughly 13million of records).
Frequently we are getting "[WARNING][jvm-pause-detector-worker][IgniteKernal] Possible too long JVM pause: xxxx milliseconds", while performing any operations such as reading, encoding etc.
Here below is the system information:
System Information (Apache Ignite Clusters)
Heap size: 32GB out of 64GB of memory
CPUs: Standard_F64s_v2
The number of server instances: 24
IOWait: within normal range
IOPS: 10k-600K per hr
JDK: Oracle JDK8, 1.8.0_281
APACHE IGNITE - 2.14.0
GC configuration in ignite.sh:
XX:-UseContainerSupport
-XX:+AlwaysPreTouch
-XX:+UseG1GC
-XX:+ScavengeBeforeFullGC
-XX:+DisableExplicitGC"
Please assist us in identifying the underlying source of this problem.
Thank you,
Srinivas
________________________________
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security, AI-powered support capabilities, and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
______________________________________________________________________________________
www.accenture.com
Re: Performance Issue at Ignite Level
Posted by Stephen Darlington <st...@gridgain.com>.
+ user mailing list
- developer mailing list
This question would be better addressed to the user mailing list.
There are a number of causes of "long JVM pauses,” so it’s difficult to be prescriptive. I’d start by logging more information about Java’s garbage collection.
More information in the documentation here: https://ignite.apache.org/docs/latest/perf-and-troubleshooting/memory-tuning#generic-gc-settings
Given the data provided, I’d say your heap space is likely excessive. Having more heap space isn’t necessarily better.
Regards,
Stephen
> On 7 Mar 2024, at 14:54, Kalwakuntla, Srinivas <sr...@accenture.com.INVALID> wrote:
>
> Dear Ignite Team,
>
> Hope this email finds your well !
>
> Reaching out to you, as currently we are facing performance issue with Apache Ignite. Our application is running on Azure Kubernetes (v1.27.1), which has 24 nodes of size "Standard_F64s_v2" (handling roughly 13million of records).
> Frequently we are getting "[WARNING][jvm-pause-detector-worker][IgniteKernal] Possible too long JVM pause: xxxx milliseconds", while performing any operations such as reading, encoding etc.
>
> Here below is the system information:
>
> System Information (Apache Ignite Clusters)
> Heap size: 32GB out of 64GB of memory
> CPUs: Standard_F64s_v2
> The number of server instances: 24
> IOWait: within normal range
> IOPS: 10k-600K per hr
> JDK: Oracle JDK8, 1.8.0_281
> APACHE IGNITE - 2.14.0
>
> GC configuration in ignite.sh:
>
> XX:-UseContainerSupport
> -XX:+AlwaysPreTouch
> -XX:+UseG1GC
> -XX:+ScavengeBeforeFullGC
> -XX:+DisableExplicitGC"
>
> Please assist us in identifying the underlying source of this problem.
> Thank you,
> Srinivas
>
Re: Performance Issue at Ignite Level
Posted by Anton Vinogradov <av...@apache.org>.
Hi, please use 'user list' to ask the questions.
But in this case, I can recomend you to check GC logs. It seems, for some
reason, you have huge GC pauses.
On Sat, Mar 9, 2024 at 12:08 PM Kalwakuntla, Srinivas
<sr...@accenture.com.invalid> wrote:
> Dear Ignite Team,
>
> Hope this email finds your well !
>
> Reaching out to you, as currently we are facing performance issue with
> Apache Ignite. Our application is running on Azure Kubernetes (v1.27.1),
> which has 24 nodes of size "Standard_F64s_v2" (handling roughly 13million
> of records).
> Frequently we are getting
> "[WARNING][jvm-pause-detector-worker][IgniteKernal] Possible too long JVM
> pause: xxxx milliseconds", while performing any operations such as
> reading, encoding etc.
>
> Here below is the system information:
>
> System Information (Apache Ignite Clusters)
> Heap size: 32GB out of 64GB of memory
> CPUs: Standard_F64s_v2
> The number of server instances: 24
> IOWait: within normal range
> IOPS: 10k-600K per hr
> JDK: Oracle JDK8, 1.8.0_281
> APACHE IGNITE - 2.14.0
>
> GC configuration in ignite.sh:
>
> XX:-UseContainerSupport
> -XX:+AlwaysPreTouch
> -XX:+UseG1GC
> -XX:+ScavengeBeforeFullGC
> -XX:+DisableExplicitGC"
>
> Please assist us in identifying the underlying source of this problem.
> Thank you,
> Srinivas
>
> ________________________________
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Where allowed
> by local law, electronic communications with Accenture and its affiliates,
> including e-mail and instant messaging (including content), may be scanned
> by our systems for the purposes of information security, AI-powered support
> capabilities, and assessment of internal compliance with Accenture policy.
> Your privacy is important to us. Accenture uses your personal data only in
> compliance with data protection laws. For further information on how
> Accenture processes your personal data, please see our privacy statement at
> https://www.accenture.com/us-en/privacy-policy.
>
> ______________________________________________________________________________________
>
> www.accenture.com
>