You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2020/03/16 08:03:06 UTC
[GitHub] [skywalking] linliaoy opened a new issue #4524: Elasticsearch JVM
has too much memory and a long gc time
linliaoy opened a new issue #4524: Elasticsearch JVM has too much memory and a long gc time
URL: https://github.com/apache/skywalking/issues/4524
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [*] Question or discussion
- [ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Question
- What do you want to know?
If anyone has ever encountered elasticsearch, heap memory takes up a lot, less memory is freed, frequent fullgc, and fullgc takes a long time.
### Bug
- Which version of SkyWalking, OS and JRE?
- Which company or project?
- What happen?
I used version 6.6, es used 6.3.2, and found that it occupied a lot of heap memory, and the fullgc time was long.
___
### Requirement or improvement
- Please describe about your requirements or improvement suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4524: Elasticsearch JVM
has too much memory and a long gc time
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4524: Elasticsearch JVM has too much memory and a long gc time
URL: https://github.com/apache/skywalking/issues/4524#issuecomment-599400134
They are both old, try latest release. The whole storage structure has been changed for better performance, even the next 7.0.0 is better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4524: Elasticsearch JVM
has too much memory and a long gc time
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4524: Elasticsearch JVM has too much memory and a long gc time
URL: https://github.com/apache/skywalking/issues/4524#issuecomment-599404756
Sorry, can't get your point. We do nothing about ES heap memory. If you think 6.1 is better, you could continue to use on that. We rechecked with difference scenarios in the production, 6.3+ solution is always better than 6.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] linliaoy commented on issue #4524: Elasticsearch JVM
has too much memory and a long gc time
Posted by GitBox <gi...@apache.org>.
linliaoy commented on issue #4524: Elasticsearch JVM has too much memory and a long gc time
URL: https://github.com/apache/skywalking/issues/4524#issuecomment-599399283
I compared it with skywalking6.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] linliaoy edited a comment on issue #4524:
Elasticsearch JVM has too much memory and a long gc time
Posted by GitBox <gi...@apache.org>.
linliaoy edited a comment on issue #4524: Elasticsearch JVM has too much memory and a long gc time
URL: https://github.com/apache/skywalking/issues/4524#issuecomment-599399283
I compared it with skywalking6.1. oap error log :
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] linliaoy commented on issue #4524: Elasticsearch JVM
has too much memory and a long gc time
Posted by GitBox <gi...@apache.org>.
linliaoy commented on issue #4524: Elasticsearch JVM has too much memory and a long gc time
URL: https://github.com/apache/skywalking/issues/4524#issuecomment-599403129
The error log was 6.6 skywalking.
At present, the problem is still that the es heap memory is too large and a lot of memory cannot be released, which leads to the longer frequency and time of fullgc, which leads to the above error log. Compared with version 6.1, there is no such problem. The es configuration used by the two skywalking versions is the same.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4524: Elasticsearch JVM
has too much memory and a long gc time
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4524: Elasticsearch JVM has too much memory and a long gc time
URL: https://github.com/apache/skywalking/issues/4524#issuecomment-599398009
That is how ES works. I think we can't resolve this issue for ES. Maybe increasing `core/persistentPeriod` to longer refresh period could help you. But still your ES cluster is required to be powerful enough.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng closed issue #4524: Elasticsearch JVM has too
much memory and a long gc time
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #4524: Elasticsearch JVM has too much memory and a long gc time
URL: https://github.com/apache/skywalking/issues/4524
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] linliaoy edited a comment on issue #4524:
Elasticsearch JVM has too much memory and a long gc time
Posted by GitBox <gi...@apache.org>.
linliaoy edited a comment on issue #4524: Elasticsearch JVM has too much memory and a long gc time
URL: https://github.com/apache/skywalking/issues/4524#issuecomment-599399283
I compared it with skywalking6.1. oap error log :2020-03-16 16:11:16,093 - org.apache.skywalking.oap.server.core.remote.client.GRPCRemoteClient -2159940 [grpc-default-executor-18] ERROR [] - DEADLINE_EXCEEDED: deadline exceeded after 19999937177ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 19999937177ns
at io.grpc.Status.asRuntimeException(Status.java:526) ~[grpc-core-1.15.1.jar:1.15.1]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] linliaoy commented on issue #4524: Elasticsearch JVM
has too much memory and a long gc time
Posted by GitBox <gi...@apache.org>.
linliaoy commented on issue #4524: Elasticsearch JVM has too much memory and a long gc time
URL: https://github.com/apache/skywalking/issues/4524#issuecomment-599413983
Okay, thanks. I'll look into it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services