You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "dingsongjie (via GitHub)" <gi...@apache.org> on 2024/07/11 02:41:45 UTC
[I] [Bug] Skywalking OAP heap memory leaks v9.7.0 [skywalking]
dingsongjie opened a new issue, #12429:
URL: https://github.com/apache/skywalking/issues/12429
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar issues.
### Apache SkyWalking Component
OAP server (apache/skywalking)
### What happened
When I upgraded SkyWalking from 8.x to 9.7.0, I found that after a while, oap would become unresponsive.I tried setting more memory but it didn't help. Here are some log snippets
```log
io.netty.channel.ChannelPipelineException: io.grpc.netty.NettyServer$1.handlerAdded() has thrown an exception; removed.
at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:624) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) [netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
2024-07-10 08:08:39,458 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-4] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,458 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-4] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,458 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-4] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,458 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-4] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,458 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-4] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,458 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-3] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,458 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-4] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,458 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-3] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,458 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-4] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,458 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-3] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,458 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-4] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,459 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-3] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,459 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-4] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,459 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-3] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,459 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-4] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,459 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-3] WARN [] - Grpc server thread pool is full, rejecting the task
2024-07-10 08:08:39,459 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-4] WARN [] - Grpc server thread pool is full, rejecting the task
```
This is the full logs .
[sky-2.log](https://github.com/user-attachments/files/16170801/sky-2.log)
I grabbed the dump file and ran it through the memory diagnostics tool to get the 3 zip files.
[heapdump_Leak_Suspects.zip](https://github.com/user-attachments/files/16170788/heapdump_Leak_Suspects.zip)
[heapdump_System_Overview.zip](https://github.com/user-attachments/files/16170789/heapdump_System_Overview.zip)
[heapdump_Top_Components.zip](https://github.com/user-attachments/files/16170790/heapdump_Top_Components.zip)
### What you expected to happen
I expect that skywalking oap will not have memory leaks after setting enough memory. Even if there is insufficient memory, the service should restart immediately after the health check fails. The service runs in k8s.
### How to reproduce
I'm not sure if I can reproduce it ，but here are my helm values
```yaml
nameOverride: "skywaling"
oap:
javaOpts: -Xmx4g -Xms4g
storageType: elasticsearch
image:
tag: 9.7.0
replicas: 1
resources:
limits:
cpu: 4
memory: 5Gi
requests:
cpu: 4
memory: 5Gi
env:
SW_STORAGE: elasticsearch
SW_CORE_RECORD_DATA_TTL: 5
SW_CORE_METRICS_DATA_TTL: 7
SW_STORAGE_ES_QUERY_SEGMENT_SIZE: "1500"
SW_NAMESPACE: sw9
SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS: k8s-mesh
SW_ENDPOINT_NAME_MAX_LENGTH: "100"
SW_RECEIVER_GRPC_POOL_QUEUE_SIZE: "15000"
SW_STORAGE_ES_RESPONSE_TIMEOUT: "15000"
envoy:
als:
enabled: true
ui:
image:
tag: 9.7.0
elasticsearch:
enabled: false
config:
```
### Anything else
_No response_
### Are you willing to submit a pull request to fix on your own?
- [ ] Yes I am willing to submit a pull request on my own!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Skywalking OAP heap memory leaks v9.7.0 [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng closed issue #12429: [Bug] Skywalking OAP heap memory leaks v9.7.0
URL: https://github.com/apache/skywalking/issues/12429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org