You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "helloworld113 (via GitHub)" <gi...@apache.org> on 2023/10/07 11:51:04 UTC
[I] [Bug] OAP server Memory usage continues to increase [skywalking]
helloworld113 opened a new issue, #11377:
URL: https://github.com/apache/skywalking/issues/11377
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar issues.
### Apache SkyWalking Component
OAP server (apache/skywalking)
### What happened
I can't figure out the reason for the memory growth, I try to restart and change the config is not work.
skywalking is running in k8s with kafka-fetcher,this is the config:
gRPCThreadPoolSize and gRPCThreadPoolQueueSize config to low size,but the mem increase also.
```
core:
selector: ${SW_CORE:default}
default:
# Mixed: Receive agent data, Level 1 aggregate, Level 2 aggregate
# Receiver: Receive agent data, Level 1 aggregate
# Aggregator: Level 2 aggregate
role: ${SW_CORE_ROLE:Mixed} # Mixed/Receiver/Aggregator
restHost: ${SW_CORE_REST_HOST:0.0.0.0}
restPort: ${SW_CORE_REST_PORT:12800}
restContextPath: ${SW_CORE_REST_CONTEXT_PATH:/}
restMaxThreads: ${SW_CORE_REST_MAX_THREADS:200}
restIdleTimeOut: ${SW_CORE_REST_IDLE_TIMEOUT:30000}
restAcceptQueueSize: ${SW_CORE_REST_QUEUE_SIZE:0}
httpMaxRequestHeaderSize: ${SW_CORE_HTTP_MAX_REQUEST_HEADER_SIZE:8192}
gRPCHost: ${SW_CORE_GRPC_HOST:0.0.0.0}
gRPCPort: ${SW_CORE_GRPC_PORT:11800}
maxConcurrentCallsPerConnection: ${SW_CORE_GRPC_MAX_CONCURRENT_CALL:100}
maxMessageSize: ${SW_CORE_GRPC_MAX_MESSAGE_SIZE:0}
gRPCThreadPoolQueueSize: ${SW_CORE_GRPC_POOL_QUEUE_SIZE:5}
gRPCThreadPoolSize: ${SW_CORE_GRPC_THREAD_POOL_SIZE:2}
gRPCSslEnabled: ${SW_CORE_GRPC_SSL_ENABLED:false}
gRPCSslKeyPath: ${SW_CORE_GRPC_SSL_KEY_PATH:""}
gRPCSslCertChainPath: ${SW_CORE_GRPC_SSL_CERT_CHAIN_PATH:""}
gRPCSslTrustedCAPath: ${SW_CORE_GRPC_SSL_TRUSTED_CA_PATH:""}
downsampling:
- Hour
- Day
...
```
kafka-fetcher Thread set to small:
```
kafka-fetcher:
selector: ${SW_KAFKA_FETCHER:default}
default:
bootstrapServers: ${SW_KAFKA_FETCHER_SERVERS:kafka-controller-0.kafka-controller-headless.skywalking.svc.cluster.local:9092,kafka-controller-1.kafka-controller-headless.skywalking.svc.cluster.local:9092,kafka-controller-2.kafka-controller-headless.skywalking.svc.cluster.local:9092}
namespace: ${SW_NAMESPACE:""}
partitions: ${SW_KAFKA_FETCHER_PARTITIONS:3}
replicationFactor: ${SW_KAFKA_FETCHER_PARTITIONS_FACTOR:2}
enableNativeProtoLog: ${SW_KAFKA_FETCHER_ENABLE_NATIVE_PROTO_LOG:true}
enableNativeJsonLog: ${SW_KAFKA_FETCHER_ENABLE_NATIVE_JSON_LOG:true}
consumers: ${SW_KAFKA_FETCHER_CONSUMERS:2}
kafkaHandlerThreadPoolSize: ${SW_KAFKA_HANDLER_THREAD_POOL_SIZE:5}
kafkaHandlerThreadPoolQueueSize: ${SW_KAFKA_HANDLER_THREAD_POOL_QUEUE_SIZE:10}
```
sample rate noly set to 10:
```
receiver-zipkin:
selector: ${SW_RECEIVER_ZIPKIN:-}
default:
# Defines a set of span tag keys which are searchable.
# The max length of key=value should be less than 256 or will be dropped.
searchableTracesTags: ${SW_ZIPKIN_SEARCHABLE_TAG_KEYS:http.method}
# The sample rate precision is 1/10000, should be between 0 and 10000
sampleRate: ${SW_ZIPKIN_SAMPLE_RATE:10}
```
jvm limit：
```
javaOpts: -Xmx10g -Xms2g
```
I tried it, these configurations are effective,but When I redeployed the skywalking service in k8s, the memory grew to the jvm limit of 10G after about 15 minutes, then the oap server is not work.
this is I can find only one error in oap server:
```
org.apache.skywalking.oap.server.core.remote.client.GRPCRemoteClient 224 [grpc-default-executor-12] ERROR [] - [9.6.0-fb61128] DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [closed=[CANCELLED], committed=[remote_addr=10.50.238.16/10.50.238.16:11800]]
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [closed=[CANCELLED], committed=[remote_addr=10.50.238.16/10.50.238.16:11800]]
at io.grpc.Status.asRuntimeException(Status.java:539) ~[grpc-api-1.53.0.jar:1.53.0]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487) [grpc-stub-1.53.0.jar:1.53.0]
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:576) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:757) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:736) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.53.0.jar:1.53.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) [grpc-core-1.53.0.jar:1.53.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
```
### What you expected to happen
It works normally at first, but after a while the memory starts to fill up. I have no idea to find out what happen.
### How to reproduce
deploy oap from the official doc [[deploying SkyWalking backend to Kubernetes cluster](https://github.com/apache/skywalking-kubernetes)](https://github.com/apache/skywalking-helm)
the version:
oap server and ui :9.6
es: 8.9.2
### Anything else
_No response_
### Are you willing to submit a pull request to fix on your own?
- [ ] Yes I am willing to submit a pull request on my own!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] OAP server Memory usage continues to increase [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng closed issue #11377: [Bug] OAP server Memory usage continues to increase
URL: https://github.com/apache/skywalking/issues/11377
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org