You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2018/11/23 06:44:07 UTC
[GitHub] sunliang3 opened a new issue #1952: CPU utilization 100% &
elasticsearch 429 too many requests
sunliang3 opened a new issue #1952: CPU utilization 100% & elasticsearch 429 too many requests
URL: https://github.com/apache/incubator-skywalking/issues/1952
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [-] Question or discussion
- [ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
### Bug
- Which version of SkyWalking, OS and JRE?
6.0.0-alpha
- Which company or project?
- What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.
After the service runs for a period of time, the CPU utilization reaches 100%. Print stack information to find nearly one thousand pieces of information as follows:
"grpc-default-executor-1226" #1549 daemon prio=5 os_prio=0 tid=0x00002b7264108800 nid=0x305f sleeping[0x00002b7328230000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.skywalking.apm.commons.datacarrier.buffer.Buffer.save(Buffer.java:64)
at org.apache.skywalking.apm.commons.datacarrier.buffer.Channels.save(Channels.java:52)
at org.apache.skywalking.apm.commons.datacarrier.DataCarrier.produce(DataCarrier.java:88)
at org.apache.skywalking.oap.server.core.analysis.worker.RecordPersistentWorker.in(RecordPersistentWorker.java:53)
at org.apache.skywalking.oap.server.core.analysis.worker.RecordProcess.in(RecordProcess.java:39)
at org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentDispatcher.dispatch(SegmentDispatcher.java:44)
at org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentDispatcher.dispatch(SegmentDispatcher.java:28)
at org.apache.skywalking.oap.server.core.analysis.DispatcherManager.forward(DispatcherManager.java:71)
at org.apache.skywalking.oap.server.core.source.SourceReceiverImpl.receive(SourceReceiverImpl.java:35)
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.listener.segment.SegmentSpanListener.build(SegmentSpanListener.java:98)
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParse$$Lambda$262/1249197766.accept(Unknown Source)
at java.lang.Iterable.forEach(Iterable.java:75)
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParse.notifyListenerToBuild(SegmentParse.java:169)
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParse.parse(SegmentParse.java:77)
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParse$Producer.send(SegmentParse.java:234)
at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v5.grpc.TraceSegmentServiceHandler$1.onNext(TraceSegmentServiceHandler.java
:49)
at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v5.grpc.TraceSegmentServiceHandler$1.onNext(TraceSegmentServiceHandler.java
:43)
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:252)
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:629)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I also got many errors in the skywalking-oap-server.log as follows:
org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker -1943661 [DataCarrier.IndicatorPersistentWorker.service_inventory.Consumser.0.Thread] ERROR [] - Elasticsearch exception [type=es_rejected_execution_exception, reason=rejected execution of org.elasticsearch.transport.TransportService$7@110d08cf on EsThreadPoolExecutor[name = WRvqRnR/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@513e0b13[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 1802831]]]
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=es_rejected_execution_exception, reason=rejected execution of org.elasticsearch.transport.TransportService$7@110d08cf on EsThreadPoolExecutor[name = WRvqRnR/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@513e0b13[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 1802831]]]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:653) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:628) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:535) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.update(RestHighLevelClient.java:366) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.forceUpdate(ElasticSearchClient.java:180) ~[library-client-6.0.0-alpha.jar:6.0.0-alpha]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.RegisterEsDAO.forceUpdate(RegisterEsDAO.java:78) ~[storage-elasticsearch-plugin-6.0.0-alpha.jar:6.0.0-alpha]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.lambda$onWork$0(RegisterPersistentWorker.java:79) ~[server-core-6.0.0-alpha.jar:6.0.0-alpha]
at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_171]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.onWork(RegisterPersistentWorker.java:74) [server-core-6.0.0-alpha.jar:6.0.0-alpha]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.access$100(RegisterPersistentWorker.java:35) [server-core-6.0.0-alpha.jar:6.0.0-alpha]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker$PersistentConsumer.consume(RegisterPersistentWorker.java:120) [server-core-6.0.0-alpha.jar:6.0.0-alpha]
at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.consume(ConsumerThread.java:101) [apm-datacarrier-6.0.0-alpha.jar:6.0.0-alpha]
at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.run(ConsumerThread.java:68) [apm-datacarrier-6.0.0-alpha.jar:6.0.0-alpha]
Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [...], URI [/service_inventory/type/question-center_0_0/_update?refresh=true&timeout=1m], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[WRvqRnR][10.6.23.132:9300][indices:data/write/update[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@110d08cf on EsThreadPoolExecutor[name = WRvqRnR/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@513e0b13[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 1802831]]"},"status":429}
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:705) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:522) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.update(RestHighLevelClient.java:366) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.forceUpdate(ElasticSearchClient.java:180) ~[library-client-6.0.0-alpha.jar:6.0.0-alpha]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.RegisterEsDAO.forceUpdate(RegisterEsDAO.java:78) ~[storage-elasticsearch-plugin-6.0.0-alpha.jar:6.0.0-alpha]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.lambda$onWork$0(RegisterPersistentWorker.java:79) ~[server-core-6.0.0-alpha.jar:6.0.0-alpha]
at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_171]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.onWork(RegisterPersistentWorker.java:74) [server-core-6.0.0-alpha.jar:6.0.0-alpha]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.access$100(RegisterPersistentWorker.java:35) [server-core-6.0.0-alpha.jar:6.0.0-alpha]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker$PersistentConsumer.consume(RegisterPersistentWorker.java:120) [server-core-6.0.0-alpha.jar:6.0.0-alpha]
at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.consume(ConsumerThread.java:101) [apm-datacarrier-6.0.0-alpha.jar:6.0.0-alpha]
at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.run(ConsumerThread.java:68) [apm-datacarrier-6.0.0-alpha.jar:6.0.0-alpha]
Caused by: org.elasticsearch.client.ResponseException: method [POST], host [...], URI [/service_inventory/type/question-center_0_0/_update?refresh=true&timeout=1m], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[WRvqRnR][10.6.23.132:9300][indices:data/write/update[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@110d08cf on EsThreadPoolExecutor[name = WRvqRnR/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@513e0b13[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 1802831]]"},"status":429}
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119) ~[httpcore-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) ~[httpasyncclient-4.1.2.jar:4.1.2]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) ~[httpcore-nio-4.4.5.jar:4.4.5]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_171]
___
### Requirement or improvement
- Please describe about your requirements or improvement suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services