You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2020/09/17 03:44:37 UTC
[GitHub] [skywalking] sodared opened a new issue #5505: Elasticsearch exception
sodared opened a new issue #5505:
URL: https://github.com/apache/skywalking/issues/5505
Please answer these questions before submitting your issue.
- Why do you submit this issue?
Question
___
### Question
- What do you want to know?
What caused the failed information in log
___
### Bug
- Which version of SkyWalking, OS and JRE?
SkyWalking6.1.0 CentOS7 Elasticsearch6.8 jdk1.8
- What happened?
If possible, provide a way to reproduce the error. e.g. demo application, component version.
IN skywalking-oap-server.log sometimes exist error
################################################################################
org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker - 113 [DataCarrier.REGISTER_L2.BulkConsumePool.0.Thread] ERROR [] - Elasticsearch exception [type=es_rejected_execution_exception, reason=rejected execution of processing of [26498660][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[elasticsearch_service_instance_inventory][1]] containing [index {[elasticsearch_service_instance_inventory][type][2_db832d795d464164afc35af6d9ceb76f_0_0], source[{"sequence":171,"heartbeat_time":1600312410914,"service_id":2,"address_id":0,"name":"common-pid:17364@ecs-test","is_address":0,"instance_uuid":"db832d795d464164afc35af6d9ceb76f","register_time":1599613931154,"properties":"{\"os_name\":\"Linux\",\"host_name\":\"ecs-test\",\"process_no\":\"17364\",\"language\":\"java\",\"ipv4s\":\"[\\\"192.9.250.121\\\"]\"}"}]}] and a refresh, target allocation id: iDMVGloXS5i5ebYP9lTpZA, primary term: 2 on EsThreadPoolExecutor[name = node-1/write, queue capacity = 200, or
g.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6ec213bb[Running, pool size = 16, active threads = 16, queued tasks = 200, completed tasks = 3999038]]]
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=es_rejected_execution_exception, reason=rejected execution of processing of [26498660][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[elasticsearch_service_instance_inventory][1]] containing [index {[elasticsearch_service_instance_inventory][type][2_db832d795d464164afc35af6d9ceb76f_0_0], source[{"sequence":171,"heartbeat_time":1600312410914,"service_id":2,"address_id":0,"name":"common-pid:17364@ecs-test","is_address":0,"instance_uuid":"db832d795d464164afc35af6d9ceb76f","register_time":1599613931154,"properties":"{\"os_name\":\"Linux\",\"host_name\":\"ecs-test\",\"process_no\":\"17364\",\"language\":\"java\",\"ipv4s\":\"[\\\"192.9.250.121\\\"]\"}"}]}] and a refresh, target allocation id: iDMVGloXS5i5ebYP9lTpZA, primary term: 2 on EsThreadPoolExecutor[name = node-1/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6ec213bb[Running, pool size = 16, activ
e threads = 16, queued tasks = 200, completed tasks = 3999038]]]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:653) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:628) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:535) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.update(RestHighLevelClient.java:366) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.forceUpdate(ElasticSearchClient.java:213) ~[library-client-6.1.0.jar:6.1.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.RegisterEsDAO.forceUpdate(RegisterEsDAO.java:60) ~[storage-elasticsearch-plugin-6.1.0.jar:6.1.0]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.lambda$onWork$0(RegisterPersistentWorker.java:90) ~[server-core-6.1.0.jar:6.1.0]
at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_212]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.onWork(RegisterPersistentWorker.java:85) [server-core-6.1.0.jar:6.1.0]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.access$100(RegisterPersistentWorker.java:36) [server-core-6.1.0.jar:6.1.0]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker$PersistentConsumer.consume(RegisterPersistentWorker.java:142) [server-core-6.1.0.jar:6.1.0]
at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.consume(MultipleChannelsConsumer.java:80) [apm-datacarrier-6.1.0.jar:6.1.0]
at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:49) [apm-datacarrier-6.1.0.jar:6.1.0]
Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://elasticsearch.eis.smec-cn.com:9200], URI [/elasticsearch_service_instance_inventory/type/2_db832d795d464164afc35af6d9ceb76f_0_0/_update?refresh=true&timeout=1m], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[node-1][192.9.250.113:9300][indices:data/write/update[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of processing of [26498660][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[elasticsearch_service_instance_inventory][1]] containing [index {[elasticsearch_service_instance_inventory][type][2_db832d795d464164afc35af6d9ceb76f_0_0], source[{\"sequence\":171,\"heartbeat_time\":1600312410914,\"service_id\":2,\"address_id\":0,\"name\":\"common-pid:17364@ecs-test\",\"is_address\":0,\"instance_uuid\":\"db832d795d464164afc35af6d9ceb76f\",\"register_time\":1599613931154,\"properties\":\"{\\\"os_name\\\":\\\"Linux\\\",\\\"host_name\\\":\\\"ecs-test\\\",\\\"process_no\\\":\\\"17364\\\",\\\"language\\\":\\\"java\\\",\\\"ipv4s\\\":\\\"[\\\\\\\"192.9.250.121\\\\\\\"]\\\"}\"}]}] and a refresh, target allocation id: iDMVGloXS5i5ebYP9lTpZA, primary term: 2 on EsThreadPoolExecutor[name =
node-1/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6ec213bb[Running, pool size = 16, active threads = 16, queued tasks = 200, completed tasks = 3999038]]"},"status":429}
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:705) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:522) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.update(RestHighLevelClient.java:366) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.forceUpdate(ElasticSearchClient.java:213) ~[library-client-6.1.0.jar:6.1.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.RegisterEsDAO.forceUpdate(RegisterEsDAO.java:60) ~[storage-elasticsearch-plugin-6.1.0.jar:6.1.0]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.lambda$onWork$0(RegisterPersistentWorker.java:90) ~[server-core-6.1.0.jar:6.1.0]
at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_212]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.onWork(RegisterPersistentWorker.java:85) [server-core-6.1.0.jar:6.1.0]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.access$100(RegisterPersistentWorker.java:36) [server-core-6.1.0.jar:6.1.0]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker$PersistentConsumer.consume(RegisterPersistentWorker.java:142) [server-core-6.1.0.jar:6.1.0]
at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.consume(MultipleChannelsConsumer.java:80) [apm-datacarrier-6.1.0.jar:6.1.0]
at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:49) [apm-datacarrier-6.1.0.jar:6.1.0]
Caused by: org.elasticsearch.client.ResponseException: method [POST], host [http://elasticsearch.eis.smec-cn.com:9200], URI [/elasticsearch_service_instance_inventory/type/2_db832d795d464164afc35af6d9ceb76f_0_0/_update?refresh=true&timeout=1m], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[node-1][192.9.250.113:9300][indices:data/write/update[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of processing of [26498660][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[elasticsearch_service_instance_inventory][1]] containing [index {[elasticsearch_service_instance_inventory][type][2_db832d795d464164afc35af6d9ceb76f_0_0], source[{\"sequence\":171,\"heartbeat_time\":1600312410914,\"service_id\":2,\"address_id\":0,\"name\":\"common-pid:17364@ecs-test\",\"is_address\":0,\"instance_uuid\":\"db832d795d464164afc35af6d9ceb76f\",\"register_time\":1599613931154,\"properties\":\"{\\\"os_name\\\":\\\"Linux\\\",\\\"host_name\\\":\\\"ecs-test\\\",\\\"process_no\\\":\\\"17364\\\",\\\"language\\\":\\\"java\\\",\\\"ipv4s\\\":\\\"[\\\\\\\"192.9.250.121\\\\\\\"]\\\"}\"}]}] and a refresh, target allocation id: iDMVGloXS5i5ebYP9lTpZA, primary term: 2 on EsThreadPoolExecutor[name =
node-1/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6ec213bb[Running, pool size = 16, active threads = 16, queued tasks = 200, completed tasks = 3999038]]"},"status":429}
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119) ~[httpcore-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) ~[httpasyncclient-4.1.2.jar:4.1.2]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) ~[httpcore-nio-4.4.5.jar:4.4.5]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]
###################################################################################
___
### Requirement or improvement
- Please describe your requirements or improvement suggestions.
Please help me for this error.
appreciate!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sodared commented on issue #5505: Elasticsearch exception
Posted by GitBox <gi...@apache.org>.
sodared commented on issue #5505:
URL: https://github.com/apache/skywalking/issues/5505#issuecomment-693809384
确实老。 seems i have to update the skw。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sodared commented on issue #5505: Elasticsearch exception
Posted by GitBox <gi...@apache.org>.
sodared commented on issue #5505:
URL: https://github.com/apache/skywalking/issues/5505#issuecomment-693790770
ps: my ES is 3-nodes cluster which is green health showing in kibana
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #5505: Elasticsearch exception
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #5505:
URL: https://github.com/apache/skywalking/issues/5505
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #5505: Elasticsearch exception
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #5505:
URL: https://github.com/apache/skywalking/issues/5505#issuecomment-693791296
6.1.0 is a very old version. Why do you still deploy that? The latest release has been 8.1.0. We can't support that old release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org