You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2019/09/06 03:07:52 UTC
[GitHub] [skywalking] liyongfa0311 opened a new issue #3424: Why does the
collector always crash after I upgrade to version 6.3?
liyongfa0311 opened a new issue #3424: Why does the collector always crash after I upgrade to version 6.3?
URL: https://github.com/apache/skywalking/issues/3424
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [X] Question or discussion
- [ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Question
- What do you want to know?
After I deleted all the indexes in es, I started the 6.3 version of the collector, and then started all the agent services. After about 1 hour, the collector log showed various exceptions, such as
2019-09-06 10:44:26,321 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler - 64 [grpc-default-executor-575] ERROR [] - CANCELLED: cancelled before receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
at io.grpc.Status.asRuntimeException(Status.java:517) ~[grpc-core-1.15.1.jar:1.15.1]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:272) [grpc-stub-1.15.1.jar:1.15.1]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:293) [grpc-core-1.15.1.jar:1.15.1]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:738) [grpc-core-1.15.1.jar:1.15.1]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.15.1.jar:1.15.1]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.15.1.jar:1.15.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
After it multiple occurrences
2019-09-06 10:44:30,572 - org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker - 112 [DataCarrier.REGISTER_L2.BulkConsumePool.0.Thread] ERROR [] - GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2019-09-06 10:44:30,573 - org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker - 112 [DataCarrier.REGISTER_L2.BulkConsumePool.0.Thread] ERROR [] - GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2019-09-06 10:44:30,573 - org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker - 112 [DataCarrier.REGISTER_L2.BulkConsumePool.0.Thread] ERROR [] - GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2019-09-06 10:44:30,573 - org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker - 112 [DataCarrier.REGISTER_L2.BulkConsumePool.0.Thread] ERROR [] - GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2019-09-06 10:44:30,574 - org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker - 112 [DataCarrier.REGISTER_L2.BulkConsumePool.0.Thread] ERROR [] - GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2019-09-06 10:44:30,574 - org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker - 112 [DataCarrier.REGISTER_L2.BulkConsumePool.0.Thread] ERROR [] - GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
Still not working after restarting collector ，There may be an exception
2019-09-06 00:47:24,324 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch.cache.NetworkAddressInventoryCacheEsDAO - 95 [pool-14-thread-1] ERROR [] - Request cannot be executed; I/O reactor status: STOPPED
java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
at org.apache.http.util.Asserts.check(Asserts.java:46) ~[httpcore-4.4.5.jar:4.4.5]
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90) ~[httpasyncclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123) ~[httpasyncclient-4.1.2.jar:4.1.2]
at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:366) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient.performRequestAsyncNoCatch(RestClient.java:351) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:522) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:404) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.search(ElasticSearchClient.java:220) ~[library-client-6.3.0.jar:6.3.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.cache.NetworkAddressInventoryCacheEsDAO.loadLastUpdate(NetworkAddressInventoryCacheEsDAO.java:89) ~[storage-elasticsearch-plugin-6.3.0.jar:6.3.0]
at org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer.updateNetAddressInventory(CacheUpdateTimer.java:83) ~[server-core-6.3.0.jar:6.3.0]
at org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer.update(CacheUpdateTimer.java:57) ~[server-core-6.3.0.jar:6.3.0]
at org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer.lambda$start$0(CacheUpdateTimer.java:48) ~[server-core-6.3.0.jar:6.3.0]
at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:36) [apm-util-6.3.0.jar:6.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_65]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_65]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_65]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
It may also appear
java.io.IOException: listener timeout after waiting for [30000] ms
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:684) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:522) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.elasticsearch.client.RestHighLevelClient.get(RestHighLevelClient.java:293) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.get(ElasticSearchClient.java:226) ~[library-client-6.3.0.jar:6.3.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.cache.EndpointInventoryCacheEsDAO.getEndpointId(EndpointInventoryCacheEsDAO.java:49) ~[storage-elasticsearch-plugin-6.3.0.jar:6.3.0]
at org.apache.skywalking.oap.server.core.cache.EndpointInventoryCache.getEndpointId(EndpointInventoryCache.java:69) ~[server-core-6.3.0.jar:6.3.0]
at org.apache.skywalking.oap.server.core.register.service.EndpointInventoryRegister.getOrCreate(EndpointInventoryRegister.java:54) ~[server-core-6.3.0.jar:6.3.0]
at org.apache.skywalking.oap.server.receiver.register.provider.handler.v6.grpc.RegisterServiceHandler.lambda$doEndpointRegister$2(RegisterServiceHandler.java:140) ~[skywalking-register-receiver-plugin-6.3.0.jar:6.3.0]
at java.util.ArrayList.forEach(ArrayList.java:1249) [?:1.8.0_65]
at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080) [?:1.8.0_65]
at org.apache.skywalking.oap.server.receiver.register.provider.handler.v6.grpc.RegisterServiceHandler.doEndpointRegister(RegisterServiceHandler.java:136) [skywalking-register-receiver-plugin-6.3.0.jar:6.3.0]
at org.apache.skywalking.apm.network.register.v2.RegisterGrpc$MethodHandlers.invoke(RegisterGrpc.java:513) [apm-network-6.3.0.jar:6.3.0]
at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) [grpc-stub-1.15.1.jar:1.15.1]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) [grpc-core-1.15.1.jar:1.15.1]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707) [grpc-core-1.15.1.jar:1.15.1]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.15.1.jar:1.15.1]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.15.1.jar:1.15.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
After these abnormalities occur, the machine cpu is super high.I want to know how I can do this to ensure that the collector works properly. Es is without any problems，The above exceptions will occur in both the cluster and the single . I have restarted the collector not only 10 times, but the collector does not work after an hour.
___
### Bug
- Which version of SkyWalking, OS and JRE?
skywalking 6.3.0 version
OS CentOs7.3
JRE 1.9
- Which company or project?
- What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.
___
### Requirement or improvement
- Please describe about your requirements or improvement suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services