You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2020/03/12 08:25:18 UTC
[GitHub] [skywalking] jol5 opened a new issue #4502: SkyWalking not work
after two date
jol5 opened a new issue #4502: SkyWalking not work after two date
URL: https://github.com/apache/skywalking/issues/4502
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [*] Question or discussion
- [*] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Question
- What do you want to know?
SkyWalking not work after two date ，i can got lots of error information like follow in SkyWalking-oap docker container：
2020-03-12 15:55:30,184 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch.cache.NetworkAddressInventoryCacheEsDAO -338999 [pool-11-thread-1] ERROR [] - String index out of range: -1
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1931) ~[?:1.8.0_212]
at org.elasticsearch.client.RequestConverters$EndpointBuilder.encodePart(RequestConverters.java:1131) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]t org.elasticsearch.client.RequestConverters$EndpointBuilder.addPathPart(RequestConverters.java:1095) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RequestConverters.endpoint(RequestConverters.java:714) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RequestConverters.getStyleRequest(RequestConverters.java:265) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RequestConverters.get(RequestConverters.java:261) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1429) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1403) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1373) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.get(RestHighLevelClient.java:699) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client.get(ElasticSearch7Client.java:187) ~[storage-elasticsearch7-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.cache.NetworkAddressInventoryCacheEsDAO.getAddressId(NetworkAddressInventoryCacheEsDAO.java:52) ~[storage-elasticsearch-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.core.cache.NetworkAddressInventoryCache.getAddressId(NetworkAddressInventoryCache.java:67) ~[server-core-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.core.register.service.NetworkAddressInventoryRegister.getOrCreate(NetworkAddressInventoryRegister.java:70) ~[server-core-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.standardization.ReferenceIdExchanger.exchange(ReferenceIdExchanger.java:102) ~[skywalking-trace-receiver-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2.preBuild(SegmentParseV2.java:166) ~[skywalking-trace-receiver-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2.parse(SegmentParseV2.java:104) ~[skywalking-trace-receiver-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2$Producer.call(SegmentParseV2.java:288) ~[skywalking-trace-receiver-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.library.buffer.DataStreamReader.reCall(DataStreamReader.java:163) ~[library-buffer-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.library.buffer.DataStreamReader.read(DataStreamReader.java:150) ~[library-buffer-6.6.0.jar:6.6.0]
at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:36) [apm-util-6.6.0.jar:6.6.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
2020-0312 15:55:30,432 - org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker -339247 [DataCarrier.REGISTER_L2.BulkConsumePool.0.Thread] ERROR [] - String index out of range: -1
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1931) ~[?:1.8.0_212]
at org.elasticsearch.client.RequestConverters$EndpointBuilder.encodePart(RequestConverters.java:1131) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RequestConverters$EndpointBuilder.addPathPart(RequestConverters.java:1095) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RequestConverters.endpoint(RequestConverters.java:714) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RequestConverters.getStyleRequest(RequestConverters.java:265) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RequestConverters.get(RequestConverters.java:261) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1429) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1403) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1373) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.get(RestHighLevelClient.java:699) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client.get(ElasticSearch7Client.java:187) ~[storage-elasticsearch7-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.RegisterEsDAO.get(RegisterEsDAO.java:41) ~[storage-elasticsearch-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.lambda$onWork$0(RegisterPersistentWorker.java:97) ~[server-core-6.6.0.jar:6.6.0]
at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_212]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.onWork(RegisterPersistentWorker.java:95) [server-core-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.access$100(RegisterPersistentWorker.java:40) [server-core-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker$PersistentConsumer.consume(RegisterPersistentWorker.java:153) [server-core-6.6.0.jar:6.6.0]
at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.consume(MultipleChannelsConsumer.java:81) [apm-datacarrier-6.6.0.jar:6.6.0]
at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:52) [apm-datacarrier-6.6.0.jar:6.6.0]
2020-03-12 15:55:40,278 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -349093 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 10074 millis, size: 262
2020-03-12 15:55:46,265 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -355080 [I/O dispatcher 1] WARN [] - Bulk [16] executed with failures
2020-03-12 15:55:53,181 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -361996 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 9696 millis, size: 230
2020-03-12 15:55:56,266 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -365081 [I/O dispatcher 1] WARN [] - Bulk [17] executed with failures
2020-03-12 15:56:05,254 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -374069 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 8928 millis, size: 101
2020-03-12 15:56:16,266 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -385081 [I/O dispatcher 1] WARN [] - Bulk [18] executed with failures
020-03-12 15:56:18,248 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -387063 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 9818 millis, size: 150
2020-03-12 15:56:26,268 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -395083 [I/O dispatcher 1] WARN [] - Bulk [19] executed with failures
2020-03-12 15:56:26,971 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -395786 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 5570 millis, size: 83
2020-03-12 15:56:40,413 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -409228 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 10081 millis, size: 215
2020-03-12 15:56:46,269 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -415084 [I/O dispatcher 1] WARN [] - Bulk [20] executed with failures
2020-03-12 15:56:53,656 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -422471 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 10048 millis, size: 15
2020-03-12 15:56:56,269 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -425084 [I/O dispatcher 2] WARN [] - Bulk [21] executed with failures
2020-03-12 15:57:06,118 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -434933 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 9342 millis, size: 101
2020-03-12 15:57:16,270 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -445085 [I/O dispatcher 1] WARN [] - Bulk [22] executed with failures
2020-03-12 15:57:19,289 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -448104 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 9975 millis, size: 150
2020-03-12 15:57:26,271 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -455086 [I/O dispatcher 1] WARN [] - Bulk [23] executed with failures
2020-03-12 15:57:27,027 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -455842 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 4611 millis, size: 83
2020-03-12 15:57:40,419 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -469234 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 10090 millis, size: 215
2020-03-12 15:57:46,272 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -475087 [I/O dispatcher 1] WARN [] - Bulk [24] executed with failures，
==================
and the SkyWalking-ui docker container will gotsome log as follow：
2020-03-10 08:25:40.017 WARN 1 --- [io-8080-exec-10] o.s.c.n.z.filters.post.SendErrorFilter : Error during filtering
com.netflix.zuul.exception.ZuulException: Forwarding error
at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.handleException(RibbonRoutingFilter.java:189)
at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:164)
at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:111)
at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:112)
at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:193)
at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:157)
at com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:118)
at com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:96)
at com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:116)
at com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:81)
at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:157)
at org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequest(ZuulController.java:44)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.netflix.client.ClientException: null
at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:118)
at org.springframework.cloud.netflix.zuul.filters.route.support.AbstractRibbonCommand.run(AbstractRibbonCommand.java:152)
at org.springframework.cloud.netflix.zuul.filters.route.support.AbstractRibbonCommand.run(AbstractRibbonCommand.java:49)
at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:302)
at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:298)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.subscribe(Observable.java:10247)
at rx.Observable.subscribe(Observable.java:10214)
at rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:51)
at rx.observables.BlockingObservable.toFuture(BlockingObservable.java:411)
at com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:378)
at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:344)
at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:158)
... 67 common frames omitted
Caused by: java.lang.RuntimeException: org.apache.http.conn.HttpHostConnectException: Connect to oap:12800 [oap/172.23.0.2] failed: Connection refused (Connection refused)
at rx.exceptions.Exceptions.propagate(Exceptions.java:58)
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:464)
at rx.observables.BlockingObservable.single(BlockingObservable.java:341)
at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:112)
... 129 common frames omitted
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to oap:12800 [oap/172.23.0.2] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.springframework.cloud.netflix.ribbon.apache.RibbonLoadBalancingHttpClient.execute(RibbonLoadBalancingHttpClient.java:82)
at org.springframework.cloud.netflix.ribbon.apache.RibbonLoadBalancingHttpClient.execute(RibbonLoadBalancingHttpClient.java:42)
at com.netflix.client.AbstractLoadBalancerAwareClient$1.call(AbstractLoadBalancerAwareClient.java:104)
at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:303)
at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:287)
at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:231)
at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:228)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.drain(OnSubscribeConcatMap.java:286)
at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.onNext(OnSubscribeConcatMap.java:144)
at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:185)
at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94)
at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber$1.call(OperatorRetryWithPredicate.java:127)
at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73)
at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52)
at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:79)
at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:45)
at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:276)
at rx.Subscriber.setProducer(Subscriber.java:209)
at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:138)
at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:129)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.subscribe(Observable.java:10247)
at rx.Observable.subscribe(Observable.java:10214)
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:444)
... 131 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
... 174 common frames omitted
___
### Bug
- Which version of SkyWalking, OS and JRE?
apache/skywalking-oap-server:6.6.0-es7
docker.elastic.co/elasticsearch/elasticsearch:7.6.0
apache/skywalking-ui:6.6.0
- Which company or project?
- What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.
___
### Requirement or improvement
- Please describe about your requirements or improvement suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] jol5 commented on issue #4502: SkyWalking not work
after two date
Posted by GitBox <gi...@apache.org>.
jol5 commented on issue #4502: SkyWalking not work after two date
URL: https://github.com/apache/skywalking/issues/4502#issuecomment-598065853
until i delete all skywalking es index
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] jol5 commented on issue #4502: SkyWalking not work
after two date
Posted by GitBox <gi...@apache.org>.
jol5 commented on issue #4502: SkyWalking not work after two date
URL: https://github.com/apache/skywalking/issues/4502#issuecomment-598504662
> Seem like someone delete some of data before. And the OAP can't find the metadata.
i don't think so! beacuse it reappear always after one day.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4502: SkyWalking not work
after two date
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4502: SkyWalking not work after two date
URL: https://github.com/apache/skywalking/issues/4502#issuecomment-598562594
I don't know why. Usually deleting index and creating indexes at middle night cause more payload, especially for some some ES cluster. The next 7.0.0 will reduce 50% of the index number, should ease this for you. But basically, that is caused by ES cluster health issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] jol5 edited a comment on issue #4502: SkyWalking not
work after two date
Posted by GitBox <gi...@apache.org>.
jol5 edited a comment on issue #4502: SkyWalking not work after two date
URL: https://github.com/apache/skywalking/issues/4502#issuecomment-598564618
> I don't know why. Usually deleting index and creating indexes at middle night cause more payload, especially for some some ES cluster. The next 7.0.0 will reduce 50% of the index number, should ease this for you. But basically, that is caused by ES cluster health issue.
ok！thanks！
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4502: SkyWalking not work
after two date
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4502: SkyWalking not work after two date
URL: https://github.com/apache/skywalking/issues/4502#issuecomment-598069913
Seem like someone delete some of data before. And the OAP can't find the metadata.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng closed issue #4502: SkyWalking not work after
two date
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #4502: SkyWalking not work after two date
URL: https://github.com/apache/skywalking/issues/4502
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] jol5 commented on issue #4502: SkyWalking not work
after two date
Posted by GitBox <gi...@apache.org>.
jol5 commented on issue #4502: SkyWalking not work after two date
URL: https://github.com/apache/skywalking/issues/4502#issuecomment-598564618
> I don't know why. Usually deleting index and creating indexes at middle night cause more payload, especially for some some ES cluster. The next 7.0.0 will reduce 50% of the index number, should ease this for you. But basically, that is caused by ES cluster health issue.
ok！
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services