You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2020/12/03 05:47:19 UTC
[GitHub] [skywalking] mubingchen opened a new issue #5940: I can't see the trace data, it disappeared after a period of reporting
mubingchen opened a new issue #5940:
URL: https://github.com/apache/skywalking/issues/5940
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [x] Question or discussion
- [ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Question
- What do you want to know?
I can see the metrics data and trace data after reporting, but there is an interval of less than a day, I can't see the trace reported recently, and the error shows that the es data cannot be obtained. I tried to modify the es of backend configuration, but no effect
I also looked at the previous issues, but it didn’t solve it.
error message: "Exception while fetching data (/data) : Elasticsearch exception [type=index_not_found_exception, reason=no such index [segment-20201128]]"
___
### Bug
- Which version of SkyWalking, OS and JRE?
skywalking 8.2.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mubingchen commented on issue #5940: I can't see the trace data, it disappeared after a period of reporting
Posted by GitBox <gi...@apache.org>.
mubingchen commented on issue #5940:
URL: https://github.com/apache/skywalking/issues/5940#issuecomment-737883049
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:273) [grpc-stub-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2020-12-03_17:41:13 at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-03_17:41:13 Dec 03, 2020 9:41:13 AM io.grpc.internal.SerializingExecutor run
2020-12-03_17:41:13 SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@5624dbf2
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:362)
2020-12-03_17:41:13 at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:85)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861)
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-12-03_17:41:13 at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-03_17:41:13
2020-12-03_17:41:13 2020-12-03 09:41:13,680 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler -282605812 [grpcServerPool-1-thread-2] ERROR [] - CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:273) [grpc-stub-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2020-12-03_17:41:13 at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-03_17:41:13 Dec 03, 2020 9:41:13 AM io.grpc.internal.SerializingExecutor run
2020-12-03_17:41:13 SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@413b2752
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:362)
2020-12-03_17:41:13 at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:85)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861)
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-12-03_17:41:13 at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-03_17:41:13
2020-12-03_17:41:13 2020-12-03 09:41:13,680 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler -282605812 [grpcServerPool-1-thread-5] ERROR [] - CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:273) [grpc-stub-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2020-12-03_17:41:13 at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-03_17:41:13 2020-12-03 09:41:13,681 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler -282605813 [grpcServerPool-1-thread-4] ERROR [] - CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:273) [grpc-stub-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2020-12-03_17:41:13 at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-03_17:41:13 2020-12-03 09:41:13,681 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler -282605813 [grpcServerPool-1-thread-6] ERROR [] - CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:273) [grpc-stub-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2020-12-03_17:41:13 at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-03_17:41:13 Dec 03, 2020 9:41:13 AM io.grpc.internal.SerializingExecutor run
2020-12-03_17:41:13 SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@35030f36
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:362)
2020-12-03_17:41:13 at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:85)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861)
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-12-03_17:41:13 at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-03_17:41:13
2020-12-03_17:41:13 2020-12-03 09:41:13,681 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler -282605813 [grpcServerPool-1-thread-7] ERROR [] - CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:273) [grpc-stub-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2020-12-03_17:41:13 at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-03_17:41:13 Dec 03, 2020 9:41:13 AM io.grpc.internal.SerializingExecutor run
2020-12-03_17:41:13 SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@6b87d8c1
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:362)
2020-12-03_17:41:13 at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:85)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861)
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-12-03_17:41:13 at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-03_17:41:13
2020-12-03_17:41:13 2020-12-03 09:41:13,682 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler -282605814 [grpcServerPool-1-thread-3] ERROR [] - CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:273) [grpc-stub-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2020-12-03_17:41:13 at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-03_17:41:13 Dec 03, 2020 9:41:13 AM io.grpc.internal.SerializingExecutor run
2020-12-03_17:41:13 SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@6acd37a1
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:362)
2020-12-03_17:41:13 at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:85)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861)
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-12-03_17:41:13 at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-03_17:41:13
2020-12-03_17:41:13 2020-12-03 09:41:13,682 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler -282605814 [grpcServerPool-1-thread-6] ERROR [] - CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:273) [grpc-stub-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2020-12-03_17:41:13 at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-03_17:41:13 Dec 03, 2020 9:41:13 AM io.grpc.internal.SerializingExecutor run
2020-12-03_17:41:13 SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@1aacf04
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:362)
2020-12-03_17:41:13 at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:85)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861)
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-12-03_17:41:13 at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-03_17:41:13
2020-12-03_17:41:13 2020-12-03 09:41:13,683 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler -282605815 [grpcServerPool-1-thread-4] ERROR [] - CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:273) [grpc-stub-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2020-12-03_17:41:13 at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-03_17:41:13 Dec 03, 2020 9:41:13 AM io.grpc.internal.SerializingExecutor run
2020-12-03_17:41:13 SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@172ec80f
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:362)
2020-12-03_17:41:13 at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:85)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861)
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-12-03_17:41:13 at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-03_17:41:13
2020-12-03_17:41:13 2020-12-03 09:41:13,683 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler -282605815 [grpcServerPool-1-thread-5] ERROR [] - CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:273) [grpc-stub-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.32.1.jar:1.32.1]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2020-12-03_17:41:13 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2020-12-03_17:41:13 at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-03_17:41:13 Dec 03, 2020 9:41:13 AM io.grpc.internal.SerializingExecutor run
2020-12-03_17:41:13 SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@4f9524ca
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:362)
2020-12-03_17:41:13 at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:85)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861)
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-12-03_17:41:13 at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-03_17:41:13
2020-12-03_17:41:13 Dec 03, 2020 9:41:13 AM io.grpc.internal.SerializingExecutor run
2020-12-03_17:41:13 SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@2107ce78
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:362)
2020-12-03_17:41:13 at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:85)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861)
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-12-03_17:41:13 at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-03_17:41:13
2020-12-03_17:41:13 Dec 03, 2020 9:41:13 AM io.grpc.internal.SerializingExecutor run
2020-12-03_17:41:13 SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@531fbf54
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:362)
2020-12-03_17:41:13 at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:85)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861)
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-12-03_17:41:13 at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-03_17:41:13
2020-12-03_17:41:13 Dec 03, 2020 9:41:13 AM io.grpc.internal.SerializingExecutor run
2020-12-03_17:41:13 SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@4fd7fea6
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:362)
2020-12-03_17:41:13 at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:85)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861)
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-12-03_17:41:13 at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-03_17:41:13
2020-12-03_17:41:13 Dec 03, 2020 9:41:13 AM io.grpc.internal.SerializingExecutor run
2020-12-03_17:41:13 SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@2cb43393
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:362)
2020-12-03_17:41:13 at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:85)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861)
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-12-03_17:41:13 at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-03_17:41:13
2020-12-03_17:41:13 Dec 03, 2020 9:41:13 AM io.grpc.internal.SerializingExecutor run
2020-12-03_17:41:13 SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@24d9e100
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:362)
2020-12-03_17:41:13 at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:85)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861)
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-12-03_17:41:13 at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-03_17:41:13
2020-12-03_17:41:13 Dec 03, 2020 9:41:13 AM io.grpc.internal.SerializingExecutor run
2020-12-03_17:41:13 SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@6ec3d861
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:362)
2020-12-03_17:41:13 at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:85)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861)
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-12-03_17:41:13 at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-03_17:41:13
2020-12-03_17:41:13 Dec 03, 2020 9:41:13 AM io.grpc.internal.SerializingExecutor run
2020-12-03_17:41:13 SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@2daf747
2020-12-03_17:41:13 io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
2020-12-03_17:41:13 at io.grpc.Status.asRuntimeException(Status.java:524)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:362)
2020-12-03_17:41:13 at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:85)
2020-12-03_17:41:13 at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
2020-12-03_17:41:13 at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
2020-12-03_17:41:13 at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:861)
2020-12-03_17:41:13 at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-12-03_17:41:13 at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-12-03_17:41:13 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-12-03_17:41:13 at java.base/java.lang.Thread.run(Thread.java:834)
2020-12-03_17:41:13
2020-12-03_17:41:16 2020-12-03 09:41:16,668 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer -282608800 [pool-10-thread-1] INFO [] - Beginning to remove expired metrics from the storage.
2020-12-03_17:41:18 2020-12-03 09:41:18,759 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -282610891 [I/O dispatcher 2] INFO [] - Bulk execution id [68] completed in 33 milliseconds, size: 153
2020-12-03_17:41:26 2020-12-03 09:41:26,887 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -282619019 [pool-9-thread-1] INFO [] - Synchronous bulk took time: 9972 millis, size: 8080
2020-12-03_17:41:28 2020-12-03 09:41:28,902 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -282621034 [I/O dispatcher 1] INFO [] - Bulk execution id [69] completed in 175 milliseconds, size: 2
2020-12-03_17:41:39 2020-12-03 09:41:39,772 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -282631904 [pool-9-thread-1] INFO [] - Synchronous bulk took time: 9669 millis, size: 5252
I tried again and found that the trace data is not stored in es.
The above is the error reported by the server, What is the reason for the wrong trace data？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mubingchen commented on issue #5940: I can't see the trace data, it disappeared after a period of reporting
Posted by GitBox <gi...@apache.org>.
mubingchen commented on issue #5940:
URL: https://github.com/apache/skywalking/issues/5940#issuecomment-738515373
Thank you for your reply，my problem was solved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #5940: I can't see the trace data, it disappeared after a period of reporting
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #5940:
URL: https://github.com/apache/skywalking/issues/5940#issuecomment-737682009
This is purely your ElasticSearch server performance issue. It can't support the traces you have produced
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] YczYanchengzhe commented on issue #5940: I can't see the trace data, it disappeared after a period of reporting
Posted by GitBox <gi...@apache.org>.
YczYanchengzhe commented on issue #5940:
URL: https://github.com/apache/skywalking/issues/5940#issuecomment-822218535
I also encountered the same problem. How did you solve it at the time? Will this grpc error affect data processing?
```log
SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@3adb481f
io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
at io.grpc.Status.asRuntimeException(Status.java:524)
at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:368)
at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onError(TraceSegmentReportServiceHandler.java:84)
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:272)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #5940: I can't see the trace data, it disappeared after a period of reporting
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #5940:
URL: https://github.com/apache/skywalking/issues/5940
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #5940: I can't see the trace data, it disappeared after a period of reporting
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #5940:
URL: https://github.com/apache/skywalking/issues/5940#issuecomment-737884901
Even no new data, so? You had, those data are still there.
Then, this log just mean, connection breaks, nothing special.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org