You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2019/12/31 09:10:30 UTC
[GitHub] [skywalking] to404 opened a new issue #4153: oap: ERROR [] -
CANCELLED: cancelled before receiving half close
to404 opened a new issue #4153: oap: ERROR [] - CANCELLED: cancelled before receiving half close
URL: https://github.com/apache/skywalking/issues/4153
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [ ] Question or discussion
- [ ] Bug
___
### Question
- What do you want to know?
In my oap-server, log is full of this :
```
2019-12-31 16:54:02,055 - org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler - 64 [grpcServerPool-1-thread-17] ERROR [] - CANCELLED: cancelled before receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
at io.grpc.Status.asRuntimeException(Status.java:517) ~[grpc-core-1.15.1.jar:1.15.1]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:272) [grpc-stub-1.15.1.jar:1.15.1]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:293) [grpc-core-1.15.1.jar:1.15.1]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:738) [grpc-core-1.15.1.jar:1.15.1]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.15.1.jar:1.15.1]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.15.1.jar:1.15.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
```
and in my all agent , log is full of this :
```
ERROR 2019-12-31 16:55:59:865 SkywalkingAgent-3-JVMService-consume-0 JVMService : send JVM metrics to Collector fail.
org.apache.skywalking.apm.dependencies.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED
at org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:222)
at org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:203)
at org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:132)
at org.apache.skywalking.apm.network.language.agent.v2.JVMMetricReportServiceGrpc$JVMMetricReportServiceBlockingStub.collect(JVMMetricReportServiceGrpc.java:158)
at org.apache.skywalking.apm.agent.core.jvm.JVMService$Sender.run(JVMService.java:145)
at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:36)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
I have already restart all agent and oap-server several times, but it's not work.
my mysql-server has no performance and other problems.
It's been going on for two days.....
I don't know how and why can causing that.....
May I trouble you to have a look?
( ignore my grammar mistakes...)
___
### Bug
- Which version of SkyWalking, OS and JRE?
6.5.0, jdk1.8, docker in k8s
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] to404 commented on issue #4153: oap: ERROR [] -
CANCELLED: cancelled before receiving half close
Posted by GitBox <gi...@apache.org>.
to404 commented on issue #4153: oap: ERROR [] - CANCELLED: cancelled before receiving half close
URL: https://github.com/apache/skywalking/issues/4153#issuecomment-572854973
Can you tell me how to change the timeout?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4153: oap: ERROR [] -
CANCELLED: cancelled before receiving half close
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4153: oap: ERROR [] - CANCELLED: cancelled before receiving half close
URL: https://github.com/apache/skywalking/issues/4153#issuecomment-582287715
10s is already a long timeout. How many agents are you using?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4153: oap: ERROR [] -
CANCELLED: cancelled before receiving half close
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4153: oap: ERROR [] - CANCELLED: cancelled before receiving half close
URL: https://github.com/apache/skywalking/issues/4153#issuecomment-569924778
These logs could only show the OAP can't support your agent payload. This log is gRPC timeout log. The threshold of timeout is 10s, set by SkyWalking
Do you try open sampling at the agent side or server side? MySQL has very limited performance.
@tristaZero is working on ShardingSphere + MySQL solution for SkyWalking to ease the MySQL users, but still WIP.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] to404 commented on issue #4153: oap: ERROR [] -
CANCELLED: cancelled before receiving half close
Posted by GitBox <gi...@apache.org>.
to404 commented on issue #4153: oap: ERROR [] - CANCELLED: cancelled before receiving half close
URL: https://github.com/apache/skywalking/issues/4153#issuecomment-582288541
About 25 agents.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng closed issue #4153: oap: ERROR [] -
CANCELLED: cancelled before receiving half close
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #4153: oap: ERROR [] - CANCELLED: cancelled before receiving half close
URL: https://github.com/apache/skywalking/issues/4153
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] to404 commented on issue #4153: oap: ERROR [] -
CANCELLED: cancelled before receiving half close
Posted by GitBox <gi...@apache.org>.
to404 commented on issue #4153: oap: ERROR [] - CANCELLED: cancelled before receiving half close
URL: https://github.com/apache/skywalking/issues/4153#issuecomment-582288538
About 25 agents.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4153: oap: ERROR [] -
CANCELLED: cancelled before receiving half close
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4153: oap: ERROR [] - CANCELLED: cancelled before receiving half close
URL: https://github.com/apache/skywalking/issues/4153#issuecomment-582289341
Then there are 1k segments per 3s.
1. Where is your OAP and ES deployment?
1. How many CPUs and memory of OAP and ES?
1. Which disk are you using?
1. Do you use cluster mode?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] to404 commented on issue #4153: oap: ERROR [] -
CANCELLED: cancelled before receiving half close
Posted by GitBox <gi...@apache.org>.
to404 commented on issue #4153: oap: ERROR [] - CANCELLED: cancelled before receiving half close
URL: https://github.com/apache/skywalking/issues/4153#issuecomment-582286438
@wu-sheng Hi, happy new year!
I have use es to storage, and set "sample_n_per_3_secs:50", but it still doesn't work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services