You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@servicecomb.apache.org by GitBox <gi...@apache.org> on 2020/03/22 09:21:07 UTC
[GitHub] [servicecomb-pack] lostpillow opened a new issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
lostpillow opened a new issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
URL: https://github.com/apache/servicecomb-pack/issues/639
我使用的是0.5.0，使用的是saga，但是没有打开状态机模式。当omega端业务抛出异常后，未执行回滚。
omega端日志如下：
[2020-03-22 16:36:47.876] [ERROR] [ReconnectStreamObserver:onError:43] [grpc-default-executor-2] [-[Failed to process grpc coordinate command.]-]
io.grpc.StatusRuntimeException: CANCELLED: Failed to read message.
at io.grpc.Status.asRuntimeException(Status.java:526) ~[grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420) [grpc-stub-1.14.0.jar!/:1.14.0]
at org.apache.skywalking.apm.plugin.grpc.v1.StreamCallClientInterceptor$CallListener.onClose(StreamCallClientInterceptor.java:131) [na:na]
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:531) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.14.0.jar!/:1.14.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
Caused by: java.lang.NullPointerException: null
at org.apache.skywalking.apm.agent.core.context.ContextManager.stopSpan(ContextManager.java:176) ~[skywalking-agent.jar:6.3.0]
at org.apache.skywalking.apm.plugin.grpc.v1.StreamCallClientInterceptor$CallListener.onMessage(StreamCallClientInterceptor.java:117) [na:na]
at io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33) [grpc-core-1.14.0.jar!/:1.14.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:519) [grpc-core-1.14.0.jar!/:1.14.0]
... 5 common frames omitted
[2020-03-22 16:36:47.899] [ERROR] [PushBackReconnectRunnable:run:58] [pool-15-thread-1] [-[Failed to reconnect to alpha at 172.16.2.32:8080]-]
io.grpc.StatusRuntimeException: UNKNOWN
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:222) ~[grpc-stub-1.14.0.jar!/:1.14.0]
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:203) ~[grpc-stub-1.14.0.jar!/:1.14.0]
at io.grpc.stub.ClientCalls.blockingUnaryCall$original$aTgkcz7Y(ClientCalls.java:132) ~[grpc-stub-1.14.0.jar!/:1.14.0]
at io.grpc.stub.ClientCalls.blockingUnaryCall$original$aTgkcz7Y$accessor$H2KLSmwB(ClientCalls.java) ~[grpc-stub-1.14.0.jar!/:1.14.0]
at io.grpc.stub.ClientCalls$auxiliary$pbnPuffH.call(Unknown Source) ~[grpc-stub-1.14.0.jar!/:1.14.0]
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.StaticMethodsInter.intercept(StaticMethodsInter.java:86) ~[skywalking-agent.jar:6.3.0]
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java) ~[grpc-stub-1.14.0.jar!/:1.14.0]
at org.apache.servicecomb.pack.contract.grpc.TxEventServiceGrpc$TxEventServiceBlockingStub.onDisconnected(TxEventServiceGrpc.java:280) ~[pack-contract-grpc-0.5.0.jar!/:0.5.0]
at org.apache.servicecomb.pack.omega.connector.grpc.saga.GrpcSagaClientMessageSender.onDisconnected(GrpcSagaClientMessageSender.java:75) ~[omega-connector-grpc-0.5.0.jar!/:0.5.0]
at org.apache.servicecomb.pack.omega.connector.grpc.core.PushBackReconnectRunnable.run(PushBackReconnectRunnable.java:52) ~[omega-connector-grpc-0.5.0.jar!/:0.5.0]
at org.apache.servicecomb.pack.omega.connector.grpc.core.PendingTaskRunner$1.run(PendingTaskRunner.java:44) [omega-connector-grpc-0.5.0.jar!/:0.5.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_152]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_152]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_152]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
alpha端日志如下：
2020-03-22 16:36:47.744 INFO 2553b2263047 --- [pool-2-thread-1] o.a.s.p.a.c.EventScanner : Compensating transaction with globalTxId 7d5162f6-2c7e-4a8e-9371-aa0db090eeee and localTxId b3cdce33-f396-42f5-bf21-e6c43379ca79
2020-03-22 16:36:47.893 ERROR 2553b2263047 --- [ault-executor-3] i.g.i.SerializingExecutor : Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@222999fc
io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
at io.grpc.Status.asRuntimeException(Status.java:517)
at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:356)
at org.apache.servicecomb.pack.alpha.server.GrpcOmegaCallback.disconnect(GrpcOmegaCallback.java:53)
at org.apache.servicecomb.pack.alpha.server.GrpcTxEventEndpointImpl.onDisconnected(GrpcTxEventEndpointImpl.java:69)
at org.apache.servicecomb.pack.contract.grpc.TxEventServiceGrpc$MethodHandlers.invoke(TxEventServiceGrpc.java:350)
at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
请问是什么问题呢
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [servicecomb-pack] lostpillow commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
Posted by GitBox <gi...@apache.org>.
lostpillow commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
URL: https://github.com/apache/servicecomb-pack/issues/639#issuecomment-604270432
> @lostpillow 有什么进展吗？
插件去掉以后好了
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [servicecomb-pack] WillemJiang commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
Posted by GitBox <gi...@apache.org>.
WillemJiang commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
URL: https://github.com/apache/servicecomb-pack/issues/639#issuecomment-604282252
这就好， 如果问题解决了可以关闭这个issue。
如果需要支持skywalking GRPC 和servicecomb pack的话， 可能需要给SkyWalking提一个issue，看看如何修改。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [servicecomb-pack] lostpillow commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
Posted by GitBox <gi...@apache.org>.
lostpillow commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
URL: https://github.com/apache/servicecomb-pack/issues/639#issuecomment-602214954
> Omega端的异常是什么？能详细介绍一下吗？
> 通过StackTrace看样子是GRPC连接断开了， 不知道业务代码有没有抛出什么异常？
omega端的框架的异常堆栈我在问题描述里已经贴出来了，至于业务异常就是为了测试回滚，直接throw new Exception()
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [servicecomb-pack] lostpillow commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
Posted by GitBox <gi...@apache.org>.
lostpillow commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
URL: https://github.com/apache/servicecomb-pack/issues/639#issuecomment-602218879
> Caused by: java.lang.NullPointerException: null
> at org.apache.skywalking.apm.agent.core.context.ContextManager.stopSpan(ContextManager.java:176) ~[skywalking-agent.jar:6.3.0]
> at org.apache.skywalking.apm.plugin.grpc.v1.StreamCallClientInterceptor$CallListener.onMessage(StreamCallClientInterceptor.java:117) [na:na]
>
> 看看能不能先把skywalking禁止一下。
现在暂时禁不了，明日测试了再答复你
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [servicecomb-pack] WillemJiang commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
Posted by GitBox <gi...@apache.org>.
WillemJiang commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
URL: https://github.com/apache/servicecomb-pack/issues/639#issuecomment-602217664
Caused by: java.lang.NullPointerException: null
at org.apache.skywalking.apm.agent.core.context.ContextManager.stopSpan(ContextManager.java:176) ~[skywalking-agent.jar:6.3.0]
at org.apache.skywalking.apm.plugin.grpc.v1.StreamCallClientInterceptor$CallListener.onMessage(StreamCallClientInterceptor.java:117) [na:na]
看看能不能先把skywalking禁止一下。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [servicecomb-pack] lostpillow commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
Posted by GitBox <gi...@apache.org>.
lostpillow commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
URL: https://github.com/apache/servicecomb-pack/issues/639#issuecomment-603146563
> Caused by: java.lang.NullPointerException: null
> at org.apache.skywalking.apm.agent.core.context.ContextManager.stopSpan(ContextManager.java:176) ~[skywalking-agent.jar:6.3.0]
> at org.apache.skywalking.apm.plugin.grpc.v1.StreamCallClientInterceptor$CallListener.onMessage(StreamCallClientInterceptor.java:117) [na:na]
>
> 看看能不能先把skywalking禁止一下。
禁了就好了。这里是因为omega通过grpc跟alpha通信的时候走到skywalking的grpc plugin的实现上了吗？因为skywalking在生产上是要使用的，那么有没有办法在和alpha通信的时候grpc指定实现？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [servicecomb-pack] WillemJiang commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
Posted by GitBox <gi...@apache.org>.
WillemJiang commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
URL: https://github.com/apache/servicecomb-pack/issues/639#issuecomment-603795073
@lostpillow 有什么进展吗？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [servicecomb-pack] lostpillow commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
Posted by GitBox <gi...@apache.org>.
lostpillow commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
URL: https://github.com/apache/servicecomb-pack/issues/639#issuecomment-602217568
> Omega端的异常是什么？能详细介绍一下吗？
> 通过StackTrace看样子是GRPC连接断开了， 不知道业务代码有没有抛出什么异常？
上面贴出的omega端的异常，我刚发现grpc的异常堆栈走的是skywalking.apm.plugin.grpc 会跟skywalking有关系吗？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [servicecomb-pack] WillemJiang commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
Posted by GitBox <gi...@apache.org>.
WillemJiang commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
URL: https://github.com/apache/servicecomb-pack/issues/639#issuecomment-602201689
Omega端的异常是什么？能详细介绍一下吗？
通过StackTrace看样子是GRPC连接断开了， 不知道业务代码有没有抛出什么异常？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [servicecomb-pack] WillemJiang closed issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
Posted by GitBox <gi...@apache.org>.
WillemJiang closed issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
URL: https://github.com/apache/servicecomb-pack/issues/639
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [servicecomb-pack] WillemJiang commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
Posted by GitBox <gi...@apache.org>.
WillemJiang commented on issue #639: omega端报Failed to process grpc coordinate command，alpha端报io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
URL: https://github.com/apache/servicecomb-pack/issues/639#issuecomment-603196900
Grpc的实现是不能替换的。
生产系统使用到了grpc吗？如果没有看看能不能把SkyWalking的grpc的插件去掉？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services