You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by GitBox <gi...@apache.org> on 2021/07/15 06:01:21 UTC
[GitHub] [dubbo] zewade opened a new issue #8297: Dubbo 2.6.3 can send request but no response in the product environment sometimes.
zewade opened a new issue #8297:
URL: https://github.com/apache/dubbo/issues/8297
### Environment
* Dubbo version: 2.6.3
* Operating System version: Linux
* Java version: openjdk8u201
在我们的线上环境偶尔会发生A服务调用其他服务Dubbo接口超时的问题，从调用链系统看其他服务已经处理完成，但是调用方A服务接受不到响应，最终报超时异常，需要重启A服务才能恢复。在些期间A可以正常接收请求，进行数据库操作并返回响应，也没有特别大的延迟，资源占用也正常。求助排查的方法。
第一种超时发生在DefaultFuture.java:136
`2021-07-12 08:35:44,124 ERROR [xxxxyy-patient-provider] [DubboServerHandler-10.130.3.104:20882-thread-6] [c.f.plugin.transmission.EmrCodeUtils:115] - trace[b34b591473535b0b:d333e4b17e676e3b:b34b591473535b0b:1] 项目名：[classes!]
com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method checkHosDeptPepList in the service cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService. Tried 3 times of the providers [10.131.4.190:20880, 10.128.3.66:20880, 10.129.6.128:20880] (3/16) from the registry zookeeper:2181 on the consumer 10.130.3.104 using the dubbo version 2.6.3. Last error is: com.alibaba.dubbo.rpc.RpcException: Invoke remote method timeout. method: checkHosDeptPepList, provider: dubbo://10.131.4.190:20880/cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService?anyhost=true&application=xxxxyy-patient-provider&check=false&default.check=false&default.reference.filter=dubboServiceFilter&default.timeout=12000&dispatcher=all&dubbo=2.0.2&generic=false&interface=cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService&methods=getStartDate,listEveryMoe,BS50002ESB,checkHosDeptPepList,listGroupMoe,stopMoeForDoc,lineDrugsDetail,similarDietary,stopMoeForDocV1,undo,selectListMoePrintLog,getLastFee
Time2,stopMoeForDocV2,havingYryMoe,countDosageForOrdCode,deprecatedMoe,docListMoePrint,endMoeV1,getAdvices,getCateMethod,listMoeSpecialTreatment,listMoe,listEveryMoeForMem,listEveryMoeV2,test,listMoeV4,listOrderCycle,listMoePrint,inquireForMedical,similarSkinTest,listMoeByRescueBindNo,getEnterDate,countDosageForOrdCodeCheck,listEveryMoeForMemPage,listMoeByEnterDate,similarDietaryEcg,getLastFeeTime,aboutEffectiveNursing,searchPatSrescueOrders,endMoe,effectiveInspection&payload=83886080&pid=1&qos.enable=false®ister.ip=10.130.3.104&remote.timestamp=1624626328873&revision=0.0.5-SNAPSHOT&side=consumer×tamp=1625275654554&version=2.0.0, cause: Waiting server-side response timeout. start time: 2021-07-12 08:35:32.122, end time: 2021-07-12 08:35:44.123, client elapsed: 0 ms, server elapsed: 12001 ms, timeout: 12000 ms, request: Request [id=3706895, version=2.0.2, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=checkHosDeptPepList, parameterTypes=[class co.faao.
plugin.transmission.request.MessageHeader], arguments=[co.faao.plugin.transmission.request.MessageHeader@72fb7075], attachments={input=2570, path=cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService, application=node-dubbo, uber-trace-id=b34b591473535b0b:d333e4b17e676e3b:b34b591473535b0b:1, interface=cc.xxxxx.xxxxyy.patient.api.service.IInpNursePatientService, version=2.0.0, timeout=12000}]], channel: /10.130.3.104:38668 -> /10.131.4.190:20880
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:102)
at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:234)
at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:75)
at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
at com.alibaba.dubbo.common.bytecode.proxy17.checkHosDeptPepList(proxy17.java)
at cc.xxxxx.xxxxyy.patient.service.impl.busines.select.InpOtherSelectServiceImpl.getMoesByInpVisistIds(InpOtherSelectServiceImpl.java:287)
at cc.xxxxx.xxxxyy.patient.service.impl.busines.InpNursePatBusServiceImpl.getMoesByInpVisistIds(InpNursePatBusServiceImpl.java:1929)
at cc.xxxxx.xxxxyy.patient.service.impl.busines.InpNursePatBusServiceImpl.queryListMoeNurCheck(InpNursePatBusServiceImpl.java:1623)
at cc.xxxxx.xxxxyy.patient.service.impl.busines.InpNursePatBusServiceImpl.getPatNumsBySelect(InpNursePatBusServiceImpl.java:2337)
at cc.xxxxx.xxxxyy.patient.service.impl.busines.InpNursePatBusServiceImpl.homePageStatistics(InpNursePatBusServiceImpl.java:2128)
at cc.xxxxx.xxxxyy.patient.service.impl.dubbo.InpNursePatientServiceImpl.homePageStatistics(InpNursePatientServiceImpl.java:110)
at com.alibaba.dubbo.common.bytecode.Wrapper58.invokeMethod(Wrapper58.java)
at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:47)
at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:76)
at com.alibaba.dubbo.config.invoker.DelegateProviderMetaDataInvoker.invoke(DelegateProviderMetaDataInvoker.java:52)
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
at io.seata.integration.dubbo.alibaba.TransactionPropagationFilter.invoke(TransactionPropagationFilter.java:61)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:62)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at co.faao.plugin.dubbofilter.DubboServiceFilter.invoke(DubboServiceFilter.java:27)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.processProviderTrace(OpenTracingFilter.java:97)
at co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.invoke(OpenTracingFilter.java:63)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at co.faao.plugin.dubbofilter.DubboServiceFilter.invoke(DubboServiceFilter.java:27)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at co.faao.plugin.starter.dubbo.filter.UserFilter.invoke(UserFilter.java:39)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:73)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:138)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:104)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:96)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:172)
at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.alibaba.dubbo.rpc.RpcException: Invoke remote method timeout. method: checkHosDeptPepList, provider: dubbo://10.131.4.190:20880/cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService?anyhost=true&application=xxxxyy-patient-provider&check=false&default.check=false&default.reference.filter=dubboServiceFilter&default.timeout=12000&dispatcher=all&dubbo=2.0.2&generic=false&interface=cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService&methods=getStartDate,listEveryMoe,BS50002ESB,checkHosDeptPepList,listGroupMoe,stopMoeForDoc,lineDrugsDetail,similarDietary,stopMoeForDocV1,undo,selectListMoePrintLog,getLastFeeTime2,stopMoeForDocV2,havingYryMoe,countDosageForOrdCode,deprecatedMoe,docListMoePrint,endMoeV1,getAdvices,getCateMethod,listMoeSpecialTreatment,listMoe,listEveryMoeForMem,listEveryMoeV2,test,listMoeV4,listOrderCycle,listMoePrint,inquireForMedical,similarSkinTest,listMoeByRescueBindNo,getEnterDate,countDosageForOrdCodeCheck,listEveryMoeForMemPage,listMoe
ByEnterDate,similarDietaryEcg,getLastFeeTime,aboutEffectiveNursing,searchPatSrescueOrders,endMoe,effectiveInspection&payload=83886080&pid=1&qos.enable=false®ister.ip=10.130.3.104&remote.timestamp=1624626328873&revision=0.0.5-SNAPSHOT&side=consumer×tamp=1625275654554&version=2.0.0, cause: Waiting server-side response timeout. start time: 2021-07-12 08:35:32.122, end time: 2021-07-12 08:35:44.123, client elapsed: 0 ms, server elapsed: 12001 ms, timeout: 12000 ms, request: Request [id=3706895, version=2.0.2, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=checkHosDeptPepList, parameterTypes=[class co.faao.plugin.transmission.request.MessageHeader], arguments=[co.faao.plugin.transmission.request.MessageHeader@72fb7075], attachments={input=2570, path=cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService, application=node-dubbo, uber-trace-id=b34b591473535b0b:d333e4b17e676e3b:b34b591473535b0b:1, interface=cc.xxxxx.xxxxyy.patient.api.service.IInpNurseP
atientService, version=2.0.0, timeout=12000}]], channel: /10.130.3.104:38668 -> /10.131.4.190:20880
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:98)
at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:154)
at co.faao.plugin.dubbofilter.DubboServiceFilter.invoke(DubboServiceFilter.java:27)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at io.seata.integration.dubbo.alibaba.TransactionPropagationFilter.invoke(TransactionPropagationFilter.java:61)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.processConsumerTrace(OpenTracingFilter.java:196)
at co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.invoke(OpenTracingFilter.java:65)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:54)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:49)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:77)
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:78)
... 50 common frames omitted
Caused by: com.alibaba.dubbo.remoting.TimeoutException: Waiting server-side response timeout. start time: 2021-07-12 08:35:32.122, end time: 2021-07-12 08:35:44.123, client elapsed: 0 ms, server elapsed: 12001 ms, timeout: 12000 ms, request: Request [id=3706895, version=2.0.2, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=checkHosDeptPepList, parameterTypes=[class co.faao.plugin.transmission.request.MessageHeader], arguments=[co.faao.plugin.transmission.request.MessageHeader@72fb7075], attachments={input=2570, path=cc.xxxxx.xxxxyy.moe.api.service.dubbo.IPrescribeDubboService, application=node-dubbo, uber-trace-id=b34b591473535b0b:d333e4b17e676e3b:b34b591473535b0b:1, interface=cc.xxxxx.xxxxyy.patient.api.service.IInpNursePatientService, version=2.0.0, timeout=12000}]], channel: /10.130.3.104:38668 -> /10.131.4.190:20880
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:136)
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:112)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:95)
... 67 common frames omitted`
第二种超时发生在DefaultFuture.java:220
`2021-07-12 08:35:38,870 ERROR [xxxxyy-patient-provider] [DubboServerHandler-10.130.3.104:20882-thread-186] [c.f.plugin.transmission.EmrCodeUtils:115] - trace[37f00de19d2909bcdcd1c6ab8e6ce44b:d2e3b49ec4b912a3:377906989b7f7603:1] 项目名：[classes!]
com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method searchAllDictByCodeList in the service cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService. Tried 3 times of the providers [10.131.4.168:20888, 10.129.4.18:20888, 10.129.6.108:20888] (3/12) from the registry zookeeper:2181 on the consumer 10.130.3.104 using the dubbo version 2.6.3. Last error is: com.alibaba.dubbo.rpc.RpcException: Invoke remote method timeout. method: searchAllDictByCodeList, provider: dubbo://10.129.4.18:20888/cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService?anyhost=true&application=xxxxyy-patient-provider&check=false&default.check=false&default.reference.filter=dubboServiceFilter&default.timeout=12000&dubbo=2.0.2&generic=false&interface=cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService&methods=searchAllDictByCodeList,searchStaffCodeByHrpCode,searchWarehouseDrugStockByDrugList,searchLikeRangeDictByConditions,searchMainDictByCompareCode,queryFeeItemByFeeType,searchAllergeSuorceByConditions,searchMainD
ictByCodeList,searchMainDictUnionByMoe,searchMainDictUnionByMoeV2,searchLikeMainDictByCodesByPages,queryBaseSubDict,queryBaseSubDictByCodes,searchMainDictUnionByAllMoe,searchMainDictByHerbal&pid=1&qos.enable=false®ister.ip=10.130.3.104&remote.timestamp=1624008835759&revision=0.0.4-SNAPSHOT&side=consumer×tamp=1625275650525&version=2.0.0, cause: Waiting server-side response timeout by scan timer. start time: 2021-07-12 08:35:26.840, end time: 2021-07-12 08:35:38.866, client elapsed: 0 ms, server elapsed: 12026 ms, timeout: 12000 ms, request: Request [id=3706836, version=2.0.2, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=searchAllDictByCodeList, parameterTypes=[class co.faao.plugin.transmission.request.MessageHeader], arguments=[co.faao.plugin.transmission.request.MessageHeader@454f904f], attachments={input=1019, path=cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService, application=node-dubbo, uber-trace-id=37f00de19d2909bcdcd1c6ab8e6ce44b:d2e3b49ec4b91
2a3:377906989b7f7603:1, interface=cc.xxxxx.xxxxyy.moe.api.service.dubbo.ILingYiDubboService, version=2.0.0, timeout=12000}]], channel: /10.130.3.104:42286 -> /10.129.4.18:20888
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:102)
at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:234)
at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:75)
at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
at com.alibaba.dubbo.common.bytecode.proxy1.searchAllDictByCodeList(proxy1.java)
at cc.xxxxx.xxxxyy.patient.common.QueryDictNameUtil.getBaseDictCode(QueryDictNameUtil.java:82)
at cc.xxxxx.xxxxyy.patient.service.impl.busines.InpHostialRecServiceImpl.queryPatInfor(InpHostialRecServiceImpl.java:922)
at cc.xxxxx.xxxxyy.patient.service.impl.dubbo.InputPatientServiceImpl.queryPatInfor(InputPatientServiceImpl.java:134)
at com.alibaba.dubbo.common.bytecode.Wrapper21.invokeMethod(Wrapper21.java)
at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:47)
at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:76)
at com.alibaba.dubbo.config.invoker.DelegateProviderMetaDataInvoker.invoke(DelegateProviderMetaDataInvoker.java:52)
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
at io.seata.integration.dubbo.alibaba.TransactionPropagationFilter.invoke(TransactionPropagationFilter.java:61)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:62)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at co.faao.plugin.dubbofilter.DubboServiceFilter.invoke(DubboServiceFilter.java:27)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.processProviderTrace(OpenTracingFilter.java:97)
at co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.invoke(OpenTracingFilter.java:63)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at co.faao.plugin.dubbofilter.DubboServiceFilter.invoke(DubboServiceFilter.java:27)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at co.faao.plugin.starter.dubbo.filter.UserFilter.invoke(UserFilter.java:39)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:73)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:138)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:104)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:96)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:172)
at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.alibaba.dubbo.rpc.RpcException: Invoke remote method timeout. method: searchAllDictByCodeList, provider: dubbo://10.129.4.18:20888/cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService?anyhost=true&application=xxxxyy-patient-provider&check=false&default.check=false&default.reference.filter=dubboServiceFilter&default.timeout=12000&dubbo=2.0.2&generic=false&interface=cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService&methods=searchAllDictByCodeList,searchStaffCodeByHrpCode,searchWarehouseDrugStockByDrugList,searchLikeRangeDictByConditions,searchMainDictByCompareCode,queryFeeItemByFeeType,searchAllergeSuorceByConditions,searchMainDictByCodeList,searchMainDictUnionByMoe,searchMainDictUnionByMoeV2,searchLikeMainDictByCodesByPages,queryBaseSubDict,queryBaseSubDictByCodes,searchMainDictUnionByAllMoe,searchMainDictByHerbal&pid=1&qos.enable=false®ister.ip=10.130.3.104&remote.timestamp=1624008835759&revision=0.0.4-SNAPSHOT&side=consumer×tamp=1625275650525&version=2.0.0, ca
use: Waiting server-side response timeout by scan timer. start time: 2021-07-12 08:35:26.840, end time: 2021-07-12 08:35:38.866, client elapsed: 0 ms, server elapsed: 12026 ms, timeout: 12000 ms, request: Request [id=3706836, version=2.0.2, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=searchAllDictByCodeList, parameterTypes=[class co.faao.plugin.transmission.request.MessageHeader], arguments=[co.faao.plugin.transmission.request.MessageHeader@454f904f], attachments={input=1019, path=cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService, application=node-dubbo, uber-trace-id=37f00de19d2909bcdcd1c6ab8e6ce44b:d2e3b49ec4b912a3:377906989b7f7603:1, interface=cc.xxxxx.xxxxyy.moe.api.service.dubbo.ILingYiDubboService, version=2.0.0, timeout=12000}]], channel: /10.130.3.104:42286 -> /10.129.4.18:20888
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:98)
at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:154)
at co.faao.plugin.dubbofilter.DubboServiceFilter.invoke(DubboServiceFilter.java:27)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at io.seata.integration.dubbo.alibaba.TransactionPropagationFilter.invoke(TransactionPropagationFilter.java:61)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.processConsumerTrace(OpenTracingFilter.java:196)
at co.faao.plugin.starter.jaeger.opentracing.filter.OpenTracingFilter.invoke(OpenTracingFilter.java:65)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:54)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:49)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:77)
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:78)
... 47 common frames omitted
Caused by: com.alibaba.dubbo.remoting.TimeoutException: Waiting server-side response timeout by scan timer. start time: 2021-07-12 08:35:26.840, end time: 2021-07-12 08:35:38.866, client elapsed: 0 ms, server elapsed: 12026 ms, timeout: 12000 ms, request: Request [id=3706836, version=2.0.2, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=searchAllDictByCodeList, parameterTypes=[class co.faao.plugin.transmission.request.MessageHeader], arguments=[co.faao.plugin.transmission.request.MessageHeader@454f904f], attachments={input=1019, path=cc.xxxxx.xxxxyy.dict.api.service.IBaseDictService, application=node-dubbo, uber-trace-id=37f00de19d2909bcdcd1c6ab8e6ce44b:d2e3b49ec4b912a3:377906989b7f7603:1, interface=cc.xxxxx.xxxxyy.moe.api.service.dubbo.ILingYiDubboService, version=2.0.0, timeout=12000}]], channel: /10.130.3.104:42286 -> /10.129.4.18:20888
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.returnFromResponse(DefaultFuture.java:220)
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:139)
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:112)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:95)
... 64 common frames omitted`
在此期间zk也有断开重连发生
`2021-07-12 08:34:22,405 WARN [xxxxyy-patient-provider] [main-SendThread(zookeeper:2181)] [org.apache.zookeeper.ClientCnxn:1112] - trace[] Client session timed out, have not heard from server in 29209ms for sessionid 0x20b1457b40e018a
2021-07-12 08:34:22,407 INFO [xxxxyy-patient-provider] [main-SendThread(zookeeper:2181)] [org.apache.zookeeper.ClientCnxn:1160] - trace[] Client session timed out, have not heard from server in 29209ms for sessionid 0x20b1457b40e018a, closing socket connection and attempting reconnect
2021-07-12 08:34:39,598 INFO [xxxxyy-patient-provider] [main-EventThread] [o.a.c.f.state.ConnectionStateManager:228] - trace[] State change: SUSPENDED
2021-07-12 08:34:45,301 ERROR [xxxxyy-patient-provider] [Curator-Framework-0] [org.apache.curator.ConnectionState:228] - trace[] Connection timed out for connection string (zookeeper:2181) and timeout (5000) / elapsed (5678)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-07-12 08:34:46,242 INFO [xxxxyy-patient-provider] [main-EventThread] [o.a.c.f.state.ConnectionStateManager:228] - trace[] State change: RECONNECTED`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] CrazyHZM closed issue #8297: Dubbo 2.6.3 can send request but no response in the product environment sometimes.
Posted by GitBox <gi...@apache.org>.
CrazyHZM closed issue #8297:
URL: https://github.com/apache/dubbo/issues/8297
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] hengyoush commented on issue #8297: Dubbo 2.6.3 can send request but no response in the product environment sometimes.
Posted by GitBox <gi...@apache.org>.
hengyoush commented on issue #8297:
URL: https://github.com/apache/dubbo/issues/8297#issuecomment-886344583
是不是你们服务有full gc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] CrazyHZM commented on issue #8297: Dubbo 2.6.3 can send request but no response in the product environment sometimes.
Posted by GitBox <gi...@apache.org>.
CrazyHZM commented on issue #8297:
URL: https://github.com/apache/dubbo/issues/8297#issuecomment-925606516
I haven’t received a reply for a long time, please close the issue temporarily, if there are other issues, you can open the issue at any time.
@zewade
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org