You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by "CVnot (via GitHub)" <gi...@apache.org> on 2023/04/17 09:44:27 UTC
[GitHub] [dubbo] CVnot opened a new issue, #12111: ConcurrentModificationException in ContextFilter.onResponse()
CVnot opened a new issue, #12111:
URL: https://github.com/apache/dubbo/issues/12111
最近线上出现了ConcurrentModificationException异常，dubbo版本是2.7.12
dubbo链路是这样的，上游dubbo服务同步调用我们dubbo服务，我们的dubbo服务作为消费者再异步去调用下游的服务；
类似这个issue：https://github.com/apache/dubbo/issues/6240
我们怀疑是RpcServerContext ThreadLocal中的对象是被多个线程共享了，但是我们的代码没有任何地方使用这个，所以目前没啥思路，提到社区看看大家有没有什么解决思路，或者有类似的问题，感激不尽。
上游dubbo消费端(DubboInvoker)获取到的堆栈如下：
Failed to invoke remote method: query, provider:....,
cause: java.util.concurrent.CompletionException: java.util.ConcurrentModificationException
java.util.concurrent.CompletionException: java.util.ConcurrentModificationException
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:769)
at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778)
at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140)
at org.apache.dubbo.rpc.AsyncRpcResult.whenCompleteWithContext(AsyncRpcResult.java:195)
at org.apache.dubbo.rpc.protocol.FilterNode.invoke(FilterNode.java:81)
at org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:146)
at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:100)
at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:175)
at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:41)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445)
at java.util.HashMap$EntryIterator.next(HashMap.java:1479)
at java.util.HashMap$EntryIterator.next(HashMap.java:1477)
at java.util.HashMap.putMapEntries(HashMap.java:512)
at java.util.HashMap.putAll(HashMap.java:785)
at org.apache.dubbo.rpc.AppResponse.addObjectAttachments(AppResponse.java:164)
at org.apache.dubbo.rpc.filter.ContextFilter.onResponse(ContextFilter.java:141)
at org.apache.dubbo.rpc.protocol.FilterNode.lambda$invoke$0(FilterNode.java:99)
at org.apache.dubbo.rpc.AsyncRpcResult.lambda$whenCompleteWithContext$0(AsyncRpcResult.java:197)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
... 13 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] CVnot commented on issue #12111: ConcurrentModificationException in ContextFilter.onResponse()
Posted by "CVnot (via GitHub)" <gi...@apache.org>.
CVnot commented on issue #12111:
URL: https://github.com/apache/dubbo/issues/12111#issuecomment-1511030088
上面的堆栈可能不太好看，补充个图
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] CVnot closed issue #12111: ConcurrentModificationException in ContextFilter.onResponse()
Posted by "CVnot (via GitHub)" <gi...@apache.org>.
CVnot closed issue #12111: ConcurrentModificationException in ContextFilter.onResponse()
URL: https://github.com/apache/dubbo/issues/12111
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] CVnot commented on issue #12111: ConcurrentModificationException in ContextFilter.onResponse()
Posted by "CVnot (via GitHub)" <gi...@apache.org>.
CVnot commented on issue #12111:
URL: https://github.com/apache/dubbo/issues/12111#issuecomment-1667381152
![Uploading image.png…]()
![Uploading image.png…]()
这两个地方可能会冲突吗
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] CVnot commented on issue #12111: ConcurrentModificationException in ContextFilter.onResponse()
Posted by "CVnot (via GitHub)" <gi...@apache.org>.
CVnot commented on issue #12111:
URL: https://github.com/apache/dubbo/issues/12111#issuecomment-1667672580
其实没太理解#10078中的问题，因为即使一个线程发起了两次调用，也不会共享一个storedServerContext的，所以问题中的并发应该不成立的
<img width="941" alt="image" src="https://github.com/apache/dubbo/assets/40309483/5f251878-1e05-4332-b319-155b05b606db">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] CVnot commented on issue #12111: ConcurrentModificationException in ContextFilter.onResponse()
Posted by "CVnot (via GitHub)" <gi...@apache.org>.
CVnot commented on issue #12111:
URL: https://github.com/apache/dubbo/issues/12111#issuecomment-1667531354
> Fixed in #10160. Please try upgrade to 3.1.9
感谢回复，看了一下 #10078 这个问题，和我们这个问题应该是一类但不是一个问题，我们的问题是RpcContext中的attachments被并发修改了，#10078中是RpcInvocation中的attachments被并发修改了；问题类似，我们这个问题应该是同一个线程发起了两次异步请求，第一个请求回来的时候回调处理线程走filter开始遍历，而相同的线程开始发起第二个异步请求，这个时候走filter的时候在ContextFilter或ConsumerContextClusterInterceptor中做了context.romove，修改了第一个AsyncRpcResult异步结果中的storedServerContext。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #12111: ConcurrentModificationException in ContextFilter.onResponse()
Posted by "AlbumenJ (via GitHub)" <gi...@apache.org>.
AlbumenJ commented on issue #12111:
URL: https://github.com/apache/dubbo/issues/12111#issuecomment-1515896319
Fixed in #10160. Please try upgrade to 3.1.9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] CVnot commented on issue #12111: ConcurrentModificationException in ContextFilter.onResponse()
Posted by "CVnot (via GitHub)" <gi...@apache.org>.
CVnot commented on issue #12111:
URL: https://github.com/apache/dubbo/issues/12111#issuecomment-1667376851
> Fixed in #10160. Please try upgrade to 3.1.9
感谢回复，看了一下 #10078 这个问题，和我们这个问题应该是一类但不是一个问题，我们的问题是RpcContext中的attachments被并发修改了，10078中是RpcInvocation被并发修改了，怀疑是相同的线程同时处理了两个异步回调的请求，ConsumerContextClusterInterceptor中去set了RpcContext中的attachments，另一个在ContextFilter中去获取了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] giraffe-tree commented on issue #12111: ConcurrentModificationException in ContextFilter.onResponse()
Posted by "giraffe-tree (via GitHub)" <gi...@apache.org>.
giraffe-tree commented on issue #12111:
URL: https://github.com/apache/dubbo/issues/12111#issuecomment-1689632568
用的是 2.7.9
```
org.apache.dubbo.rpc.RpcException: Failed to invoke the method inWithResult in the service xxxx.xxxxService. Tried 1 times of the providers [xxxx:17071] (1/68) from the registry xxxx:7400 on the consumer xxxx using the dubbo version 2.7.9. Last error is: null
at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:113)
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:265)
at org.apache.dubbo.rpc.cluster.interceptor.ClusterInterceptor.intercept(ClusterInterceptor.java:47)
at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$InterceptorInvokerNode.invoke(AbstractCluster.java:92)
at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:93)
at org.apache.dubbo.registry.integration.InterfaceCompatibleRegistryProtocol$MigrationInvoker.invoke(InterfaceCompatibleRegistryProtocol.java:132)
at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:83)
at xxx 业务堆栈
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at org.apache.dubbo.rpc.protocol.dubbo.FutureAdapter.lambda$new$0(FutureAdapter.java:49)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at org.apache.dubbo.remoting.exchange.support.DefaultFuture.doReceived(DefaultFuture.java:208)
at org.apache.dubbo.remoting.exchange.support.DefaultFuture.received(DefaultFuture.java:175)
at org.apache.dubbo.remoting.exchange.support.DefaultFuture.received(DefaultFuture.java:163)
at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleResponse(HeaderExchangeHandler.java:60)
at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:181)
at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445)
at java.util.HashMap$EntryIterator.next(HashMap.java:1479)
at java.util.HashMap$EntryIterator.next(HashMap.java:1477)
at java.util.HashMap.putMapEntries(HashMap.java:512)
at java.util.HashMap.putAll(HashMap.java:785)
at org.apache.dubbo.rpc.RpcInvocation.addObjectAttachments(RpcInvocation.java:337)
at org.apache.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:160)
at org.apache.dubbo.rpc.protocol.AsyncToSyncInvoker.invoke(AsyncToSyncInvoker.java:52)
at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:78)
at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83)
at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83)
at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:89)
at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83)
at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:51)
at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83)
at org.apache.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:69)
at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83)
at org.apache.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:82)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org