You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by "fxli4 (via GitHub)" <gi...@apache.org> on 2025/10/21 01:53:00 UTC
[I] [Bug] BatchExecutorQueue performance risk [dubbo]
fxli4 opened a new issue, #15732:
URL: https://github.com/apache/dubbo/issues/15732
### Pre-check
- [x] I am sure that all the content I provide is in English.
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues.
### Apache Dubbo Component
Java SDK (apache/dubbo)
### Dubbo Version
Dubbo 3.3.6 JDK21
### Steps to reproduce this issue
**Root Cause:**
The issue occurs in a BatchExecutorQueue implementation where:
- Multiple worker threads produce messages into a ConcurrentLinkedQueue
- A single netty EventLoop thread consumes messages by copying all elements from ConcurrentLinkedQueue to a LinkedList in one operation
**Problem:**
In extreme scenes, when multiple worker threads generate large volumes of messages simultaneously, the bulk copy operation causes:
1. Long polling/blocking: The entire copy process blocks the EventLoop thread
2. Availability impact: Extended blocking reduces system responsiveness
3. Network I/O disruption: Since this uses Netty's EventLoop thread, the blocking prevents timely reading of incoming messages from other services, creating a cascading failure effect
**SourceCode:** [BatchExecutorQueue.run()](https://github.com/apache/dubbo/blob/dea02069082226b4677579474db4d156dbb8fb0e/dubbo-common/src/main/java/org/apache/dubbo/common/BatchExecutorQueue.java#L53)
<img width="576" height="867" alt="Image" src="https://github.com/user-attachments/assets/fa665eb1-a501-4c82-8d65-af3f3566c7a4" />
### What you expected to happen
**Impact:**
- EventLoop thread starvation
- Delayed message processing
- Potential service unavailability during high load
- Impaired ability to handle incoming requests from other services
The core issue is the synchronous bulk copy operation that doesn't yield the EventLoop thread, violating the non-blocking principle essential for proper EventLoop performance.
### Anything else
I conducted a comprehensive performance test to evaluate and compare three distinct implementation strategies for [BatchExecutorQueue.run()](https://github.com/apache/dubbo/blob/dea02069082226b4677579474db4d156dbb8fb0e/dubbo-common/src/main/java/org/apache/dubbo/common/BatchExecutorQueue.java#L53).
1. Old Implemention
<img width="406" height="1010" alt="Image" src="https://github.com/user-attachments/assets/ab4aa053-dbca-4fbe-a88c-b3ec342db955" />
2. Simple Implemention
<img width="434" height="830" alt="Image" src="https://github.com/user-attachments/assets/af2230db-9e35-4590-b366-8c65d256eabc" />
3. Two queue swap Implemention
<img width="643" height="1040" alt="Image" src="https://github.com/user-attachments/assets/3bbbe035-54b3-4d22-984d-af48df830f65" />
**Test Case**
<img width="1224" height="2045" alt="Image" src="https://github.com/user-attachments/assets/2d8cab97-2751-4423-9e73-5024ea2e3aa6" />
**Test Results**
---------------- current 1 test start ---------------
old implement message total at first second: 191046
old implement message test ended, flush 20000000 message spend: 8634s
simple implement message total at first second: 5685992
simple implement message test ended, flush 20000000 message spend: 3255s
new implement message total at first second: 6114666
new implement message test ended, flush 20000000 message spend: 3065s
---------------- current 1 test end ---------------
---------------- current 2 test start ---------------
old implement message total at first second: 5674259
old implement message test ended, flush 20000000 message spend: 3683s
simple implement message total at first second: 6138832
simple implement message test ended, flush 20000000 message spend: 3248s
new implement message total at first second: 6770830
new implement message test ended, flush 20000000 message spend: 2601s
---------------- current 2 test end ---------------
---------------- current 3 test start ---------------
old implement happened long loop
old implement message total at first second: 0
old implement message test ended, flush 20000000 message spend: 7799s
simple implement message total at first second: 2525975
simple implement message test ended, flush 20000000 message spend: 5145s
new implement message total at first second: 7497753
new implement message test ended, flush 20000000 message spend: 2492s
---------------- current 3 test end ---------------
---------------- current 4 test start ---------------
old implement message total at first second: 4634112
old implement message test ended, flush 20000000 message spend: 4768s
simple implement message total at first second: 3782242
simple implement message test ended, flush 20000000 message spend: 4839s
new implement message total at first second: 8016594
new implement message test ended, flush 20000000 message spend: 2528s
---------------- current 4 test end ---------------
---------------- current 5 test start ---------------
old implement happened long loop
old implement message total at first second: 0
old implement message test ended, flush 20000000 message spend: 8214s
simple implement message total at first second: 3491755
simple implement message test ended, flush 20000000 message spend: 4466s
new implement message total at first second: 7299851
new implement message test ended, flush 20000000 message spend: 2924s
---------------- current 5 test end ---------------
---------------- current 6 test start ---------------
old implement happened long loop
old implement message total at first second: 0
old implement message test ended, flush 20000000 message spend: 8783s
simple implement message total at first second: 2149255
simple implement message test ended, flush 20000000 message spend: 4661s
new implement message total at first second: 6844237
new implement message test ended, flush 20000000 message spend: 3036s
---------------- current 6 test end ---------------
---------------- current 7 test start ---------------
old implement happened long loop
old implement message total at first second: 0
old implement message test ended, flush 20000000 message spend: 5546s
simple implement message total at first second: 2256523
simple implement message test ended, flush 20000000 message spend: 5691s
new implement message total at first second: 8699966
new implement message test ended, flush 20000000 message spend: 2311s
---------------- current 7 test end ---------------
---------------- current 8 test start ---------------
old implement message total at first second: 5715617
old implement message test ended, flush 20000000 message spend: 4799s
simple implement message total at first second: 2465812
simple implement message test ended, flush 20000000 message spend: 5467s
new implement message total at first second: 7249972
new implement message test ended, flush 20000000 message spend: 2795s
---------------- current 8 test end ---------------
---------------- current 9 test start ---------------
old implement message total at first second: 4161615
old implement message test ended, flush 20000000 message spend: 6156s
simple implement message total at first second: 2383010
simple implement message test ended, flush 20000000 message spend: 5085s
new implement message total at first second: 7212382
new implement message test ended, flush 20000000 message spend: 2835s
---------------- current 9 test end ---------------
---------------- current 10 test start ---------------
old implement message total at first second: 6002999
old implement message test ended, flush 20000000 message spend: 5510s
simple implement message total at first second: 5626736
simple implement message test ended, flush 20000000 message spend: 3273s
new implement message total at first second: 7298384
new implement message test ended, flush 20000000 message spend: 2796s
---------------- current 10 test end ---------------
message total average for old's 10 test: 6428.25s
message total average for simple's 10 test: 4523.875s
message total average for new's 10 test: 2750.875s
### Are you willing to submit a pull request to fix on your own?
- [x] Yes I am willing to submit a pull request on my own!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] [Bug] BatchExecutorQueue performance risk [dubbo]
Posted by "zrlw (via GitHub)" <gi...@apache.org>.
zrlw commented on issue #15732:
URL: https://github.com/apache/dubbo/issues/15732#issuecomment-3425657485
it might be better to use JMH for bechmark testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] [Bug] BatchExecutorQueue performance risk [dubbo]
Posted by "fxli4 (via GitHub)" <gi...@apache.org>.
fxli4 closed issue #15732: [Bug] BatchExecutorQueue performance risk
URL: https://github.com/apache/dubbo/issues/15732
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org