You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "rickyma (via GitHub)" <gi...@apache.org> on 2024/04/10 14:28:08 UTC
[I] [Bug] Use thread pool to control the concurrency of data reading threads when enabling Netty [incubator-uniffle]
rickyma opened a new issue, #1639:
URL: https://github.com/apache/incubator-uniffle/issues/1639
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
As described in https://github.com/apache/incubator-uniffle/issues/1596#issuecomment-2012168823, I find out https://github.com/apache/incubator-uniffle/pull/1605 is not enough.
For example, because Netty processing is asynchronous, even if we only configure the number of Netty workers to be 200, the number of threads running concurrently to read local files may far exceed 200. This is because when the Netty worker thread finishes calling `writeAndFlush`, it immediately goes to receive and process the next client request, without waiting for the data to be sent completely. The process of sending data is asynchronous.
So we should limit the concurrency when reading local files.
### Affects Version(s)
master
### Uniffle Server Log Output
_No response_
### Uniffle Engine Log Output
_No response_
### Uniffle Server Configurations
_No response_
### Uniffle Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Improvement] Use thread pool to control the concurrency of data reading threads when enabling Netty [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma closed issue #1639: [Improvement] Use thread pool to control the concurrency of data reading threads when enabling Netty
URL: https://github.com/apache/incubator-uniffle/issues/1639
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Improvement] Use thread pool to control the concurrency of data reading threads when enabling Netty [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on issue #1639:
URL: https://github.com/apache/incubator-uniffle/issues/1639#issuecomment-2102685279
Actually, whether it's reading or writing, they are both performance loads, and I don't see the necessity to distinguish and handle them separately for now. Let's close this requirement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org