You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "zuston (via GitHub)" <gi...@apache.org> on 2025/10/14 10:07:56 UTC
[I] Do we need to align the number of Tokio runtime threads with the number of executor cores? [datafusion-comet]
zuston opened a new issue, #2572:
URL: https://github.com/apache/datafusion-comet/issues/2572
### What is the problem the feature request solves?
Right now, each executor’s Tokio runtime spins up one thread per physical core, which doesn’t seem reasonable. Do we need to align the number of Tokio runtime threads with the number of `spark executor cores?`
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Do we need to align the number of Tokio runtime threads with the number of executor cores? [datafusion-comet]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #2572:
URL: https://github.com/apache/datafusion-comet/issues/2572#issuecomment-3413485159
> @zuston are you asking/suggesting that we try and automate this by passing the number of executor cores into native lib and dynamically adjusting the tokio thread pool size instead of relying on an env var?
Yes. the default thread pool size could be aligned with the spark executor cores. and the env var could also be reserved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Do we need to align the number of Tokio runtime threads with the number of executor cores? [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #2572:
URL: https://github.com/apache/datafusion-comet/issues/2572#issuecomment-3403445885
> Right now, each executor’s Tokio runtime spins up one thread per physical core, which doesn’t seem reasonable. Do we need to align the number of Tokio runtime threads with the number of `spark executor cores?`
Yeah I suspect this is fairly over-subscribing on threads. This might actually make sense for I/O heavy workloads with a lot of parallel requests (if we can manage to submit all of that), but it's definitely a good point of discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Do we need to align the number of Tokio runtime threads with the number of executor cores? [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2572:
URL: https://github.com/apache/datafusion-comet/issues/2572#issuecomment-3412619683
> > Maybe the default number of Tokio worker threads should be aligned with the number of Spark executor cores.
>
> Is it necessary to create a PR for alignment? [@andygrove](https://github.com/andygrove) [@mbutrovich](https://github.com/mbutrovich)
We added this to the tuning guide:
```
It is recommended that COMET_WORKER_THREADS be set to the number of executor cores. This may not be necessary in some environments, such as Kubernetes, where the number of cores allocated to a pod will already be equal to the number of executor cores.
```
@zuston are you asking/suggesting that we try and automate this by passing the number of executor cores into native lib and dynamically adjusting the tokio thread pool size instead of relying on an env var?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Do we need to align the number of Tokio runtime threads with the number of executor cores? [datafusion-comet]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #2572:
URL: https://github.com/apache/datafusion-comet/issues/2572#issuecomment-3404199030
> Yeah I suspect this is fairly over-subscribing on threads. This might actually make sense for I/O heavy workloads with a lot of parallel requests (if we can manage to submit all of that), but it's definitely a good point of discussion.
According to Tokio best practices, heavy I/O workloads should be delegated to the Tokio blocking thread pool.
> In some environments, such as Kubernetes, the number of spark executor cores and number of "physical" cores in the container would typically be the same.
In the Hadoop YARN environment, this behavior is inconsistent with how it runs on Kubernetes. Maybe the default number of Tokio worker threads should be aligned with the number of Spark executor cores.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Do we need to align the number of Tokio runtime threads with the number of executor cores? [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2572:
URL: https://github.com/apache/datafusion-comet/issues/2572#issuecomment-3403786939
> Do we need to align the number of Tokio runtime threads with the number of `spark executor cores?`
Yes, that would make sense to me. It would be good to update the tuning guide to explain this.
In some environments, such as Kubernetes, the number of spark executor cores and number of "physical" cores in the container would typically be the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Do we need to align the number of Tokio runtime threads with the number of executor cores? [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2572: Do we need to align the number of Tokio runtime threads with the number of executor cores?
URL: https://github.com/apache/datafusion-comet/issues/2572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Do we need to align the number of Tokio runtime threads with the number of executor cores? [datafusion-comet]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #2572:
URL: https://github.com/apache/datafusion-comet/issues/2572#issuecomment-3408935226
> Maybe the default number of Tokio worker threads should be aligned with the number of Spark executor cores.
Is it necessary to create a PR for alignment? @andygrove @mbutrovich
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org