You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "milenkovicm (via GitHub)" <gi...@apache.org> on 2024/11/26 21:45:41 UTC
[PR] feat: Add option to limit number of executor worker threads [datafusion-ballista]
milenkovicm opened a new pull request, #1139:
URL: https://github.com/apache/datafusion-ballista/pull/1139
this configuration option is similar to `spark.executor.cores`, it sets number of tokio worker threads limiting executor cpu utilisation.
# Which issue does this PR close?
Closes none .
# Rationale for this change
provide ability to users to limit number of cores which executor can utilise limiting overall resource utilisation
# What changes are included in this PR?
- new configuration option
- tokio executors got their own name
# Are there any user-facing changes?
new configuration option exposed to user
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add option to limit number of executor worker threads [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1139:
URL: https://github.com/apache/datafusion-ballista/pull/1139#issuecomment-2502014315
will open another PR to remove `cache_` executor config options
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add option to limit number of executor worker threads [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1139:
URL: https://github.com/apache/datafusion-ballista/pull/1139#issuecomment-2503488497
this PR may make no sense as there is dedicated executor which has limited number of runners
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add option to limit number of executor worker threads [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1139:
URL: https://github.com/apache/datafusion-ballista/pull/1139#issuecomment-2503510326
DedicatedExecutor will limit number of threads based on `concurrent_tasks`, maybe we should document it better. Closing this task, apologies for the noise
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add option to limit number of executor worker threads [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1139:
URL: https://github.com/apache/datafusion-ballista/pull/1139#issuecomment-2501996535
there is also `concurrent_tasks` which configuration option, i believe if `executor_cores` is set it `concurrent_tasks` should be <= to `executor_cores`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add option to limit number of executor worker threads [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm closed pull request #1139: feat: Add option to limit number of executor worker threads
URL: https://github.com/apache/datafusion-ballista/pull/1139
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org