You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "ctsk (via GitHub)" <gi...@apache.org> on 2025/04/20 16:26:12 UTC
[PR] Feature/benchmark config from env [datafusion]
ctsk opened a new pull request, #15782:
URL: https://github.com/apache/datafusion/pull/15782
## Which issue does this PR close?
- Closes #15684
## Rationale for this change
When benchmarking, it is convenient to modify some datafusion settings between runs and compare the results.
## What changes are included in this PR?
This PR makes the benchmark executables pick up the environment variables. Additionally, it changes the target_partitions and batch_size options to fall back on the datafusion defaults when not set - this was necessary so that environment variables can override those options if they weren't provided.
## Are these changes tested?
No.
## Are there any user-facing changes?
Only developers of datafusion are affected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Feature/benchmark config from env [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on PR #15782:
URL: https://github.com/apache/datafusion/pull/15782#issuecomment-2817587293
Thanks, I think it's good to go after adding some simple docs for such configurations, in:
https://github.com/apache/datafusion/tree/main/benchmarks
and also the output of `./bench.sh help`
I think it's also great if we could add some output to show the actual set configurations like
```
export DATAFUSION_EXECUTION_TARGET_PARTITIONS=1
export DATAFUSION_EXECUTION_FOO=1
./bench.sh run tpch
```
Terminal output:
```
Set datafusion.execution.target_partitions to 1 from the environment variable
... start running the benchmark
```
We can just only an issue for it now, and leave it as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Feature/benchmark config from env [datafusion]
Posted by "ctsk (via GitHub)" <gi...@apache.org>.
ctsk commented on PR #15782:
URL: https://github.com/apache/datafusion/pull/15782#issuecomment-2820537298
I can't see an easy way to check what environment variables were actually picked up. I've opted to add some logging to `ConfigOptions::from_env`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Feature/benchmark config from env [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead merged PR #15782:
URL: https://github.com/apache/datafusion/pull/15782
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org