You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/05/08 17:28:58 UTC
[GitHub] [arrow-datafusion] alamb opened a new issue, #6287: [DISCUSS] Set DataFusion settings for maximum "out of the box" performance
alamb opened a new issue, #6287:
URL: https://github.com/apache/arrow-datafusion/issues/6287
### Is your feature request related to a problem or challenge?
DataFusion has many configuration options that control various performance optimization.
There is a tradeoff between some of these options between faster query execution (for more than linear resource consumption) and pure efficiency.
We have benchmarks such as a [tpch](https://github.com/apache/arrow-datafusion/blob/main/benchmarks/src/bin/tpch.rs) runner that typically run with a single core. These are great as performance unit tests in well controlled environments (and avoid task overhead, and other non determinism introduced with multi-core execution), however they don't mimic what users typically run with.
Up to now we have taken a conservative approach and only enabled optimizations by default if they make everything faster. I would like to change our philosophy and optimize for "out of the box" performance
You can see examples of other systems tuning knobs up for performance:
IOx: https://github.com/influxdata/influxdb_iox/blob/ad28ebb7650d1cd21b995ee5b514d8da3580f22b/datafusion_util/src/config.rs#L20-L23
Here is a recent example from https://hussainsultan.com/posts/unbundled-datafusion/
```python
runtime = RuntimeConfig().with_disk_manager_os().with_fair_spill_pool(100000000)
config = (
SessionConfig()
.with_create_default_catalog_and_schema(True)
.with_target_partitions(8)
.with_information_schema(True)
.with_repartition_joins(True)
.with_repartition_aggregations(True)
.with_repartition_windows(True)
.with_parquet_pruning(True)
.set("datafusion.execution.parquet.pushdown_filters", "true")
)
ctx = SessionContext(config, runtime)
ctx.register_parquet("orders", "../../../fanniemae-benchmark/sf10/raw/orders.parquet")
```
### Describe the solution you'd like
I would like to change defaults to:
1. Repartition always when possible (to increase parallelism by default)
2. Push down all parquet filters (e.g. https://github.com/apache/arrow-datafusion/issues/4085 and https://github.com/apache/arrow-datafusion/issues/3463)
### Describe alternatives you've considered
We can leave the defaults alone and make users change the defaults
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #6287: [DISCUSS] Set DataFusion settings for maximum "out of the box" performance
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6287:
URL: https://github.com/apache/arrow-datafusion/issues/6287#issuecomment-1542147503
#6325 is another project to improve the performance out of the box
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #6287: [DISCUSS] Set DataFusion settings for maximum "out of the box" performance
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6287:
URL: https://github.com/apache/arrow-datafusion/issues/6287#issuecomment-1538768949
Here is another example of where we had to finagle settings to make DataFusion fast: https://github.com/apache/arrow-datafusion/issues/5942
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] comphead commented on issue #6287: [DISCUSS] Set DataFusion settings for maximum "out of the box" performance
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #6287:
URL: https://github.com/apache/arrow-datafusion/issues/6287#issuecomment-1538796242
Another example what I have found is increasing batch size solves some performance troubles related to excessive collection allocation or loops.
I suppose there is also a trouble when the code gets triggered for empty batches, so the code spins CPU but in fact does nothing, I think noticed this behavior couple of times but need to have a good reproduce test to prove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ozankabak commented on issue #6287: [DISCUSS] Set DataFusion settings for maximum "out of the box" performance
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on issue #6287:
URL: https://github.com/apache/arrow-datafusion/issues/6287#issuecomment-1538806854
Thank you @alamb for putting this together. I definitely agree that we should choose the default configuration settings to squeeze maximum performance. OOTB performance is a common blogging subject, and search results that yield to such posts is a big entry to point to the user funnel. We should give users the right impression in terms of where Datafusion stands in terms of the performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org