You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "milenkovicm (via GitHub)" <gi...@apache.org> on 2024/10/30 19:15:24 UTC
[I] Trim down `BallistaConfig` [datafusion-ballista]
milenkovicm opened a new issue, #1104:
URL: https://github.com/apache/datafusion-ballista/issues/1104
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I would like to propose to trim down `BallistaConfig`, make `SessionConfig` as main way to configure Ballista and make `BallistaConfig` just a part of it, with ballista specific configurations
**Describe the solution you'd like**
merge following configuration:
```
+-------------------------------------------------------------------------+---------------------------+
| name | value |
+-------------------------------------------------------------------------+---------------------------+
| ballista.batch.size | 8192 |
| datafusion.execution.batch_size | 8192 |
| ballista.collect_statistics | false |
| datafusion.execution.collect_statistics | false |
| ballista.optimizer.hash_join_single_partition_threshold | 1048576 |
| datafusion.optimizer.hash_join_single_partition_threshold | 1048576 |
| ballista.parquet.pruning | true |
| datafusion.execution.parquet.pruning | true |
| ballista.repartition.aggregations | true |
| datafusion.optimizer.repartition_aggregations | true |
| ballista.repartition.joins | true |
| datafusion.optimizer.repartition_joins | true |
| ballista.repartition.windows | true |
| datafusion.optimizer.repartition_windows | true |
| ballista.with_information_schema | false |
| datafusion.catalog.information_schema | true |
| ballista.shuffle.partitions | 16 |
| datafusion.execution.target_partitions | 8 |
| ballista.standalone.parallelism | 8 |
| datafusion.execution.target_partitions | 8 |
+-------------------------------------------------------------------------+---------------------------+
```
If we check `/ballista/scheduler/src/state/session_manager.rs`
```rust
pub fn create_datafusion_context(
ballista_config: &BallistaConfig,
session_builder: SessionBuilder,
) -> Arc<SessionContext> {
let config =
SessionConfig::from_string_hash_map(&ballista_config.settings().clone()).unwrap();
let config = config
.with_target_partitions(ballista_config.default_shuffle_partitions())
.with_batch_size(ballista_config.default_batch_size())
.with_repartition_joins(ballista_config.repartition_joins())
.with_repartition_aggregations(ballista_config.repartition_aggregations())
.with_repartition_windows(ballista_config.repartition_windows())
.with_collect_statistics(ballista_config.collect_statistics())
.with_parquet_pruning(ballista_config.parquet_pruning())
.set_usize(
"datafusion.optimizer.hash_join_single_partition_threshold",
ballista_config.hash_join_single_partition_threshold(),
)
.set_bool("datafusion.optimizer.enable_round_robin_repartition", false);
let session_state = session_builder(config);
Arc::new(SessionContext::new_with_state(session_state))
}
```
ballista specific configuration, which should be preserved:
```
+-------------------------------------------------------------------------+---------------------------+
| name | value |
+-------------------------------------------------------------------------+---------------------------+
| ballista.grpc_client_max_message_size | 16777216 |
| ballista.job.name | |
+-------------------------------------------------------------------------+---------------------------+
```
we can see that configuration map 1 to 1 to datafusion configuration
Note: `datafusion.optimizer.enable_round_robin_repartition` has to be false `false` ;
**Describe alternatives you've considered**
keep `BallistaConfig` but at the moment i see no benefit of keeping it or scenarios which `SessionConfig` cant support
**Additional context**
As it stands with introduction of `SessionContextExt`, `BallistaConfig` has been removed from all public interfaces, with this change `BallistaConfig` will probably be removed from most internal interfaces as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Trim down `BallistaConfig` [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1104:
URL: https://github.com/apache/datafusion-ballista/issues/1104#issuecomment-2448143177
will take this once #1103 and #1099 get merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Trim down `BallistaConfig` [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #1104: Trim down `BallistaConfig`
URL: https://github.com/apache/datafusion-ballista/issues/1104
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org