You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/22 15:55:52 UTC
[I] Generate the config reference documentation from the config registry [datafusion-ballista]
andygrove opened a new issue, #2145:
URL: https://github.com/apache/datafusion-ballista/issues/2145
`docs/source/user-guide/configs.md` is hand-maintained, so it drifts from the
config keys the code actually reads.
Auditing every `BALLISTA_*` constant in `ballista/core/src/config.rs` against
`docs/source/` turns up nine keys documented nowhere:
- `ballista.planner.coalesce.enabled`
- `ballista.planner.coalesce.target_partition_bytes`
- `ballista.planner.coalesce.small_partition_factor`
- `ballista.planner.coalesce.merged_partition_factor`
- `ballista.planner.propagate_empty.enabled`
- `ballista.shuffle.compression.codec`
- `ballista.shuffle.writer_channel_capacity`
- `ballista.testing.chaos_execution.enabled`, `.probability`, `.fault_type`, `.seed`
Two more are documented only incidentally, outside the config reference:
`ballista.planner.adaptive_join.enabled` appears solely in the 54.0.0 upgrade
notes, and `ballista.optimizer.hash_join_max_build_partition_bytes` solely in
the benchmarking guide.
The AQE settings illustrate the failure mode: they were documented in
`tuning-guide.md` next to the prose explaining them, and `configs.md` was never
updated or cross-linked.
Fixing the current gaps by hand would leave the same problem in place. Since
`ballista/core/src/config.rs` already builds a runtime registry
(`CONFIG_ENTRIES`) carrying each key's name, description, type, and default,
the reference tables can be generated from it instead, with CI failing when a
registered key is documented nowhere.
The sibling DataFusion repo does this already via `dev/update_config_docs.sh`
and a CI check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Generate the config reference documentation from the config registry [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2145: Generate the config reference documentation from the config registry
URL: https://github.com/apache/datafusion-ballista/issues/2145
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org