You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/08/13 18:08:40 UTC
(datafusion) branch gh-readonly-queue/main/pr-24338-8c099dc15058cb95665e4a2312fd852d3c7042e4 created (now f17158738b)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-24338-8c099dc15058cb95665e4a2312fd852d3c7042e4
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at f17158738b perf(build): don't build the doc-generator binaries by default (#24338)
This branch includes the following new commits:
new f17158738b perf(build): don't build the doc-generator binaries by default (#24338)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: perf(build): don't build the doc-generator binaries by default (#24338)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-24338-8c099dc15058cb95665e4a2312fd852d3c7042e4
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit f17158738b74afbe71b009f197e0c84272062615
Author: Daniël Heres <da...@gmail.com>
AuthorDate: Thu Aug 13 18:08:15 2026 +0000
perf(build): don't build the doc-generator binaries by default (#24338)
## Which issue does this PR close?
Adresses: https://github.com/apache/datafusion/issues/13814
Found while profiling compile times for #24325 / #24326 / #24329 /
#24330.
## Rationale for this change
`datafusion/core/src/bin/` holds three binaries that regenerate the docs
under
`docs/source/user-guide`: `print_config_docs`,
`print_runtime_config_docs` and
`print_functions_docs`. Cargo auto-discovers them and they have no
`required-features`, so **every `cargo build` links all three** — each
one
~174MB, since each links the whole `datafusion` rlib.
Nothing in normal development uses them. They are run by
`dev/update_config_docs.sh` and `dev/update_function_docs.sh`, and by
the CI job
that checks the committed docs are up to date.
Two places where this shows up:
**Cold builds.** The three binaries link *after* every other unit has
finished,
so they sit on the critical path with nothing to overlap with.
`cargo build --timings` shows them occupying the last **3.5s** of a
`cargo build -p datafusion` (~8.8s of CPU), after the last library unit
completes.
**The tightest inner loop** — touch a file in core, rebuild. All three
are
relinked every time:
```
before: 3.0s 2.4s
after: 1.3s 1.1s
```
## What changes are included in this PR?
The three binaries move behind a new non-default `docs_generation`
feature, and
the two `dev/` scripts pass `--features docs_generation`.
Using `required-features` means declaring the `[[bin]]` targets
explicitly, since
auto-discovered targets cannot carry it.
## Are these changes tested?
- `cargo build -p datafusion` no longer produces the three binaries
- `cargo build -p datafusion --features docs_generation` does
- `./dev/update_config_docs.sh` still regenerates
`docs/source/user-guide/configs.md` byte-identically (empty `git diff`
afterwards), which is what the CI doc check compares
`dev/update_function_docs.sh` uses the same invocation pattern and all
three of
its call sites were updated; CI exercises both scripts.
## Are there any user-facing changes?
The three binaries are no longer built by a default `cargo build`.
Anyone who
ran them directly needs `--features docs_generation` — same as the
`dev/` scripts
now do. No library API changes.
If you would rather these lived outside the published crate altogether,
moving
them to a small non-published `dev/` crate would have the same effect on
build
times; I went with the smaller change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 <no...@anthropic.com>
---
datafusion/core/Cargo.toml | 22 ++++++++++++++++++++++
dev/update_config_docs.sh | 4 ++--
dev/update_function_docs.sh | 6 +++---
3 files changed, 27 insertions(+), 5 deletions(-)
diff --git a/datafusion/core/Cargo.toml b/datafusion/core/Cargo.toml
index 0fe48ddf6a..81e0418dd2 100644
--- a/datafusion/core/Cargo.toml
+++ b/datafusion/core/Cargo.toml
@@ -39,6 +39,13 @@ all-features = true
workspace = true
[features]
+# Enables the `print_config_docs`, `print_runtime_config_docs` and
+# `print_functions_docs` binaries, which regenerate the docs under
+# `docs/source/user-guide`. Off by default: they are only run by
+# `dev/update_config_docs.sh` and `dev/update_function_docs.sh`, and each one
+# links a ~174MB binary that would otherwise be relinked on every build of this
+# crate.
+docs_generation = []
nested_expressions = ["datafusion-functions-nested"]
# This feature is deprecated. Use the `nested_expressions` feature instead.
array_expressions = ["nested_expressions"]
@@ -190,6 +197,21 @@ ignored = ["datafusion-doc", "datafusion-macros", "dashmap"]
[target.'cfg(not(target_os = "windows"))'.dev-dependencies]
nix = { version = "0.31.3", features = ["fs"] }
+[[bin]]
+name = "print_config_docs"
+path = "src/bin/print_config_docs.rs"
+required-features = ["docs_generation"]
+
+[[bin]]
+name = "print_runtime_config_docs"
+path = "src/bin/print_runtime_config_docs.rs"
+required-features = ["docs_generation"]
+
+[[bin]]
+name = "print_functions_docs"
+path = "src/bin/print_functions_docs.rs"
+required-features = ["docs_generation"]
+
[[bench]]
harness = false
name = "aggregate_query_sql"
diff --git a/dev/update_config_docs.sh b/dev/update_config_docs.sh
index 7ab998f3da..df40c65210 100755
--- a/dev/update_config_docs.sh
+++ b/dev/update_config_docs.sh
@@ -27,8 +27,8 @@ cd "${ROOT_DIR}"
source "${ROOT_DIR}/ci/scripts/utils/tool_versions.sh"
TARGET_FILE="docs/source/user-guide/configs.md"
-PRINT_CONFIG_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --bin print_config_docs"
-PRINT_RUNTIME_CONFIG_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --bin print_runtime_config_docs"
+PRINT_CONFIG_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --features docs_generation --bin print_config_docs"
+PRINT_RUNTIME_CONFIG_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --features docs_generation --bin print_runtime_config_docs"
echo "Inserting header"
cat <<'EOF' > "$TARGET_FILE"
diff --git a/dev/update_function_docs.sh b/dev/update_function_docs.sh
index 86a272ae19..04266b8cbc 100755
--- a/dev/update_function_docs.sh
+++ b/dev/update_function_docs.sh
@@ -27,7 +27,7 @@ cd "${ROOT_DIR}"
source "${ROOT_DIR}/ci/scripts/utils/tool_versions.sh"
TARGET_FILE="docs/source/user-guide/sql/aggregate_functions.md"
-PRINT_AGGREGATE_FUNCTION_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --bin print_functions_docs -- aggregate"
+PRINT_AGGREGATE_FUNCTION_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --features docs_generation --bin print_functions_docs -- aggregate"
echo "Inserting header"
cat <<'EOF' > "$TARGET_FILE"
@@ -121,7 +121,7 @@ npx "prettier@${PRETTIER_VERSION}" --write "$TARGET_FILE"
echo "'$TARGET_FILE' successfully updated!"
TARGET_FILE="docs/source/user-guide/sql/scalar_functions.md"
-PRINT_SCALAR_FUNCTION_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --bin print_functions_docs -- scalar"
+PRINT_SCALAR_FUNCTION_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --features docs_generation --bin print_functions_docs -- scalar"
echo "Inserting header"
cat <<'EOF' > "$TARGET_FILE"
@@ -165,7 +165,7 @@ npx "prettier@${PRETTIER_VERSION}" --write "$TARGET_FILE"
echo "'$TARGET_FILE' successfully updated!"
TARGET_FILE="docs/source/user-guide/sql/window_functions.md"
-PRINT_WINDOW_FUNCTION_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --bin print_functions_docs -- window"
+PRINT_WINDOW_FUNCTION_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --features docs_generation --bin print_functions_docs -- window"
echo "Inserting header"
cat <<'EOF' > "$TARGET_FILE"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org