You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/04/27 15:33:25 UTC
(datafusion) branch gh-readonly-queue/main/pr-21831-bff0ffb682e723669a65584e6757d81264ce00db created (now 2676297749)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-21831-bff0ffb682e723669a65584e6757d81264ce00db
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at 2676297749 chore: add internal markdown link check (#21831)
This branch includes the following new commits:
new 2676297749 chore: add internal markdown link check (#21831)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: chore: add internal markdown link check (#21831)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-21831-bff0ffb682e723669a65584e6757d81264ce00db
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 26762977499c3ea581a8c3f698c855c4d607a2c6
Author: Geethapranay1 <16...@users.noreply.github.com>
AuthorDate: Mon Apr 27 21:03:02 2026 +0530
chore: add internal markdown link check (#21831)
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #21747
## Rationale for this change
datafusion did not have a CI check for broken links in markdown content,
docs workflows build and deploy docs, and dev checks formatting and
spelling, but none of them validate link targets.
This pr adds a dedicated link check for internal markdown links so
broken references fail early in PRs.
I kept the scope internal-only to avoid flaky CI failures from external
websites and rate limits.
Rust doc comments remain covered by the existing rustdoc CI job.
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
- Added a new Dev workflow job, **Check Markdown Links**, in `dev.yml`.
- Added `LYCHEE_VERSION` pin in `tool_versions.sh`.
- Added `markdown_link_check.sh` to run lychee on the selected markdown
paths.
- Added `lychee.toml` with internal-link policy and exclusions.
- Added check markdown links to required status checks in `.asf.yaml`.
- Updated contributor testing docs with the new local command and scope
note.
- Fixed internal markdown links that failed under the new check in:
- `roadmap.md`
- `49.0.0.md`
- `overview.md`
- `dataframe.md`
- `format_options.md`
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
Yes,
- `python3 ci/scripts/check_asf_yaml_status_checks.py` passed.
- `bash -n ci/scripts/markdown_link_check.sh` passed.
- `bash ci/scripts/markdown_link_check.sh` passed with 0 errors.
- `cargo fmt --all --check` passed.
OK: All 5 required_status_checks match existing GitHub Actions jobs.
🔍 12824 Total (in 0s) ✅ 490 OK 🚫 0 Errors 👻 12334 Excluded
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
No,
There is one contributor-facing CI change: PRs now fail when internal
markdown links break in the checked markdown files.
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
---------
Co-authored-by: Oleks V <co...@users.noreply.github.com>
---
.asf.yaml | 1 +
.github/workflows/dev.yml | 19 +++++++++++++++
.../tool_versions.sh => markdown_link_check.sh} | 16 ++++++++++---
ci/scripts/utils/tool_versions.sh | 1 +
docs/source/contributor-guide/roadmap.md | 2 +-
docs/source/contributor-guide/testing.md | 28 ++++++++++++++++++++++
docs/source/library-user-guide/upgrading/49.0.0.md | 2 +-
docs/source/user-guide/cli/overview.md | 4 ++--
docs/source/user-guide/dataframe.md | 2 +-
docs/source/user-guide/sql/format_options.md | 2 +-
ci/scripts/utils/tool_versions.sh => lychee.toml | 19 +++++++++++----
11 files changed, 82 insertions(+), 14 deletions(-)
diff --git a/.asf.yaml b/.asf.yaml
index 73adb1c058..d953f9c585 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -55,6 +55,7 @@ github:
contexts:
- "Check License Header"
- "Use prettier to check formatting of documents"
+ - "Check Markdown Links"
- "Validate required_status_checks in .asf.yaml"
- "Spell Check with Typos"
# needs to be updated as part of the release process
diff --git a/.github/workflows/dev.yml b/.github/workflows/dev.yml
index a247f07333..55e8572408 100644
--- a/.github/workflows/dev.yml
+++ b/.github/workflows/dev.yml
@@ -23,6 +23,9 @@ on:
pull_request:
merge_group:
+permissions:
+ contents: read
+
concurrency:
group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
cancel-in-progress: true
@@ -51,6 +54,22 @@ jobs:
# if you encounter error, see instructions inside the script
run: ci/scripts/doc_prettier_check.sh
+ markdown-link-check:
+ name: Check Markdown Links
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+ - name: Load tool versions
+ run: |
+ source ci/scripts/utils/tool_versions.sh
+ echo "LYCHEE_VERSION=${LYCHEE_VERSION}" >> "$GITHUB_ENV"
+ - name: Install lychee
+ uses: taiki-e/install-action@055f5df8c3f65ea01cd41e9dc855becd88953486 # v2.75.18
+ with:
+ tool: lychee@${{ env.LYCHEE_VERSION }}
+ - name: Run markdown link check
+ run: bash ci/scripts/markdown_link_check.sh
+
asf-yaml-check:
name: Validate required_status_checks in .asf.yaml
runs-on: ubuntu-latest
diff --git a/ci/scripts/utils/tool_versions.sh b/ci/scripts/markdown_link_check.sh
similarity index 66%
copy from ci/scripts/utils/tool_versions.sh
copy to ci/scripts/markdown_link_check.sh
index ac731ed0d5..65fede4111 100644
--- a/ci/scripts/utils/tool_versions.sh
+++ b/ci/scripts/markdown_link_check.sh
@@ -17,7 +17,17 @@
# specific language governing permissions and limitations
# under the License.
-# This file defines centralized tool versions used by CI and development scripts.
-# It is intended to be sourced by other scripts and should not be executed directly.
+set -euo pipefail
-PRETTIER_VERSION="2.7.1"
+ROOT_DIR="$(git rev-parse --show-toplevel)"
+
+cd "${ROOT_DIR}"
+
+MARKDOWN_FILES=()
+while IFS= read -r file; do
+ MARKDOWN_FILES+=("${file}")
+done < <(
+ git -C "${ROOT_DIR}" ls-files 'README.md' 'CONTRIBUTING.md' 'docs/**/*.md' 'datafusion-cli/README.md' 'datafusion-examples/README.md' 'dev/**/*.md'
+)
+
+lychee --no-progress --config "${ROOT_DIR}/lychee.toml" "${MARKDOWN_FILES[@]}"
diff --git a/ci/scripts/utils/tool_versions.sh b/ci/scripts/utils/tool_versions.sh
index ac731ed0d5..237b18b62e 100644
--- a/ci/scripts/utils/tool_versions.sh
+++ b/ci/scripts/utils/tool_versions.sh
@@ -21,3 +21,4 @@
# It is intended to be sourced by other scripts and should not be executed directly.
PRETTIER_VERSION="2.7.1"
+LYCHEE_VERSION="0.23.0"
diff --git a/docs/source/contributor-guide/roadmap.md b/docs/source/contributor-guide/roadmap.md
index aac0710dad..bfaf398d3f 100644
--- a/docs/source/contributor-guide/roadmap.md
+++ b/docs/source/contributor-guide/roadmap.md
@@ -19,7 +19,7 @@ under the License.
# Roadmap and Improvement Proposals
-The [project introduction](../user-guide/introduction) explains the
+The [project introduction](../user-guide/introduction.md) explains the
overview and goals of DataFusion, and our development efforts largely
align to that vision.
diff --git a/docs/source/contributor-guide/testing.md b/docs/source/contributor-guide/testing.md
index 6b8e4568ec..3b644f610b 100644
--- a/docs/source/contributor-guide/testing.md
+++ b/docs/source/contributor-guide/testing.md
@@ -186,6 +186,34 @@ tested in the same way using the [doc_comment] crate. See the end of
[doc_comment]: https://docs.rs/doc-comment/latest/doc_comment
[core/src/lib.rs]: https://github.com/apache/datafusion/blob/main/datafusion/core/src/lib.rs#L583
+## Documentation Link Checks
+
+Run the internal markdown link check locally:
+
+```shell
+source ci/scripts/utils/tool_versions.sh
+cargo install lychee --locked --version "${LYCHEE_VERSION}"
+bash ci/scripts/markdown_link_check.sh
+```
+
+Notes:
+
+- The script is run with `bash` and is compatible with the default Bash on macOS (no `mapfile` dependency).
+- The CI configuration currently checks internal markdown links only. External `http(s)` and `mailto` links are excluded to avoid flaky failures.
+
+When a link is broken, lychee prints the file and URL/path that failed. For example:
+
+```text
+[docs/source/user-guide/cli/overview.md]:
+ [ERROR] file:///.../docs/source/user-guide/cli/missing-page.md | Cannot find file: File not found. Check if file exists and path is correct
+```
+
+Rust doc comments are validated by rustdoc in CI and can be checked locally with:
+
+```shell
+bash ci/scripts/rust_docs.sh
+```
+
## Benchmarks
### Criterion Benchmarks
diff --git a/docs/source/library-user-guide/upgrading/49.0.0.md b/docs/source/library-user-guide/upgrading/49.0.0.md
index 92dee81355..92267a80fa 100644
--- a/docs/source/library-user-guide/upgrading/49.0.0.md
+++ b/docs/source/library-user-guide/upgrading/49.0.0.md
@@ -123,7 +123,7 @@ Or via SQL:
SET datafusion.execution.spill_compression = 'zstd';
```
-For more details about this configuration option, including performance trade-offs between different compression codecs, see the [Configuration Settings](../../user-guide/configs) documentation.
+For more details about this configuration option, including performance trade-offs between different compression codecs, see the [Configuration Settings](../../user-guide/configs.md) documentation.
### Deprecated `map_varchar_to_utf8view` configuration option
diff --git a/docs/source/user-guide/cli/overview.md b/docs/source/user-guide/cli/overview.md
index 86beea0e82..e0228d3ea0 100644
--- a/docs/source/user-guide/cli/overview.md
+++ b/docs/source/user-guide/cli/overview.md
@@ -41,5 +41,5 @@ DataFusion CLI v37.0.0
Elapsed 1.969 seconds.
```
-For more information, see the [Installation](installation), [Usage Guide](usage)
-and [Data Sources](datasources) sections.
+For more information, see the [Installation](installation.md), [Usage Guide](usage.md)
+and [Data Sources](datasources.md) sections.
diff --git a/docs/source/user-guide/dataframe.md b/docs/source/user-guide/dataframe.md
index 85724a7239..d35b543bfd 100644
--- a/docs/source/user-guide/dataframe.md
+++ b/docs/source/user-guide/dataframe.md
@@ -122,4 +122,4 @@ async fn main() -> Result<()> {
[`collect`]: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html#method.collect
[library users guide]: ../library-user-guide/using-the-dataframe-api.md
[api reference on docs.rs]: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html
-[expressions reference]: expressions
+[expressions reference]: expressions.md
diff --git a/docs/source/user-guide/sql/format_options.md b/docs/source/user-guide/sql/format_options.md
index 3385080314..46d251c18e 100644
--- a/docs/source/user-guide/sql/format_options.md
+++ b/docs/source/user-guide/sql/format_options.md
@@ -29,7 +29,7 @@ Format-related options can be specified in three ways, in decreasing order of pr
- `COPY` option tuples
- Session-level config defaults
-For a list of supported session-level config defaults, see [Configuration Settings](../configs). These defaults apply to all operations but have the lowest level of precedence.
+For a list of supported session-level config defaults, see [Configuration Settings](../configs.md). These defaults apply to all operations but have the lowest level of precedence.
If creating an external table, table-specific format options can be specified when the table is created using the `OPTIONS` clause:
diff --git a/ci/scripts/utils/tool_versions.sh b/lychee.toml
similarity index 78%
copy from ci/scripts/utils/tool_versions.sh
copy to lychee.toml
index ac731ed0d5..74245ee7ec 100644
--- a/ci/scripts/utils/tool_versions.sh
+++ b/lychee.toml
@@ -1,5 +1,3 @@
-#!/usr/bin/env bash
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -17,7 +15,18 @@
# specific language governing permissions and limitations
# under the License.
-# This file defines centralized tool versions used by CI and development scripts.
-# It is intended to be sourced by other scripts and should not be executed directly.
+timeout = 20
+max_retries = 2
+retry_wait_time = 2
+
+exclude_path = [
+ "target",
+ "docs/build",
+ "datafusion/core/benches/tpch-csv",
+]
-PRETTIER_VERSION="2.7.1"
+exclude = [
+ "^http://",
+ "^https://",
+ "^mailto:",
+]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org