You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/05/11 10:42:14 UTC

(datafusion) branch gh-readonly-queue/main/pr-22003-2b1aa439e3c4959ab8f1c05a9208464e0e806888 created (now 2131d8b87c)

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-22003-2b1aa439e3c4959ab8f1c05a9208464e0e806888
in repository https://gitbox.apache.org/repos/asf/datafusion.git


      at 2131d8b87c docs: add llms.txt ecosystem hub at site root (#22003)

This branch includes the following new commits:

     new 2131d8b87c docs: add llms.txt ecosystem hub at site root (#22003)

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org


(datafusion) 01/01: docs: add llms.txt ecosystem hub at site root (#22003)

Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-22003-2b1aa439e3c4959ab8f1c05a9208464e0e806888
in repository https://gitbox.apache.org/repos/asf/datafusion.git

commit 2131d8b87caf675bd13a58a3d2f509ba42ebf908
Author: Tim Saucer <ti...@gmail.com>
AuthorDate: Mon May 11 06:41:54 2026 -0400

    docs: add llms.txt ecosystem hub at site root (#22003)
    
    ## Which issue does this PR close?
    
    - Addresses part of apache/datafusion-python#1394 ("Make it easier for
    agents to generate datafusion-python code") — this is **PR 6** in the
    implementation plan
    ([comment](https://github.com/apache/datafusion-python/issues/1394#issuecomment-4252413645)):
    the upstream `apache/datafusion` `llms.txt` hub.
    
    ## Rationale for this change
    
    [llms.txt](https://llmstxt.org) is an emerging convention for exposing a
    machine-readable, agent-facing entry point at a site's docs root.
    Subprojects in the DataFusion ecosystem are starting to publish their
    own (`apache/datafusion-python` PR apache/datafusion-python#1505 added
    one). The main `datafusion.apache.org` site is the natural top-level
    discovery point for the whole ecosystem, so it should expose a hub
    `llms.txt` that points agents at:
    
    - the core DataFusion (Rust) user / library / contributor guides and
    Rust API docs,
    - each subproject's docs root, where agents following the llmstxt.org
    convention can probe `<docs root>/llms.txt` for project-specific
    guidance.
    
    Net effect: an agent fetching `https://datafusion.apache.org/llms.txt`
    lands in a categorized directory of the entire ecosystem's agent
    guidance.
    
    ## What changes are included in this PR?
    
    - `docs/source/llms.txt` — new file, llmstxt.org schema. Sections: Core
    DataFusion (Rust), Subprojects, Optional. The Subprojects section links
    to docs roots (not pending `llms.txt` URLs) and includes a one-line note
    describing the probe convention so the hub stays correct as subprojects
    ship their own files.
    - `docs/source/conf.py` — `html_extra_path = ["llms.txt"]` so Sphinx
    copies the file verbatim to the build output root, served at
    `https://datafusion.apache.org/llms.txt`.
    - `dev/release/rat_exclude_files.txt` — exclude `docs/source/llms.txt`
    from the RAT license-header check (the file body is rendered markdown
    and cannot carry the standard `..` comment header without breaking the
    format).
    
    ## Are these changes tested?
    
    No automated tests. The change is a single static file plus a Sphinx
    config line that mirrors a pattern already used in
    `apache/datafusion-python` (`html_extra_path = ["llms.txt"]`, PR
    apache/datafusion-python#1505). Verification will be done at deploy
    time: confirm `https://datafusion.apache.org/llms.txt` resolves and
    renders.
    
    ## Are there any user-facing changes?
    
    Yes — adds a new public URL `https://datafusion.apache.org/llms.txt`. No
    existing pages are modified. No API changes.
    
    Co-authored-by: Claude Opus 4.7 (1M context) <no...@anthropic.com>
---
 dev/release/rat_exclude_files.txt |  1 +
 docs/source/conf.py               |  4 ++++
 docs/source/llms.txt              | 26 ++++++++++++++++++++++++++
 3 files changed, 31 insertions(+)

diff --git a/dev/release/rat_exclude_files.txt b/dev/release/rat_exclude_files.txt
index 7953a5b4e2..f5ce368df7 100644
--- a/dev/release/rat_exclude_files.txt
+++ b/dev/release/rat_exclude_files.txt
@@ -60,6 +60,7 @@ datafusion/proto-common/src/generated/prost.rs
 .github/ISSUE_TEMPLATE/bug_report.yml
 .github/ISSUE_TEMPLATE/feature_request.yml
 .github/workflows/docs.yaml
+docs/source/llms.txt
 **/node_modules/*
 datafusion/wasmtest/pkg/*
 clippy.toml
diff --git a/docs/source/conf.py b/docs/source/conf.py
index 03dcfb5bfa..c8027fc71b 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -109,6 +109,10 @@ html_context = {
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ["_static"]
 
+# Copy agent-facing files (llms.txt) verbatim to the site root so they
+# resolve at the conventional URL `https://datafusion.apache.org/llms.txt`.
+html_extra_path = ["llms.txt"]
+
 html_logo = "_static/images/2x_bgwhite_original.png"
 
 html_css_files = ["theme_overrides.css"]
diff --git a/docs/source/llms.txt b/docs/source/llms.txt
new file mode 100644
index 0000000000..5d738107c8
--- /dev/null
+++ b/docs/source/llms.txt
@@ -0,0 +1,26 @@
+# Apache DataFusion
+
+> Apache DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory format. This file is a directory of agent-facing entry points for the DataFusion ecosystem — the Rust core query engine and its subprojects. Subproject `llms.txt` files contain the project-specific guidance for writing code against each one.
+
+## Core DataFusion (Rust)
+
+- [User guide](https://datafusion.apache.org/user-guide/introduction.html): install, example usage, SQL, DataFrame, expressions, configuration, explain plans.
+- [Library user guide](https://datafusion.apache.org/library-user-guide/index.html): embedding DataFusion, extending SQL, custom table providers, building logical plans, the query optimizer.
+- [Contributor guide](https://datafusion.apache.org/contributor-guide/index.html): development environment, architecture, testing, release management, governance.
+- [Rust API docs (`docs.rs`)](https://docs.rs/datafusion/latest/datafusion/): generated reference for the `datafusion` crate.
+- [GitHub repository](https://github.com/apache/datafusion): source, issues, pull requests.
+
+## Subprojects
+
+Each subproject may expose its own `llms.txt` at `<docs root>/llms.txt` — agents following the [llmstxt.org](https://llmstxt.org) convention can probe these paths for project-specific guidance.
+
+- [DataFusion Python](https://datafusion.apache.org/python/): Python bindings — SQL and lazy DataFrame API over Apache Arrow.
+- [DataFusion Ballista](https://datafusion.apache.org/ballista/): distributed execution extension for DataFusion.
+- [DataFusion Comet](https://datafusion.apache.org/comet/): Apache Spark accelerator built on DataFusion.
+
+## Optional
+
+- [Blog](https://datafusion.apache.org/blog/): release notes and ecosystem updates.
+- [crates.io `datafusion`](https://crates.io/crates/datafusion): published crate.
+- [Code of conduct](https://github.com/apache/datafusion/blob/main/CODE_OF_CONDUCT.md)
+- [Apache Software Foundation](https://apache.org)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org