(datafusion) branch main updated: feat: Add robots.txt to site (#24333)

[gi...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion.git


The following commit(s) were added to refs/heads/main by this push:
     new bbf739a3cf feat: Add robots.txt to site (#24333)
bbf739a3cf is described below

commit bbf739a3cfe1d5cc7646e37250c947c6a56b5b0e
Author: Narendra <na...@gmail.com>
AuthorDate: Sun Aug 16 14:20:33 2026 +0000

    feat: Add robots.txt to site (#24333)
    
    ## Which issue does this PR close?
    
    - Closes #23258
    
    ## What changes are included in this PR?
    There are 2 PR's for this issue. This one contains the changes for
    _robots.txt_
    
    - added the robots.txt , added license header and allow rules.
    
    ## Are these changes tested?
    Yes. Tested by building the docs and verifying the files getting
    generated in the build/
    
    ## Are there any user-facing changes?
    No.
    
    ---------
    
    Co-authored-by: Jeffrey Vo <je...@gmail.com>
---
 dev/release/rat_exclude_files.txt | 1 +
 docs/source/conf.py               | 2 +-
 docs/source/robots.txt            | 4 ++++
 3 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/dev/release/rat_exclude_files.txt b/dev/release/rat_exclude_files.txt
index 77da7db87e..74e4279a6a 100644
--- a/dev/release/rat_exclude_files.txt
+++ b/dev/release/rat_exclude_files.txt
@@ -61,6 +61,7 @@ datafusion/proto-common/src/generated/prost.rs
 .github/ISSUE_TEMPLATE/feature_request.yml
 .github/workflows/docs.yaml
 docs/source/llms.txt
+docs/source/robots.txt
 **/node_modules/*
 datafusion/wasmtest/pkg/*
 clippy.toml
diff --git a/docs/source/conf.py b/docs/source/conf.py
index 81ffce3ad1..68ba628bba 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -112,7 +112,7 @@ html_static_path = ["_static"]
 
 # Copy agent-facing files (llms.txt) verbatim to the site root so they
 # resolve at the conventional URL `https://datafusion.apache.org/llms.txt`.
-html_extra_path = ["llms.txt"]
+html_extra_path = ["llms.txt", "robots.txt"]
 
 # Base URL of the published site. Required by sphinx-sitemap to build
 # absolute <loc> entries in the generated sitemap.xml.
diff --git a/docs/source/robots.txt b/docs/source/robots.txt
new file mode 100644
index 0000000000..9c80d8219c
--- /dev/null
+++ b/docs/source/robots.txt
@@ -0,0 +1,4 @@
+User-agent: *
+Allow: /
+
+Sitemap: https://datafusion.apache.org/sitemap.xml


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org