You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Harshil-1603 (via GitHub)" <gi...@apache.org> on 2026/10/14 11:59:44 UTC
[PR] docs: show runnable query examples in API documentation [datafusion]
Harshil-1603 opened a new pull request, #26291:
URL: https://github.com/apache/datafusion/pull/26291
## Which issue does this PR close?
Closes #16654.
## Rationale for this change
API readers currently need to follow manually maintained links to discover complete runnable examples. Enable rustdoc's example scraping so the API reference displays real call sites next to the methods they demonstrate.
## What changes are included in this PR?
- Enable example scraping for the published `datafusion` crate's docs.rs build.
- Add a self-contained SQL and DataFrame query example, opt it into scraping, and include its source in the published package.
- Document the local nightly build and the requirements for adding further scraped examples.
This establishes scraping for the core crate. The separate `datafusion-examples` catalogue is not moved or bundled into the published library.
## What is the testing strategy for this PR?
Passed locally on Rust 1.99.0, with nightly 1.101.0-nightly (2026-10-14) for rustdoc:
- `cargo fmt --all -- --check`
- `cargo clippy --locked -p datafusion --example query --all-features -- -D warnings`
- `cargo run --locked -p datafusion --example query` (checks the SQL and DataFrame results against expected rows)
- `cargo +nightly doc --locked -p datafusion --all-features --lib --no-deps -Z unstable-options -Z rustdoc-scrape-examples`, with `RUSTDOCFLAGS=-D warnings`
- `cargo package --locked -p datafusion --allow-dirty --no-verify` and inspection of the resulting archive to confirm the example and scraping configuration are included
- `./ci/scripts/doc_prettier_check.sh --write --allow-dirty`
- `git diff --check`
A normal nightly documentation build had no scraped example panels. With scraping enabled, the `SessionContext` and `DataFrame` pages contain source-linked panels from `query.rs`. The `SessionContext::sql` panel was inspected in a browser; no console errors were present (rustdoc produced font-preload warnings).
The required full `cargo clippy --locked --all-targets --all-features -- -D warnings` fails with three existing `clippy::mutable_key_type` errors in unchanged `datafusion/optimizer/src/push_down_filter.rs`, tracked by #26181. The focused example check passes. The full lint suite and extended tests have not passed.
## Are there any user-facing changes?
The API documentation now includes automatically extracted query examples and links to their source. No runtime behavior or public API changes.
Prepared with Codex assistance. Author and maintainer review remain necessary; hosted docs.rs behavior will be verified after publication.
@alamb Could you trigger CI for this first contribution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: show runnable query examples in API documentation [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #26291:
URL: https://github.com/apache/datafusion/pull/26291#discussion_r4299194200
##########
datafusion/core/examples/query.rs:
##########
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Query in-memory data with SQL and the DataFrame API.
+//!
+//! Run with `cargo run -p datafusion --example query`.
+//! This example is also included in the published crate so rustdoc can show
+//! its call sites alongside the API documentation.
+
+use std::sync::Arc;
+
+use datafusion::arrow::array::Int32Array;
+use datafusion::arrow::record_batch::RecordBatch;
+use datafusion::common::assert_batches_eq;
+use datafusion::error::Result;
+use datafusion::prelude::*;
+
+#[tokio::main(flavor = "current_thread")]
Review Comment:
There is a whole crate of examples here: https://github.com/apache/datafusion/tree/main/datafusion-examples
This one, for example, is pretty similar to a bunch of existing examples
- https://github.com/apache/datafusion/blob/0d79252f61bd8777990b2749c216ffd07079c30d/docs/source/library-user-guide/using-the-dataframe-api.md#L78-L112 (register_batch + sql + filter + collect + assert_batches_eq)
- https://github.com/apache/datafusion/blob/0d79252f61bd8777990b2749c216ffd07079c30d/datafusion-examples/examples/dataframe/dataframe.rs#L160-L178 (read_memory runnable example)
I realize we can't reuse datafusion-examples without moving them all to the core crate which is a more substantial change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: show runnable query examples in API documentation [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #26291:
URL: https://github.com/apache/datafusion/pull/26291#issuecomment-6224448142
Or maybe given the doc-scrape-examples feature, we should consider moving all the examples back into the core crate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: show runnable query examples in API documentation [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #26291:
URL: https://github.com/apache/datafusion/pull/26291#issuecomment-6169562792
Can you please post an annotated screen shot of what this change does?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: show runnable query examples in API documentation [datafusion]
Posted by "Harshil-1603 (via GitHub)" <gi...@apache.org>.
Harshil-1603 commented on PR #26291:
URL: https://github.com/apache/datafusion/pull/26291#issuecomment-6179470415
Here is the annotated screenshot of the locally generated API documentation for `SessionContext::sql`:
1. The new **Examples found in repository** panel displays the real call site from the runnable query example alongside the API documentation.
2. The source link opens `datafusion/core/examples/query.rs` at line 44, where `ctx.sql(...)` is called.
The blue callouts are review annotations, not part of the documentation styling shipped by this PR. The example code and source links are the original rustdoc output. Without example scraping enabled, this repository-example panel is absent.
<img width="1265" height="712" alt="DataFusion-annotated-example" src="https://github.com/user-attachments/assets/2ebcc4bd-3651-496d-9f92-7244239312ee" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org