You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@datafusion.apache.org by "edmondop (via GitHub)" <gi...@apache.org> on 2026/08/21 22:10:48 UTC

[I] Proposal: DataFusion Cookbook [datafusion]

edmondop opened a new issue, #24569:
URL: https://github.com/apache/datafusion/issues/24569

   A **cookbook** is a collection of short, self-contained, task-framed recipes ("I want to do X → here's how, here's why, here's the gotchas") — as opposed to reference docs (API surface) or a linear tutorial (one path through the basics). It's a proven format once a technology has enough surface area that no single tutorial covers it well.
   
   **Why it'd help DataFusion**
   
   DataFusion is powerful enough that the hard part for new adopters isn't "does this feature exist" but "how do I wire it together for my use case" — custom `TableProvider`, which UDF/UDAF/UDWF/UDTF variant to use, pushdown, catalog integration, remote object stores. A cookbook is the standard way projects close that gap.
   
   **Examples of cookbooks elsewhere**
   
   Published books:
   - AWS Cookbook (O'Reilly) — https://www.amazon.com/AWS-Cookbook-Recipes-Success/dp/1492092606
   - Terraform Cookbook (O'Reilly) — https://www.amazon.com/Terraform-Cookbook-Kerim-Satirli/dp/1804613477
   - Django Web Development Cookbook (Packt, 4th ed.) — https://www.amazon.com/Django-Web-Development-Cookbook-development/dp/1838987428
   - Polars Cookbook (Packt) — https://www.amazon.com/Polars-Cookbook-practical-transform-manipulate/dp/1805121154
   
   Same-ecosystem, in-project precedents:
   - Rust Cookbook — https://github.com/rust-lang-nursery/rust-cookbook
   - ADBC Python Cookbook, embedded in the Arrow docs — https://arrow.apache.org/adbc/current/python/recipe/index.html
   
   **Relationship with examples**
   
   `datafusion-examples` (https://github.com/apache/datafusion/blob/main/datafusion-examples/README.md) already has ~50 runnable, tested examples across 11 groups: UDF/UDAF/UDWF/UDTF, custom `TableProvider`, custom file formats, Parquet encryption (incl. KMS), custom optimizer/analyzer rules, Flight SQL, querying S3, remote catalogs, and more.
   
   These examples can be turned into recipe for a cookbook by rewriting each as a short step-by-step recipe — problem statement, code, explanation, pitfalls.
   
   Probably 50 recipes are excessive, a good number would be in the 15-20, at least to start. 
   
   **Organization options**
   
   Existing cookbooks adopt both the colocated and the separate repo approach, so we would need to decide between these two options:
      - As a new section of the existing docs site, next to `datafusion-examples`, versioned with the code (closest precedent: ADBC's cookbook, which lives directly in the Sphinx docs).
      - A separate repo, lower barrier for outside contributors, easier to restructure freely, but risks drifting from releases (precedent: Rust Cookbook, Spice.ai's cookbook repo).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org