You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by al...@apache.org on 2025/05/21 18:34:59 UTC
(datafusion) branch main updated: Improve the DML / DDL Documentation (#16115)
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/main by this push:
new dc45f60bb2 Improve the DML / DDL Documentation (#16115)
dc45f60bb2 is described below
commit dc45f60bb2e1125e2f589bd1302351869be9cad6
Author: Andrew Lamb <an...@nerdnetworks.org>
AuthorDate: Wed May 21 14:34:55 2025 -0400
Improve the DML / DDL Documentation (#16115)
* Update documentation about DDL and DML
* Improve the DML Documentation
* Apply suggestions from code review
Co-authored-by: Oleks V <co...@users.noreply.github.com>
* Fix docs
* Fix docs
---------
Co-authored-by: Oleks V <co...@users.noreply.github.com>
---
benchmarks/README.md | 4 +--
benchmarks/queries/clickbench/README.md | 2 +-
datafusion/expr/src/logical_plan/dml.rs | 31 ++++++++++++++++++++--
docs/source/library-user-guide/catalogs.md | 5 +++-
.../library-user-guide/custom-table-providers.md | 13 ++++++---
5 files changed, 45 insertions(+), 10 deletions(-)
diff --git a/benchmarks/README.md b/benchmarks/README.md
index cb6b276c41..b19b3385af 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -33,8 +33,8 @@ benchmarks that compare performance with other engines. For example:
- [ClickBench] scripts are in the [ClickBench repo](https://github.com/ClickHouse/ClickBench/tree/main/datafusion)
- [H2o.ai `db-benchmark`] scripts are in [db-benchmark](https://github.com/apache/datafusion/tree/main/benchmarks/src/h2o.rs)
-[ClickBench]: https://github.com/ClickHouse/ClickBench/tree/main
-[H2o.ai `db-benchmark`]: https://github.com/h2oai/db-benchmark
+[clickbench]: https://github.com/ClickHouse/ClickBench/tree/main
+[h2o.ai `db-benchmark`]: https://github.com/h2oai/db-benchmark
# Running the benchmarks
diff --git a/benchmarks/queries/clickbench/README.md b/benchmarks/queries/clickbench/README.md
index ad4ae8f60f..e5acd8f348 100644
--- a/benchmarks/queries/clickbench/README.md
+++ b/benchmarks/queries/clickbench/README.md
@@ -9,7 +9,7 @@ ClickBench is focused on aggregation and filtering performance (though it has no
- `queries.sql` - Actual ClickBench queries, downloaded from the [ClickBench repository]
- `extended.sql` - "Extended" DataFusion specific queries.
-[ClickBench repository]: https://github.com/ClickHouse/ClickBench/blob/main/datafusion/queries.sql
+[clickbench repository]: https://github.com/ClickHouse/ClickBench/blob/main/datafusion/queries.sql
## "Extended" Queries
diff --git a/datafusion/expr/src/logical_plan/dml.rs b/datafusion/expr/src/logical_plan/dml.rs
index d4d50ac4ea..f3c95e696b 100644
--- a/datafusion/expr/src/logical_plan/dml.rs
+++ b/datafusion/expr/src/logical_plan/dml.rs
@@ -89,8 +89,28 @@ impl Hash for CopyTo {
}
}
-/// The operator that modifies the content of a database (adapted from
-/// substrait WriteRel)
+/// Modifies the content of a database
+///
+/// This operator is used to perform DML operations such as INSERT, DELETE,
+/// UPDATE, and CTAS (CREATE TABLE AS SELECT).
+///
+/// * `INSERT` - Appends new rows to the existing table. Calls
+/// [`TableProvider::insert_into`]
+///
+/// * `DELETE` - Removes rows from the table. Currently NOT supported by the
+/// [`TableProvider`] trait or builtin sources.
+///
+/// * `UPDATE` - Modifies existing rows in the table. Currently NOT supported by
+/// the [`TableProvider`] trait or builtin sources.
+///
+/// * `CREATE TABLE AS SELECT` - Creates a new table and populates it with data
+/// from a query. This is similar to the `INSERT` operation, but it creates a new
+/// table instead of modifying an existing one.
+///
+/// Note that the structure is adapted from substrait WriteRel)
+///
+/// [`TableProvider`]: https://docs.rs/datafusion/latest/datafusion/datasource/trait.TableProvider.html
+/// [`TableProvider::insert_into`]: https://docs.rs/datafusion/latest/datafusion/datasource/trait.TableProvider.html#method.insert_into
#[derive(Clone)]
pub struct DmlStatement {
/// The table name
@@ -177,11 +197,18 @@ impl PartialOrd for DmlStatement {
}
}
+/// The type of DML operation to perform.
+///
+/// See [`DmlStatement`] for more details.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Hash)]
pub enum WriteOp {
+ /// `INSERT INTO` operation
Insert(InsertOp),
+ /// `DELETE` operation
Delete,
+ /// `UPDATE` operation
Update,
+ /// `CREATE TABLE AS SELECT` operation
Ctas,
}
diff --git a/docs/source/library-user-guide/catalogs.md b/docs/source/library-user-guide/catalogs.md
index 906039ba23..d4e6633d40 100644
--- a/docs/source/library-user-guide/catalogs.md
+++ b/docs/source/library-user-guide/catalogs.md
@@ -23,11 +23,14 @@ This section describes how to create and manage catalogs, schemas, and tables in
## General Concepts
-CatalogProviderList, Catalogs, schemas, and tables are organized in a hierarchy. A CatalogProviderList contains catalog providers, a catalog provider contains schemas and a schema contains tables.
+Catalog providers, catalogs, schemas, and tables are organized in a hierarchy. A `CatalogProviderList` contains `CatalogProvider`s, a `CatalogProvider` contains `SchemaProviders` and a `SchemaProvider` contains `TableProvider`s.
DataFusion comes with a basic in memory catalog functionality in the [`catalog` module]. You can use these in memory implementations as is, or extend DataFusion with your own catalog implementations, for example based on local files or files on remote object storage.
+DataFusion supports DDL queries (e.g. `CREATE TABLE`) using the catalog API described in this section. See the [TableProvider] section for information on DML queries (e.g. `INSERT INTO`).
+
[`catalog` module]: https://docs.rs/datafusion/latest/datafusion/catalog/index.html
+[tableprovider]: ./custom-table-providers.md
Similarly to other concepts in DataFusion, you'll implement various traits to create your own catalogs, schemas, and tables. The following sections describe the traits you'll need to implement.
diff --git a/docs/source/library-user-guide/custom-table-providers.md b/docs/source/library-user-guide/custom-table-providers.md
index 886ac96295..54f79a4218 100644
--- a/docs/source/library-user-guide/custom-table-providers.md
+++ b/docs/source/library-user-guide/custom-table-providers.md
@@ -19,17 +19,22 @@
# Custom Table Provider
-Like other areas of DataFusion, you extend DataFusion's functionality by implementing a trait. The `TableProvider` and associated traits, have methods that allow you to implement a custom table provider, i.e. use DataFusion's other functionality with your custom data source.
+Like other areas of DataFusion, you extend DataFusion's functionality by implementing a trait. The [`TableProvider`] and associated traits allow you to implement a custom table provider, i.e. use DataFusion's other functionality with your custom data source.
-This section will also touch on how to have DataFusion use the new `TableProvider` implementation.
+This section describes how to create a [`TableProvider`] and how to configure DataFusion to use it for reading.
## Table Provider and Scan
-The `scan` method on the `TableProvider` is likely its most important. It returns an `ExecutionPlan` that DataFusion will use to read the actual data during execution of the query.
+The [`TableProvider::scan`] method reads data from the table and is likely the most important. It returns an [`ExecutionPlan`] that DataFusion will use to read the actual data during execution of the query. The [`TableProvider::insert_into`] method is used to `INSERT` data into the table.
### Scan
-As mentioned, `scan` returns an execution plan, and in particular a `Result<Arc<dyn ExecutionPlan>>`. The core of this is returning something that can be dynamically dispatched to an `ExecutionPlan`. And as per the general DataFusion idea, we'll need to implement it.
+As mentioned, [`TableProvider::scan`] returns an execution plan, and in particular a `Result<Arc<dyn ExecutionPlan>>`. The core of this is returning something that can be dynamically dispatched to an `ExecutionPlan`. And as per the general DataFusion idea, we'll need to implement it.
+
+[`tableprovider`]: https://docs.rs/datafusion/latest/datafusion/datasource/trait.TableProvider.html
+[`tableprovider::scan`]: https://docs.rs/datafusion/latest/datafusion/datasource/trait.TableProvider.html#tymethod.scan
+[`tableprovider::insert_into`]: https://docs.rs/datafusion/latest/datafusion/datasource/trait.TableProvider.html#tymethod.insert_into
+[`executionplan`]: https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html
#### Execution Plan
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org