You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2025/07/02 12:13:54 UTC
(datafusion-python) branch asf-staging updated: Publish built docs triggered by 9362f53150e5423581757ed56883b3ca2c95b8a2
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/datafusion-python.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new d7a9e029 Publish built docs triggered by 9362f53150e5423581757ed56883b3ca2c95b8a2
d7a9e029 is described below
commit d7a9e029bf4644c8b1be007c763fe458d98c98a3
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Wed Jul 2 12:13:50 2025 +0000
Publish built docs triggered by 9362f53150e5423581757ed56883b3ca2c95b8a2
---
_sources/autoapi/datafusion/catalog/index.rst.txt | 195 +++++++++-
_sources/autoapi/datafusion/context/index.rst.txt | 26 ++
.../autoapi/datafusion/dataframe/index.rst.txt | 6 +-
_sources/autoapi/datafusion/index.rst.txt | 54 ++-
_sources/user-guide/data-sources.rst.txt | 56 +++
autoapi/datafusion/catalog/index.html | 421 +++++++++++++++++++--
autoapi/datafusion/context/index.html | 78 +++-
autoapi/datafusion/dataframe/index.html | 7 +-
autoapi/datafusion/index.html | 130 ++++---
genindex.html | 128 +++++--
objects.inv | Bin 8552 -> 8673 bytes
searchindex.js | 2 +-
user-guide/common-operations/aggregations.html | 32 +-
user-guide/common-operations/basic-info.html | 34 +-
user-guide/common-operations/functions.html | 20 +-
user-guide/data-sources.html | 129 +++++--
16 files changed, 1091 insertions(+), 227 deletions(-)
diff --git a/_sources/autoapi/datafusion/catalog/index.rst.txt b/_sources/autoapi/datafusion/catalog/index.rst.txt
index feccc9bb..9364aae2 100644
--- a/_sources/autoapi/datafusion/catalog/index.rst.txt
+++ b/_sources/autoapi/datafusion/catalog/index.rst.txt
@@ -15,14 +15,16 @@ Classes
.. autoapisummary::
datafusion.catalog.Catalog
- datafusion.catalog.Database
+ datafusion.catalog.CatalogProvider
+ datafusion.catalog.Schema
+ datafusion.catalog.SchemaProvider
datafusion.catalog.Table
Module Contents
---------------
-.. py:class:: Catalog(catalog: datafusion._internal.Catalog)
+.. py:class:: Catalog(catalog: datafusion._internal.catalog.RawCatalog)
DataFusion data catalog.
@@ -35,50 +37,211 @@ Module Contents
- .. py:method:: database(name: str = 'public') -> Database
+ .. py:method:: database(name: str = 'public') -> Schema
Returns the database with the given ``name`` from this catalog.
- .. py:method:: names() -> list[str]
+ .. py:method:: deregister_schema(name: str, cascade: bool = True) -> Schema | None
- Returns the list of databases in this catalog.
+ Deregister a schema from this catalog.
+
+
+
+ .. py:method:: memory_catalog() -> Catalog
+ :staticmethod:
+
+
+ Create an in-memory catalog provider.
+
+
+
+ .. py:method:: names() -> set[str]
+
+ This is an alias for `schema_names`.
+
+
+
+ .. py:method:: register_schema(name, schema) -> Schema | None
+
+ Register a schema with this catalog.
+
+
+
+ .. py:method:: schema(name: str = 'public') -> Schema
+
+ Returns the database with the given ``name`` from this catalog.
+
+
+
+ .. py:method:: schema_names() -> set[str]
+
+ Returns the list of schemas in this catalog.
.. py:attribute:: catalog
-.. py:class:: Database(db: datafusion._internal.Database)
+.. py:class:: CatalogProvider
+
+ Bases: :py:obj:`abc.ABC`
+
+
+ Abstract class for defining a Python based Catalog Provider.
+
- DataFusion Database.
+ .. py:method:: deregister_schema(name: str, cascade: bool) -> None
+
+ Remove a schema from this catalog.
+
+ This method is optional. If your catalog provides a fixed list of schemas, you
+ do not need to implement this method.
+
+ :param name: The name of the schema to remove.
+ :param cascade: If true, deregister the tables within the schema.
+
+
+
+ .. py:method:: register_schema(name: str, schema: SchemaProviderExportable | SchemaProvider | Schema) -> None
+
+ Add a schema to this catalog.
+
+ This method is optional. If your catalog provides a fixed list of schemas, you
+ do not need to implement this method.
+
+
+
+ .. py:method:: schema(name: str) -> Schema | None
+ :abstractmethod:
+
+
+ Retrieve a specific schema from this catalog.
+
+
+
+ .. py:method:: schema_names() -> set[str]
+ :abstractmethod:
+
+
+ Set of the names of all schemas in this catalog.
+
+
+
+.. py:class:: Schema(schema: datafusion._internal.catalog.RawSchema)
+
+ DataFusion Schema.
This constructor is not typically called by the end user.
.. py:method:: __repr__() -> str
- Print a string representation of the database.
+ Print a string representation of the schema.
+
+
+
+ .. py:method:: deregister_table(name: str) -> None
+
+ Deregister a table provider from this schema.
+
+
+
+ .. py:method:: memory_schema() -> Schema
+ :staticmethod:
+
+
+ Create an in-memory schema provider.
.. py:method:: names() -> set[str]
- Returns the list of all tables in this database.
+ This is an alias for `table_names`.
+
+
+
+ .. py:method:: register_table(name, table) -> None
+
+ Register a table provider in this schema.
.. py:method:: table(name: str) -> Table
- Return the table with the given ``name`` from this database.
+ Return the table with the given ``name`` from this schema.
+
+
+
+ .. py:method:: table_names() -> set[str]
+
+ Returns the list of all tables in this schema.
+
+
+
+ .. py:attribute:: _raw_schema
+
+
+.. py:class:: SchemaProvider
+
+ Bases: :py:obj:`abc.ABC`
+
+
+ Abstract class for defining a Python based Schema Provider.
+
+
+ .. py:method:: deregister_table(name, cascade: bool) -> None
+
+ Remove a table from this schema.
+
+ This method is optional. If your schema provides a fixed list of tables, you do
+ not need to implement this method.
+
+
+ .. py:method:: owner_name() -> str | None
+ Returns the owner of the schema.
- .. py:attribute:: db
+ This is an optional method. The default return is None.
-.. py:class:: Table(table: datafusion._internal.Table)
+
+ .. py:method:: register_table(name: str, table: Table) -> None
+
+ Add a table from this schema.
+
+ This method is optional. If your schema provides a fixed list of tables, you do
+ not need to implement this method.
+
+
+
+ .. py:method:: table(name: str) -> Table | None
+ :abstractmethod:
+
+
+ Retrieve a specific table from this schema.
+
+
+
+ .. py:method:: table_exist(name: str) -> bool
+ :abstractmethod:
+
+
+ Returns true if the table exists in this schema.
+
+
+
+ .. py:method:: table_names() -> set[str]
+ :abstractmethod:
+
+
+ Set of the names of all tables in this schema.
+
+
+
+.. py:class:: Table(table: datafusion._internal.catalog.RawTable)
DataFusion table.
@@ -91,6 +254,14 @@ Module Contents
+ .. py:method:: from_dataset(dataset: pyarrow.dataset.Dataset) -> Table
+ :staticmethod:
+
+
+ Turn a pyarrow Dataset into a Table.
+
+
+
.. py:property:: kind
:type: str
diff --git a/_sources/autoapi/datafusion/context/index.rst.txt b/_sources/autoapi/datafusion/context/index.rst.txt
index 0a653655..672346d6 100644
--- a/_sources/autoapi/datafusion/context/index.rst.txt
+++ b/_sources/autoapi/datafusion/context/index.rst.txt
@@ -16,6 +16,7 @@ Classes
datafusion.context.ArrowArrayExportable
datafusion.context.ArrowStreamExportable
+ datafusion.context.CatalogProviderExportable
datafusion.context.RuntimeConfig
datafusion.context.RuntimeEnvBuilder
datafusion.context.SQLOptions
@@ -53,6 +54,19 @@ Module Contents
.. py:method:: __arrow_c_stream__(requested_schema: object | None = None) -> object
+.. py:class:: CatalogProviderExportable
+
+ Bases: :py:obj:`Protocol`
+
+
+ Type hint for object that has __datafusion_catalog_provider__ PyCapsule.
+
+ https://docs.rs/datafusion/latest/datafusion/catalog/trait.CatalogProvider.html
+
+
+ .. py:method:: __datafusion_catalog_provider__() -> object
+
+
.. py:class:: RuntimeConfig
Bases: :py:obj:`RuntimeEnvBuilder`
@@ -428,6 +442,12 @@ Module Contents
+ .. py:method:: catalog_names() -> set[str]
+
+ Returns the list of catalogs in this context.
+
+
+
.. py:method:: create_dataframe(partitions: list[list[pyarrow.RecordBatch]], name: str | None = None, schema: pyarrow.Schema | None = None) -> datafusion.dataframe.DataFrame
Create and return a dataframe using the provided partitions.
@@ -654,6 +674,12 @@ Module Contents
+ .. py:method:: register_catalog_provider(name: str, provider: CatalogProviderExportable | datafusion.catalog.CatalogProvider | datafusion.catalog.Catalog) -> None
+
+ Register a catalog provider.
+
+
+
.. py:method:: register_csv(name: str, path: str | pathlib.Path | list[str | pathlib.Path], schema: pyarrow.Schema | None = None, has_header: bool = True, delimiter: str = ',', schema_infer_max_records: int = 1000, file_extension: str = '.csv', file_compression_type: str | None = None) -> None
Register a CSV file as a table.
diff --git a/_sources/autoapi/datafusion/dataframe/index.rst.txt b/_sources/autoapi/datafusion/dataframe/index.rst.txt
index e79be6cf..f0272cc7 100644
--- a/_sources/autoapi/datafusion/dataframe/index.rst.txt
+++ b/_sources/autoapi/datafusion/dataframe/index.rst.txt
@@ -282,17 +282,15 @@ Module Contents
- .. py:method:: explain(verbose: bool = False, analyze: bool = False) -> DataFrame
+ .. py:method:: explain(verbose: bool = False, analyze: bool = False) -> None
- Return a DataFrame with the explanation of its plan so far.
+ Print an explanation of the DataFrame's plan so far.
If ``analyze`` is specified, runs the plan and reports metrics.
:param verbose: If ``True``, more details will be included.
:param analyze: If ``Tru`e``, the plan will run and metrics reported.
- :returns: DataFrame with the explanation of its plan.
-
.. py:method:: fill_null(value: Any, subset: list[str] | None = None) -> DataFrame
diff --git a/_sources/autoapi/datafusion/index.rst.txt b/_sources/autoapi/datafusion/index.rst.txt
index 20e07f0c..a7157dd4 100644
--- a/_sources/autoapi/datafusion/index.rst.txt
+++ b/_sources/autoapi/datafusion/index.rst.txt
@@ -231,7 +231,7 @@ Package Contents
.. py:attribute:: _udaf
-.. py:class:: Catalog(catalog: datafusion._internal.Catalog)
+.. py:class:: Catalog(catalog: datafusion._internal.catalog.RawCatalog)
DataFusion data catalog.
@@ -244,47 +244,61 @@ Package Contents
- .. py:method:: database(name: str = 'public') -> Database
+ .. py:method:: database(name: str = 'public') -> Schema
Returns the database with the given ``name`` from this catalog.
- .. py:method:: names() -> list[str]
+ .. py:method:: deregister_schema(name: str, cascade: bool = True) -> Schema | None
- Returns the list of databases in this catalog.
+ Deregister a schema from this catalog.
- .. py:attribute:: catalog
+ .. py:method:: memory_catalog() -> Catalog
+ :staticmethod:
-.. py:class:: Database(db: datafusion._internal.Database)
+ Create an in-memory catalog provider.
- DataFusion Database.
- This constructor is not typically called by the end user.
+ .. py:method:: names() -> set[str]
- .. py:method:: __repr__() -> str
+ This is an alias for `schema_names`.
- Print a string representation of the database.
+ .. py:method:: register_schema(name, schema) -> Schema | None
+
+ Register a schema with this catalog.
+
+
+
+ .. py:method:: schema(name: str = 'public') -> Schema
+
+ Returns the database with the given ``name`` from this catalog.
- .. py:method:: names() -> set[str]
- Returns the list of all tables in this database.
+ .. py:method:: schema_names() -> set[str]
+ Returns the list of schemas in this catalog.
- .. py:method:: table(name: str) -> Table
- Return the table with the given ``name`` from this database.
+ .. py:attribute:: catalog
+
+
+.. py:class:: Database(schema: datafusion._internal.catalog.RawSchema)
+ Bases: :py:obj:`Schema`
- .. py:attribute:: db
+
+ See `Schema`.
+
+ This constructor is not typically called by the end user.
.. py:class:: ExecutionPlan(plan: datafusion._internal.ExecutionPlan)
@@ -2053,7 +2067,7 @@ Package Contents
.. py:attribute:: config_internal
-.. py:class:: Table(table: datafusion._internal.Table)
+.. py:class:: Table(table: datafusion._internal.catalog.RawTable)
DataFusion table.
@@ -2066,6 +2080,14 @@ Package Contents
+ .. py:method:: from_dataset(dataset: pyarrow.dataset.Dataset) -> Table
+ :staticmethod:
+
+
+ Turn a pyarrow Dataset into a Table.
+
+
+
.. py:property:: kind
:type: str
diff --git a/_sources/user-guide/data-sources.rst.txt b/_sources/user-guide/data-sources.rst.txt
index ba5967c9..9c95d58e 100644
--- a/_sources/user-guide/data-sources.rst.txt
+++ b/_sources/user-guide/data-sources.rst.txt
@@ -185,3 +185,59 @@ the interface as describe in the :ref:`Custom Table Provider <io_custom_table_pr
section. This is an advanced topic, but a
`user example <https://github.com/apache/datafusion-python/tree/main/examples/ffi-table-provider>`_
is provided in the DataFusion repository.
+
+Catalog
+=======
+
+A common technique for organizing tables is using a three level hierarchical approach. DataFusion
+supports this form of organizing using the :py:class:`~datafusion.catalog.Catalog`,
+:py:class:`~datafusion.catalog.Schema`, and :py:class:`~datafusion.catalog.Table`. By default,
+a :py:class:`~datafusion.context.SessionContext` comes with a single Catalog and a single Schema
+with the names ``datafusion`` and ``default``, respectively.
+
+The default implementation uses an in-memory approach to the catalog and schema. We have support
+for adding additional in-memory catalogs and schemas. This can be done like in the following
+example:
+
+.. code-block:: python
+
+ from datafusion.catalog import Catalog, Schema
+
+ my_catalog = Catalog.memory_catalog()
+ my_schema = Schema.memory_schema()
+
+ my_catalog.register_schema("my_schema_name", my_schema)
+
+ ctx.register_catalog("my_catalog_name", my_catalog)
+
+You could then register tables in ``my_schema`` and access them either through the DataFrame
+API or via sql commands such as ``"SELECT * from my_catalog_name.my_schema_name.my_table"``.
+
+User Defined Catalog and Schema
+-------------------------------
+
+If the in-memory catalogs are insufficient for your uses, there are two approaches you can take
+to implementing a custom catalog and/or schema. In the below discussion, we describe how to
+implement these for a Catalog, but the approach to implementing for a Schema is nearly
+identical.
+
+DataFusion supports Catalogs written in either Rust or Python. If you write a Catalog in Rust,
+you will need to export it as a Python library via PyO3. There is a complete example of a
+catalog implemented this way in the
+`examples folder <https://github.com/apache/datafusion-python/tree/main/examples/>`_
+of our repository. Writing catalog providers in Rust provides typically can lead to significant
+performance improvements over the Python based approach.
+
+To implement a Catalog in Python, you will need to inherit from the abstract base class
+:py:class:`~datafusion.catalog.CatalogProvider`. There are examples in the
+`unit tests <https://github.com/apache/datafusion-python/tree/main/python/tests>`_ of
+implementing a basic Catalog in Python where we simply keep a dictionary of the
+registered Schemas.
+
+One important note for developers is that when we have a Catalog defined in Python, we have
+two different ways of accessing this Catalog. First, we register the catalog with a Rust
+wrapper. This allows for any rust based code to call the Python functions as necessary.
+Second, if the user access the Catalog via the Python API, we identify this and return back
+the original Python object that implements the Catalog. This is an important distinction
+for developers because we do *not* return a Python wrapper around the Rust wrapper of the
+original Python object.
diff --git a/autoapi/datafusion/catalog/index.html b/autoapi/datafusion/catalog/index.html
index b52504f9..64d0e67c 100644
--- a/autoapi/datafusion/catalog/index.html
+++ b/autoapi/datafusion/catalog/index.html
@@ -444,6 +444,24 @@
</code>
</a>
</li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.Catalog.deregister_schema">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Catalog.deregister_schema()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.Catalog.memory_catalog">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Catalog.memory_catalog()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.catalog.Catalog.names">
<code class="docutils literal notranslate">
@@ -453,6 +471,33 @@
</code>
</a>
</li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.Catalog.register_schema">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Catalog.register_schema()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.Catalog.schema">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Catalog.schema()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.Catalog.schema_names">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Catalog.schema_names()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.catalog.Catalog.catalog">
<code class="docutils literal notranslate">
@@ -465,46 +510,194 @@
</ul>
</li>
<li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#datafusion.catalog.Database">
+ <a class="reference internal nav-link" href="#datafusion.catalog.CatalogProvider">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ CatalogProvider
+ </span>
+ </code>
+ </a>
+ <ul class="nav section-nav flex-column">
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.CatalogProvider.deregister_schema">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ CatalogProvider.deregister_schema()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.CatalogProvider.register_schema">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ CatalogProvider.register_schema()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.CatalogProvider.schema">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ CatalogProvider.schema()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.CatalogProvider.schema_names">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ CatalogProvider.schema_names()
+ </span>
+ </code>
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.Schema">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Schema
+ </span>
+ </code>
+ </a>
+ <ul class="nav section-nav flex-column">
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.Schema.__repr__">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Schema.__repr__()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.Schema.deregister_table">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Schema.deregister_table()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.Schema.memory_schema">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Schema.memory_schema()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.Schema.names">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Schema.names()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.Schema.register_table">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Schema.register_table()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.Schema.table">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Schema.table()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.Schema.table_names">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Schema.table_names()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.Schema._raw_schema">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Schema._raw_schema
+ </span>
+ </code>
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.SchemaProvider">
<code class="docutils literal notranslate">
<span class="pre">
- Database
+ SchemaProvider
</span>
</code>
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h4 nav-item toc-entry">
- <a class="reference internal nav-link" href="#datafusion.catalog.Database.__repr__">
+ <a class="reference internal nav-link" href="#datafusion.catalog.SchemaProvider.deregister_table">
<code class="docutils literal notranslate">
<span class="pre">
- Database.__repr__()
+ SchemaProvider.deregister_table()
</span>
</code>
</a>
</li>
<li class="toc-h4 nav-item toc-entry">
- <a class="reference internal nav-link" href="#datafusion.catalog.Database.names">
+ <a class="reference internal nav-link" href="#datafusion.catalog.SchemaProvider.owner_name">
<code class="docutils literal notranslate">
<span class="pre">
- Database.names()
+ SchemaProvider.owner_name()
</span>
</code>
</a>
</li>
<li class="toc-h4 nav-item toc-entry">
- <a class="reference internal nav-link" href="#datafusion.catalog.Database.table">
+ <a class="reference internal nav-link" href="#datafusion.catalog.SchemaProvider.register_table">
<code class="docutils literal notranslate">
<span class="pre">
- Database.table()
+ SchemaProvider.register_table()
</span>
</code>
</a>
</li>
<li class="toc-h4 nav-item toc-entry">
- <a class="reference internal nav-link" href="#datafusion.catalog.Database.db">
+ <a class="reference internal nav-link" href="#datafusion.catalog.SchemaProvider.table">
<code class="docutils literal notranslate">
<span class="pre">
- Database.db
+ SchemaProvider.table()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.SchemaProvider.table_exist">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ SchemaProvider.table_exist()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.SchemaProvider.table_names">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ SchemaProvider.table_names()
</span>
</code>
</a>
@@ -529,6 +722,15 @@
</code>
</a>
</li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.catalog.Table.from_dataset">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Table.from_dataset()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.catalog.Table.kind">
<code class="docutils literal notranslate">
@@ -591,8 +793,14 @@
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.catalog.Catalog" title="datafusion.catalog.Catalog"><code class="xref py py-obj docutils literal notranslate"><span class="pre">Catalog</span></code></a></p></td>
<td><p>DataFusion data catalog.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.catalog.Database" title="datafusion.catalog.Database"><code class="xref py py-obj docutils literal notranslate"><span class="pre">Database</span></code></a></p></td>
-<td><p>DataFusion Database.</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.catalog.CatalogProvider" title="datafusion.catalog.CatalogProvider"><code class="xref py py-obj docutils literal notranslate"><span class="pre">CatalogProvider</span></code></a></p></td>
+<td><p>Abstract class for defining a Python based Catalog Provider.</p></td>
+</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.catalog.Schema" title="datafusion.catalog.Schema"><code class="xref py py-obj docutils literal notranslate"><span class="pre">Schema</span></code></a></p></td>
+<td><p>DataFusion Schema.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.catalog.SchemaProvider" title="datafusion.catalog.SchemaProvider"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SchemaProvider</span></code></a></p></td>
+<td><p>Abstract class for defining a Python based Schema Provider.</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.catalog.Table" title="datafusion.catalog.Table"><code class="xref py py-obj docutils literal notranslate"><span class="pre">Table</span></code></a></p></td>
<td><p>DataFusion table.</p></td>
@@ -604,7 +812,7 @@
<h2>Module Contents<a class="headerlink" href="#module-contents" title="Link to this heading">¶</a></h2>
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.catalog.Catalog">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.catalog.</span></span><span class="sig-name descname"><span class="pre">Catalog</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">catalog</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.Catalog</span [...]
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.catalog.</span></span><span class="sig-name descname"><span class="pre">Catalog</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">catalog</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.catalog.RawCa [...]
<dd><p>DataFusion data catalog.</p>
<p>This constructor is not typically called by the end user.</p>
<dl class="py method">
@@ -615,14 +823,44 @@
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.catalog.Catalog.database">
-<span class="sig-name descname"><span class="pre">database</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">'public'</span></span></em><span class="sig-par [...]
+<span class="sig-name descname"><span class="pre">database</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">'public'</span></span></em><span class="sig-par [...]
<dd><p>Returns the database with the given <code class="docutils literal notranslate"><span class="pre">name</span></code> from this catalog.</p>
</dd></dl>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.Catalog.deregister_schema">
+<span class="sig-name descname"><span class="pre">deregister_schema</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">cascade</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span cl [...]
+<dd><p>Deregister a schema from this catalog.</p>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.Catalog.memory_catalog">
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">memory_catalog</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#datafusion.catalog.Catalog" title="datafusion.catalog.Catalog"><span class="pre">Catalog</span></a></span></span><a class="hea [...]
+<dd><p>Create an in-memory catalog provider.</p>
+</dd></dl>
+
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.catalog.Catalog.names">
-<span class="sig-name descname"><span class="pre">names</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#datafusion.catalog.Catalog.names" title="Link to this d [...]
-<dd><p>Returns the list of databases in this catalog.</p>
+<span class="sig-name descname"><span class="pre">names</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">set</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#datafusion.catalog.Catalog.names" title="Link to this de [...]
+<dd><p>This is an alias for <cite>schema_names</cite>.</p>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.Catalog.register_schema">
+<span class="sig-name descname"><span class="pre">register_schema</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">schema</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#datafusion.catalog.Schema" title="datafusion [...]
+<dd><p>Register a schema with this catalog.</p>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.Catalog.schema">
+<span class="sig-name descname"><span class="pre">schema</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">'public'</span></span></em><span class="sig-paren [...]
+<dd><p>Returns the database with the given <code class="docutils literal notranslate"><span class="pre">name</span></code> from this catalog.</p>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.Catalog.schema_names">
+<span class="sig-name descname"><span class="pre">schema_names</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">set</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#datafusion.catalog.Catalog.schema_names" title="L [...]
+<dd><p>Returns the list of schemas in this catalog.</p>
</dd></dl>
<dl class="py attribute">
@@ -633,38 +871,153 @@
</dd></dl>
<dl class="py class">
-<dt class="sig sig-object py" id="datafusion.catalog.Database">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.catalog.</span></span><span class="sig-name descname"><span class="pre">Database</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">db</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.Database</span></ [...]
-<dd><p>DataFusion Database.</p>
+<dt class="sig sig-object py" id="datafusion.catalog.CatalogProvider">
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.catalog.</span></span><span class="sig-name descname"><span class="pre">CatalogProvider</span></span><a class="headerlink" href="#datafusion.catalog.CatalogProvider" title="Link to this definition">¶</a></dt>
+<dd><p>Bases: <code class="xref py py-obj docutils literal notranslate"><span class="pre">abc.ABC</span></code></p>
+<p>Abstract class for defining a Python based Catalog Provider.</p>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.CatalogProvider.deregister_schema">
+<span class="sig-name descname"><span class="pre">deregister_schema</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">cascade</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span cl [...]
+<dd><p>Remove a schema from this catalog.</p>
+<p>This method is optional. If your catalog provides a fixed list of schemas, you
+do not need to implement this method.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters<span class="colon">:</span></dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>name</strong> – The name of the schema to remove.</p></li>
+<li><p><strong>cascade</strong> – If true, deregister the tables within the schema.</p></li>
+</ul>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.CatalogProvider.register_schema">
+<span class="sig-name descname"><span class="pre">register_schema</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">schema</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class [...]
+<dd><p>Add a schema to this catalog.</p>
+<p>This method is optional. If your catalog provides a fixed list of schemas, you
+do not need to implement this method.</p>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.CatalogProvider.schema">
+<em class="property"><span class="pre">abstract</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">schema</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> [...]
+<dd><p>Retrieve a specific schema from this catalog.</p>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.CatalogProvider.schema_names">
+<em class="property"><span class="pre">abstract</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">schema_names</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">set</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></ [...]
+<dd><p>Set of the names of all schemas in this catalog.</p>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="py class">
+<dt class="sig sig-object py" id="datafusion.catalog.Schema">
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.catalog.</span></span><span class="sig-name descname"><span class="pre">Schema</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">schema</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.catalog.RawSche [...]
+<dd><p>DataFusion Schema.</p>
<p>This constructor is not typically called by the end user.</p>
<dl class="py method">
-<dt class="sig sig-object py" id="datafusion.catalog.Database.__repr__">
-<span class="sig-name descname"><span class="pre">__repr__</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">str</span></span></span><a class="headerlink" href="#datafusion.catalog.Database.__repr__" title="Link to this definition">¶</a></dt>
-<dd><p>Print a string representation of the database.</p>
+<dt class="sig sig-object py" id="datafusion.catalog.Schema.__repr__">
+<span class="sig-name descname"><span class="pre">__repr__</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">str</span></span></span><a class="headerlink" href="#datafusion.catalog.Schema.__repr__" title="Link to this definition">¶</a></dt>
+<dd><p>Print a string representation of the schema.</p>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.Schema.deregister_table">
+<span class="sig-name descname"><span class="pre">deregister_table</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">None</span></span></s [...]
+<dd><p>Deregister a table provider from this schema.</p>
</dd></dl>
<dl class="py method">
-<dt class="sig sig-object py" id="datafusion.catalog.Database.names">
-<span class="sig-name descname"><span class="pre">names</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">set</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#datafusion.catalog.Database.names" title="Link to this d [...]
-<dd><p>Returns the list of all tables in this database.</p>
+<dt class="sig sig-object py" id="datafusion.catalog.Schema.memory_schema">
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">memory_schema</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#datafusion.catalog.Schema" title="datafusion.catalog.Schema"><span class="pre">Schema</span></a></span></span><a class="headerl [...]
+<dd><p>Create an in-memory schema provider.</p>
</dd></dl>
<dl class="py method">
-<dt class="sig sig-object py" id="datafusion.catalog.Database.table">
-<span class="sig-name descname"><span class="pre">table</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#datafusion.ca [...]
-<dd><p>Return the table with the given <code class="docutils literal notranslate"><span class="pre">name</span></code> from this database.</p>
+<dt class="sig sig-object py" id="datafusion.catalog.Schema.names">
+<span class="sig-name descname"><span class="pre">names</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">set</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#datafusion.catalog.Schema.names" title="Link to this def [...]
+<dd><p>This is an alias for <cite>table_names</cite>.</p>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.Schema.register_table">
+<span class="sig-name descname"><span class="pre">register_table</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">table</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">None</span></span></span><a class="headerlink" href="#datafusion. [...]
+<dd><p>Register a table provider in this schema.</p>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.Schema.table">
+<span class="sig-name descname"><span class="pre">table</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#datafusion.ca [...]
+<dd><p>Return the table with the given <code class="docutils literal notranslate"><span class="pre">name</span></code> from this schema.</p>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.Schema.table_names">
+<span class="sig-name descname"><span class="pre">table_names</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">set</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#datafusion.catalog.Schema.table_names" title="Link [...]
+<dd><p>Returns the list of all tables in this schema.</p>
</dd></dl>
<dl class="py attribute">
-<dt class="sig sig-object py" id="datafusion.catalog.Database.db">
-<span class="sig-name descname"><span class="pre">db</span></span><a class="headerlink" href="#datafusion.catalog.Database.db" title="Link to this definition">¶</a></dt>
+<dt class="sig sig-object py" id="datafusion.catalog.Schema._raw_schema">
+<span class="sig-name descname"><span class="pre">_raw_schema</span></span><a class="headerlink" href="#datafusion.catalog.Schema._raw_schema" title="Link to this definition">¶</a></dt>
<dd></dd></dl>
</dd></dl>
+<dl class="py class">
+<dt class="sig sig-object py" id="datafusion.catalog.SchemaProvider">
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.catalog.</span></span><span class="sig-name descname"><span class="pre">SchemaProvider</span></span><a class="headerlink" href="#datafusion.catalog.SchemaProvider" title="Link to this definition">¶</a></dt>
+<dd><p>Bases: <code class="xref py py-obj docutils literal notranslate"><span class="pre">abc.ABC</span></code></p>
+<p>Abstract class for defining a Python based Schema Provider.</p>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.SchemaProvider.deregister_table">
+<span class="sig-name descname"><span class="pre">deregister_table</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">cascade</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon"> [...]
+<dd><p>Remove a table from this schema.</p>
+<p>This method is optional. If your schema provides a fixed list of tables, you do
+not need to implement this method.</p>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.SchemaProvider.owner_name">
+<span class="sig-name descname"><span class="pre">owner_name</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">str</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span></span><a class="headerlink" href="#datafusion.catalog.SchemaProvider.owner_name" title [...]
+<dd><p>Returns the owner of the schema.</p>
+<p>This is an optional method. The default return is None.</p>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.SchemaProvider.register_table">
+<span class="sig-name descname"><span class="pre">register_table</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">table</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="ref [...]
+<dd><p>Add a table from this schema.</p>
+<p>This method is optional. If your schema provides a fixed list of tables, you do
+not need to implement this method.</p>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.SchemaProvider.table">
+<em class="property"><span class="pre">abstract</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">table</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> < [...]
+<dd><p>Retrieve a specific table from this schema.</p>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.SchemaProvider.table_exist">
+<em class="property"><span class="pre">abstract</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">table_exist</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</s [...]
+<dd><p>Returns true if the table exists in this schema.</p>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.SchemaProvider.table_names">
+<em class="property"><span class="pre">abstract</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">table_names</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">set</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></s [...]
+<dd><p>Set of the names of all tables in this schema.</p>
+</dd></dl>
+
+</dd></dl>
+
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.catalog.Table">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.catalog.</span></span><span class="sig-name descname"><span class="pre">Table</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">table</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.Table</span></spa [...]
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.catalog.</span></span><span class="sig-name descname"><span class="pre">Table</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">table</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.catalog.RawTable< [...]
<dd><p>DataFusion table.</p>
<p>This constructor is not typically called by the end user.</p>
<dl class="py method">
@@ -673,6 +1026,12 @@
<dd><p>Print a string representation of the table.</p>
</dd></dl>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.Table.from_dataset">
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_dataset</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">dataset</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">pyarrow.dataset.Dataset</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-re [...]
+<dd><p>Turn a pyarrow Dataset into a Table.</p>
+</dd></dl>
+
<dl class="py property">
<dt class="sig sig-object py" id="datafusion.catalog.Table.kind">
<em class="property"><span class="pre">property</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">kind</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">str</span></em><a class="headerlink" href="#datafusion.catalog.Table.kind" title="Link to this definition">¶</a></dt>
diff --git a/autoapi/datafusion/context/index.html b/autoapi/datafusion/context/index.html
index 738b79dd..5da05964 100644
--- a/autoapi/datafusion/context/index.html
+++ b/autoapi/datafusion/context/index.html
@@ -457,6 +457,26 @@
</li>
</ul>
</li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.context.CatalogProviderExportable">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ CatalogProviderExportable
+ </span>
+ </code>
+ </a>
+ <ul class="nav section-nav flex-column">
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.context.CatalogProviderExportable.__datafusion_catalog_provider__">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ CatalogProviderExportable.__datafusion_catalog_provider__()
+ </span>
+ </code>
+ </a>
+ </li>
+ </ul>
+ </li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.context.RuntimeConfig">
<code class="docutils literal notranslate">
@@ -769,6 +789,15 @@
</code>
</a>
</li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.context.SessionContext.catalog_names">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ SessionContext.catalog_names()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.context.SessionContext.create_dataframe">
<code class="docutils literal notranslate">
@@ -940,6 +969,15 @@
</code>
</a>
</li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.context.SessionContext.register_catalog_provider">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ SessionContext.register_catalog_provider()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.context.SessionContext.register_csv">
<code class="docutils literal notranslate">
@@ -1178,22 +1216,25 @@
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.context.ArrowStreamExportable" title="datafusion.context.ArrowStreamExportable"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ArrowStreamExportable</span></code></a></p></td>
<td><p>Type hint for object exporting Arrow C Stream via Arrow PyCapsule Interface.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.context.RuntimeConfig" title="datafusion.context.RuntimeConfig"><code class="xref py py-obj docutils literal notranslate"><span class="pre">RuntimeConfig</span></code></a></p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.context.CatalogProviderExportable" title="datafusion.context.CatalogProviderExportable"><code class="xref py py-obj docutils literal notranslate"><span class="pre">CatalogProviderExportable</span></code></a></p></td>
+<td><p>Type hint for object that has __datafusion_catalog_provider__ PyCapsule.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.context.RuntimeConfig" title="datafusion.context.RuntimeConfig"><code class="xref py py-obj docutils literal notranslate"><span class="pre">RuntimeConfig</span></code></a></p></td>
<td><p>See <cite>RuntimeEnvBuilder</cite>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.context.RuntimeEnvBuilder" title="datafusion.context.RuntimeEnvBuilder"><code class="xref py py-obj docutils literal notranslate"><span class="pre">RuntimeEnvBuilder</span></code></a></p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.context.RuntimeEnvBuilder" title="datafusion.context.RuntimeEnvBuilder"><code class="xref py py-obj docutils literal notranslate"><span class="pre">RuntimeEnvBuilder</span></code></a></p></td>
<td><p>Runtime configuration options.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.context.SQLOptions" title="datafusion.context.SQLOptions"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SQLOptions</span></code></a></p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.context.SQLOptions" title="datafusion.context.SQLOptions"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SQLOptions</span></code></a></p></td>
<td><p>Options to be used when performing SQL queries.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.context.SessionConfig" title="datafusion.context.SessionConfig"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SessionConfig</span></code></a></p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.context.SessionConfig" title="datafusion.context.SessionConfig"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SessionConfig</span></code></a></p></td>
<td><p>Session configuration options.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.context.SessionContext" title="datafusion.context.SessionContext"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SessionContext</span></code></a></p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.context.SessionContext" title="datafusion.context.SessionContext"><code class="xref py py-obj docutils literal notranslate"><span class="pre">SessionContext</span></code></a></p></td>
<td><p>This is the main interface for executing queries and creating DataFrames.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.context.TableProviderExportable" title="datafusion.context.TableProviderExportable"><code class="xref py py-obj docutils literal notranslate"><span class="pre">TableProviderExportable</span></code></a></p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.context.TableProviderExportable" title="datafusion.context.TableProviderExportable"><code class="xref py py-obj docutils literal notranslate"><span class="pre">TableProviderExportable</span></code></a></p></td>
<td><p>Type hint for object that has __datafusion_table_provider__ PyCapsule.</p></td>
</tr>
</tbody>
@@ -1227,6 +1268,19 @@
</dd></dl>
+<dl class="py class">
+<dt class="sig sig-object py" id="datafusion.context.CatalogProviderExportable">
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.context.</span></span><span class="sig-name descname"><span class="pre">CatalogProviderExportable</span></span><a class="headerlink" href="#datafusion.context.CatalogProviderExportable" title="Link to this definition">¶</a></dt>
+<dd><p>Bases: <code class="xref py py-obj docutils literal notranslate"><span class="pre">Protocol</span></code></p>
+<p>Type hint for object that has __datafusion_catalog_provider__ PyCapsule.</p>
+<p><a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/catalog/trait.CatalogProvider.html">https://docs.rs/datafusion/latest/datafusion/catalog/trait.CatalogProvider.html</a></p>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.context.CatalogProviderExportable.__datafusion_catalog_provider__">
+<span class="sig-name descname"><span class="pre">__datafusion_catalog_provider__</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">object</span></span></span><a class="headerlink" href="#datafusion.context.CatalogProviderExportable.__datafusion_catalog_provider__" title="Link to this definition">¶</a></dt>
+<dd></dd></dl>
+
+</dd></dl>
+
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.context.RuntimeConfig">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.context.</span></span><span class="sig-name descname"><span class="pre">RuntimeConfig</span></span><a class="headerlink" href="#datafusion.context.RuntimeConfig" title="Link to this definition">¶</a></dt>
@@ -1691,6 +1745,12 @@ a query against a CSV data source using the <code class="xref py py-class docuti
<dd><p>Retrieve a catalog by name.</p>
</dd></dl>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.context.SessionContext.catalog_names">
+<span class="sig-name descname"><span class="pre">catalog_names</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">set</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#datafusion.context.SessionContext.catalog_names" [...]
+<dd><p>Returns the list of catalogs in this context.</p>
+</dd></dl>
+
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.context.SessionContext.create_dataframe">
<span class="sig-name descname"><span class="pre">create_dataframe</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">partitions</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">pyarrow.RecordBatch</span><span class [...]
@@ -1985,6 +2045,12 @@ this context.</p>
</dl>
</dd></dl>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.context.SessionContext.register_catalog_provider">
+<span class="sig-name descname"><span class="pre">register_catalog_provider</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">provider</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n" [...]
+<dd><p>Register a catalog provider.</p>
+</dd></dl>
+
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.context.SessionContext.register_csv">
<span class="sig-name descname"><span class="pre">register_csv</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">path</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre [...]
diff --git a/autoapi/datafusion/dataframe/index.html b/autoapi/datafusion/dataframe/index.html
index 8d4d27d5..965fe819 100644
--- a/autoapi/datafusion/dataframe/index.html
+++ b/autoapi/datafusion/dataframe/index.html
@@ -1880,8 +1880,8 @@ for non-numeric datatypes.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.explain">
-<span class="sig-name descname"><span class="pre">explain</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">verbose</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-par [...]
-<dd><p>Return a DataFrame with the explanation of its plan so far.</p>
+<span class="sig-name descname"><span class="pre">explain</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">verbose</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-par [...]
+<dd><p>Print an explanation of the DataFrame’s plan so far.</p>
<p>If <code class="docutils literal notranslate"><span class="pre">analyze</span></code> is specified, runs the plan and reports metrics.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -1890,9 +1890,6 @@ for non-numeric datatypes.</p>
<li><p><strong>analyze</strong> – If <code class="docutils literal notranslate"><span class="pre">Tru`e</span></code>, the plan will run and metrics reported.</p></li>
</ul>
</dd>
-<dt class="field-even">Returns<span class="colon">:</span></dt>
-<dd class="field-even"><p>DataFrame with the explanation of its plan.</p>
-</dd>
</dl>
</dd></dl>
diff --git a/autoapi/datafusion/index.html b/autoapi/datafusion/index.html
index 941aa543..fefb9ac9 100644
--- a/autoapi/datafusion/index.html
+++ b/autoapi/datafusion/index.html
@@ -554,72 +554,79 @@
</a>
</li>
<li class="toc-h4 nav-item toc-entry">
- <a class="reference internal nav-link" href="#datafusion.Catalog.names">
+ <a class="reference internal nav-link" href="#datafusion.Catalog.deregister_schema">
<code class="docutils literal notranslate">
<span class="pre">
- Catalog.names()
+ Catalog.deregister_schema()
</span>
</code>
</a>
</li>
<li class="toc-h4 nav-item toc-entry">
- <a class="reference internal nav-link" href="#datafusion.Catalog.catalog">
+ <a class="reference internal nav-link" href="#datafusion.Catalog.memory_catalog">
<code class="docutils literal notranslate">
<span class="pre">
- Catalog.catalog
+ Catalog.memory_catalog()
</span>
</code>
</a>
</li>
- </ul>
- </li>
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#datafusion.Database">
- <code class="docutils literal notranslate">
- <span class="pre">
- Database
- </span>
- </code>
- </a>
- <ul class="nav section-nav flex-column">
<li class="toc-h4 nav-item toc-entry">
- <a class="reference internal nav-link" href="#datafusion.Database.__repr__">
+ <a class="reference internal nav-link" href="#datafusion.Catalog.names">
<code class="docutils literal notranslate">
<span class="pre">
- Database.__repr__()
+ Catalog.names()
</span>
</code>
</a>
</li>
<li class="toc-h4 nav-item toc-entry">
- <a class="reference internal nav-link" href="#datafusion.Database.names">
+ <a class="reference internal nav-link" href="#datafusion.Catalog.register_schema">
<code class="docutils literal notranslate">
<span class="pre">
- Database.names()
+ Catalog.register_schema()
</span>
</code>
</a>
</li>
<li class="toc-h4 nav-item toc-entry">
- <a class="reference internal nav-link" href="#datafusion.Database.table">
+ <a class="reference internal nav-link" href="#datafusion.Catalog.schema">
<code class="docutils literal notranslate">
<span class="pre">
- Database.table()
+ Catalog.schema()
</span>
</code>
</a>
</li>
<li class="toc-h4 nav-item toc-entry">
- <a class="reference internal nav-link" href="#datafusion.Database.db">
+ <a class="reference internal nav-link" href="#datafusion.Catalog.schema_names">
<code class="docutils literal notranslate">
<span class="pre">
- Database.db
+ Catalog.schema_names()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.Catalog.catalog">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Catalog.catalog
</span>
</code>
</a>
</li>
</ul>
</li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.Database">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Database
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.ExecutionPlan">
<code class="docutils literal notranslate">
@@ -2847,6 +2854,15 @@
</code>
</a>
</li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.Table.from_dataset">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ Table.from_dataset()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.Table.kind">
<code class="docutils literal notranslate">
@@ -3298,7 +3314,7 @@ See <a class="reference external" href="https://datafusion.apache.org/python">ht
<td><p>DataFusion data catalog.</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.Database" title="datafusion.Database"><code class="xref py py-obj docutils literal notranslate"><span class="pre">Database</span></code></a></p></td>
-<td><p>DataFusion Database.</p></td>
+<td><p>See <cite>Schema</cite>.</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.ExecutionPlan" title="datafusion.ExecutionPlan"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ExecutionPlan</span></code></a></p></td>
<td><p>Represent nodes in the DataFusion Physical Plan.</p></td>
@@ -3517,7 +3533,7 @@ aggregation or window function calls.</p>
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.Catalog">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">Catalog</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">catalog</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.Catalog</span></span> [...]
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">Catalog</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">catalog</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.catalog.RawCatalog</s [...]
<dd><p>DataFusion data catalog.</p>
<p>This constructor is not typically called by the end user.</p>
<dl class="py method">
@@ -3528,53 +3544,61 @@ aggregation or window function calls.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.Catalog.database">
-<span class="sig-name descname"><span class="pre">database</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">'public'</span></span></em><span class="sig-par [...]
+<span class="sig-name descname"><span class="pre">database</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">'public'</span></span></em><span class="sig-par [...]
<dd><p>Returns the database with the given <code class="docutils literal notranslate"><span class="pre">name</span></code> from this catalog.</p>
</dd></dl>
<dl class="py method">
-<dt class="sig sig-object py" id="datafusion.Catalog.names">
-<span class="sig-name descname"><span class="pre">names</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#datafusion.Catalog.names" title="Link to this definitio [...]
-<dd><p>Returns the list of databases in this catalog.</p>
+<dt class="sig sig-object py" id="datafusion.Catalog.deregister_schema">
+<span class="sig-name descname"><span class="pre">deregister_schema</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">cascade</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span cl [...]
+<dd><p>Deregister a schema from this catalog.</p>
</dd></dl>
-<dl class="py attribute">
-<dt class="sig sig-object py" id="datafusion.Catalog.catalog">
-<span class="sig-name descname"><span class="pre">catalog</span></span><a class="headerlink" href="#datafusion.Catalog.catalog" title="Link to this definition">¶</a></dt>
-<dd></dd></dl>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.Catalog.memory_catalog">
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">memory_catalog</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#datafusion.Catalog" title="datafusion.Catalog"><span class="pre">Catalog</span></a></span></span><a class="headerlink" href="# [...]
+<dd><p>Create an in-memory catalog provider.</p>
+</dd></dl>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.Catalog.names">
+<span class="sig-name descname"><span class="pre">names</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">set</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#datafusion.Catalog.names" title="Link to this definition [...]
+<dd><p>This is an alias for <cite>schema_names</cite>.</p>
</dd></dl>
-<dl class="py class">
-<dt class="sig sig-object py" id="datafusion.Database">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">Database</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">db</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.Database</span></span></e [...]
-<dd><p>DataFusion Database.</p>
-<p>This constructor is not typically called by the end user.</p>
<dl class="py method">
-<dt class="sig sig-object py" id="datafusion.Database.__repr__">
-<span class="sig-name descname"><span class="pre">__repr__</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">str</span></span></span><a class="headerlink" href="#datafusion.Database.__repr__" title="Link to this definition">¶</a></dt>
-<dd><p>Print a string representation of the database.</p>
+<dt class="sig sig-object py" id="datafusion.Catalog.register_schema">
+<span class="sig-name descname"><span class="pre">register_schema</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">schema</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="catalog/index.html#datafusion.catalog.Schema" [...]
+<dd><p>Register a schema with this catalog.</p>
</dd></dl>
<dl class="py method">
-<dt class="sig sig-object py" id="datafusion.Database.names">
-<span class="sig-name descname"><span class="pre">names</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">set</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#datafusion.Database.names" title="Link to this definitio [...]
-<dd><p>Returns the list of all tables in this database.</p>
+<dt class="sig sig-object py" id="datafusion.Catalog.schema">
+<span class="sig-name descname"><span class="pre">schema</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">'public'</span></span></em><span class="sig-paren [...]
+<dd><p>Returns the database with the given <code class="docutils literal notranslate"><span class="pre">name</span></code> from this catalog.</p>
</dd></dl>
<dl class="py method">
-<dt class="sig sig-object py" id="datafusion.Database.table">
-<span class="sig-name descname"><span class="pre">table</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#datafusion.Ta [...]
-<dd><p>Return the table with the given <code class="docutils literal notranslate"><span class="pre">name</span></code> from this database.</p>
+<dt class="sig sig-object py" id="datafusion.Catalog.schema_names">
+<span class="sig-name descname"><span class="pre">schema_names</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">set</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span></span><a class="headerlink" href="#datafusion.Catalog.schema_names" title="Link to t [...]
+<dd><p>Returns the list of schemas in this catalog.</p>
</dd></dl>
<dl class="py attribute">
-<dt class="sig sig-object py" id="datafusion.Database.db">
-<span class="sig-name descname"><span class="pre">db</span></span><a class="headerlink" href="#datafusion.Database.db" title="Link to this definition">¶</a></dt>
+<dt class="sig sig-object py" id="datafusion.Catalog.catalog">
+<span class="sig-name descname"><span class="pre">catalog</span></span><a class="headerlink" href="#datafusion.Catalog.catalog" title="Link to this definition">¶</a></dt>
<dd></dd></dl>
</dd></dl>
+<dl class="py class">
+<dt class="sig sig-object py" id="datafusion.Database">
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">Database</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">schema</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.catalog.RawSchema</sp [...]
+<dd><p>Bases: <code class="xref py py-obj docutils literal notranslate"><span class="pre">Schema</span></code></p>
+<p>See <cite>Schema</cite>.</p>
+<p>This constructor is not typically called by the end user.</p>
+</dd></dl>
+
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.ExecutionPlan">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">ExecutionPlan</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">plan</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.ExecutionPlan</spa [...]
@@ -5488,7 +5512,7 @@ automatically created.</p>
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.Table">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">Table</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">table</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.Table</span></span></em>< [...]
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">Table</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">table</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.catalog.RawTable</span></ [...]
<dd><p>DataFusion table.</p>
<p>This constructor is not typically called by the end user.</p>
<dl class="py method">
@@ -5497,6 +5521,12 @@ automatically created.</p>
<dd><p>Print a string representation of the table.</p>
</dd></dl>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.Table.from_dataset">
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_dataset</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">dataset</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">pyarrow.dataset.Dataset</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-re [...]
+<dd><p>Turn a pyarrow Dataset into a Table.</p>
+</dd></dl>
+
<dl class="py property">
<dt class="sig sig-object py" id="datafusion.Table.kind">
<em class="property"><span class="pre">property</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">kind</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">str</span></em><a class="headerlink" href="#datafusion.Table.kind" title="Link to this definition">¶</a></dt>
diff --git a/genindex.html b/genindex.html
index bf0462f7..b8dc426e 100644
--- a/genindex.html
+++ b/genindex.html
@@ -491,6 +491,8 @@
<li><a href="autoapi/datafusion/index.html#datafusion.WindowUDF.__call__">(datafusion.WindowUDF method)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/context/index.html#datafusion.context.CatalogProviderExportable.__datafusion_catalog_provider__">__datafusion_catalog_provider__() (datafusion.context.CatalogProviderExportable method)</a>
+</li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.TableProviderExportable.__datafusion_table_provider__">__datafusion_table_provider__() (datafusion.context.TableProviderExportable method)</a>
</li>
<li><a href="autoapi/datafusion/index.html#datafusion.Expr.__eq__">__eq__() (datafusion.Expr method)</a>
@@ -592,13 +594,11 @@
</li>
<li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Catalog.__repr__">(datafusion.catalog.Catalog method)</a>
</li>
- <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Database.__repr__">(datafusion.catalog.Database method)</a>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Schema.__repr__">(datafusion.catalog.Schema method)</a>
</li>
<li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Table.__repr__">(datafusion.catalog.Table method)</a>
</li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.__repr__">(datafusion.context.SessionContext method)</a>
-</li>
- <li><a href="autoapi/datafusion/index.html#datafusion.Database.__repr__">(datafusion.Database method)</a>
</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.__repr__">(datafusion.dataframe.DataFrame method)</a>
</li>
@@ -773,6 +773,8 @@
<li><a href="autoapi/datafusion/plan/index.html#datafusion.plan.LogicalPlan._raw_plan">(datafusion.plan.LogicalPlan attribute)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Schema._raw_schema">_raw_schema (datafusion.catalog.Schema attribute)</a>
+</li>
<li><a href="autoapi/datafusion/dataframe_formatter/index.html#datafusion.dataframe_formatter._refresh_formatter_reference">_refresh_formatter_reference() (in module datafusion.dataframe_formatter)</a>
</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame._repr_html_">_repr_html_() (datafusion.dataframe.DataFrame method)</a>
@@ -1214,6 +1216,12 @@
</li>
</ul></li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.catalog">catalog() (datafusion.context.SessionContext method)</a>
+</li>
+ <li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.catalog_names">catalog_names() (datafusion.context.SessionContext method)</a>
+</li>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.CatalogProvider">CatalogProvider (class in datafusion.catalog)</a>
+</li>
+ <li><a href="autoapi/datafusion/context/index.html#datafusion.context.CatalogProviderExportable">CatalogProviderExportable (class in datafusion.context)</a>
</li>
<li><a href="autoapi/datafusion/index.html#datafusion.Expr.cbrt">cbrt() (datafusion.Expr method)</a>
@@ -1289,14 +1297,14 @@
<li><a href="autoapi/datafusion/index.html#datafusion.ParquetWriterOptions.column_index_truncate_length">(datafusion.ParquetWriterOptions attribute)</a>, <a href="autoapi/datafusion/index.html#id12">[1]</a>
</li>
</ul></li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/index.html#datafusion.Expr.column_name">column_name() (datafusion.Expr method)</a>
<ul>
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.Expr.column_name">(datafusion.expr.Expr method)</a>
</li>
</ul></li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.ParquetWriterOptions.column_specific_options">column_specific_options (datafusion.dataframe.ParquetWriterOptions attribute)</a>, <a href="autoapi/datafusion/dataframe/index.html#id12">[1]</a>
<ul>
@@ -1434,11 +1442,7 @@
</li>
</ul></li>
<li><a href="autoapi/datafusion/index.html#datafusion.Database">Database (class in datafusion)</a>
-
- <ul>
- <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Database">(class in datafusion.catalog)</a>
</li>
- </ul></li>
<li><a href="autoapi/datafusion/index.html#datafusion.Catalog.database">database() (datafusion.Catalog method)</a>
<ul>
@@ -1585,18 +1589,12 @@
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.datetrunc">datetrunc() (in module datafusion.functions)</a>
</li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
- <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Database.db">db (datafusion.catalog.Database attribute)</a>
-
- <ul>
- <li><a href="autoapi/datafusion/index.html#datafusion.Database.db">(datafusion.Database attribute)</a>
-</li>
- </ul></li>
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.Deallocate">Deallocate (in module datafusion.expr)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.decode">decode() (in module datafusion.functions)</a>
</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/unparser/index.html#datafusion.unparser.Dialect.default">default() (datafusion.unparser.Dialect static method)</a>
</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.default_str_repr">default_str_repr() (datafusion.dataframe.DataFrame static method)</a>
@@ -1613,8 +1611,22 @@
</ul></li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.dense_rank">dense_rank() (in module datafusion.functions)</a>
</li>
- <li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.deregister_table">deregister_table() (datafusion.context.SessionContext method)</a>
+ <li><a href="autoapi/datafusion/index.html#datafusion.Catalog.deregister_schema">deregister_schema() (datafusion.Catalog method)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Catalog.deregister_schema">(datafusion.catalog.Catalog method)</a>
+</li>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.CatalogProvider.deregister_schema">(datafusion.catalog.CatalogProvider method)</a>
+</li>
+ </ul></li>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Schema.deregister_table">deregister_table() (datafusion.catalog.Schema method)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.SchemaProvider.deregister_table">(datafusion.catalog.SchemaProvider method)</a>
+</li>
+ <li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.deregister_table">(datafusion.context.SessionContext method)</a>
</li>
+ </ul></li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.describe">describe() (datafusion.dataframe.DataFrame method)</a>
</li>
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.DescribeTable">DescribeTable (in module datafusion.expr)</a>
@@ -1891,6 +1903,12 @@
</li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.from_arrow_table">from_arrow_table() (datafusion.context.SessionContext method)</a>
</li>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Table.from_dataset">from_dataset() (datafusion.catalog.Table static method)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/index.html#datafusion.Table.from_dataset">(datafusion.Table static method)</a>
+</li>
+ </ul></li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.from_pandas">from_pandas() (datafusion.context.SessionContext method)</a>
</li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.from_polars">from_polars() (datafusion.context.SessionContext method)</a>
@@ -2378,6 +2396,14 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.median">median() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/user_defined/index.html#datafusion.user_defined.WindowEvaluator.memoize">memoize() (datafusion.user_defined.WindowEvaluator method)</a>
+</li>
+ <li><a href="autoapi/datafusion/index.html#datafusion.Catalog.memory_catalog">memory_catalog() (datafusion.Catalog static method)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Catalog.memory_catalog">(datafusion.catalog.Catalog static method)</a>
+</li>
+ </ul></li>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Schema.memory_schema">memory_schema() (datafusion.catalog.Schema static method)</a>
</li>
<li><a href="autoapi/datafusion/index.html#datafusion.Accumulator.merge">merge() (datafusion.Accumulator method)</a>
@@ -2449,9 +2475,7 @@
<ul>
<li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Catalog.names">(datafusion.catalog.Catalog method)</a>
</li>
- <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Database.names">(datafusion.catalog.Database method)</a>
-</li>
- <li><a href="autoapi/datafusion/index.html#datafusion.Database.names">(datafusion.Database method)</a>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Schema.names">(datafusion.catalog.Schema method)</a>
</li>
</ul></li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.nanvl">nanvl() (in module datafusion.functions)</a>
@@ -2529,6 +2553,8 @@
</li>
</ul></li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.overlay">overlay() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.SchemaProvider.owner_name">owner_name() (datafusion.catalog.SchemaProvider method)</a>
</li>
</ul></td>
</tr></table>
@@ -2685,6 +2711,8 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.regexp_replace">regexp_replace() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.register_avro">register_avro() (datafusion.context.SessionContext method)</a>
+</li>
+ <li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.register_catalog_provider">register_catalog_provider() (datafusion.context.SessionContext method)</a>
</li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.register_csv">register_csv() (datafusion.context.SessionContext method)</a>
</li>
@@ -2702,10 +2730,24 @@
</li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.register_record_batches">register_record_batches() (datafusion.context.SessionContext method)</a>
</li>
+ <li><a href="autoapi/datafusion/index.html#datafusion.Catalog.register_schema">register_schema() (datafusion.Catalog method)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Catalog.register_schema">(datafusion.catalog.Catalog method)</a>
+</li>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.CatalogProvider.register_schema">(datafusion.catalog.CatalogProvider method)</a>
+</li>
+ </ul></li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
- <li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.register_table">register_table() (datafusion.context.SessionContext method)</a>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Schema.register_table">register_table() (datafusion.catalog.Schema method)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.SchemaProvider.register_table">(datafusion.catalog.SchemaProvider method)</a>
+</li>
+ <li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.register_table">(datafusion.context.SessionContext method)</a>
</li>
+ </ul></li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.register_table_provider">register_table_provider() (datafusion.context.SessionContext method)</a>
</li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.register_udaf">register_udaf() (datafusion.context.SessionContext method)</a>
@@ -2817,6 +2859,8 @@
</li>
</ul></li>
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.ScalarVariable">ScalarVariable (in module datafusion.expr)</a>
+</li>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Schema">Schema (class in datafusion.catalog)</a>
</li>
<li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Table.schema">schema (datafusion.catalog.Table property)</a>
@@ -2824,14 +2868,32 @@
<li><a href="autoapi/datafusion/index.html#datafusion.Table.schema">(datafusion.Table property)</a>
</li>
</ul></li>
- <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.schema">schema() (datafusion.dataframe.DataFrame method)</a>
+ <li><a href="autoapi/datafusion/index.html#datafusion.Catalog.schema">schema() (datafusion.Catalog method)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Catalog.schema">(datafusion.catalog.Catalog method)</a>
+</li>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.CatalogProvider.schema">(datafusion.catalog.CatalogProvider method)</a>
+</li>
+ <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.schema">(datafusion.dataframe.DataFrame method)</a>
</li>
+ </ul></li>
<li><a href="autoapi/datafusion/index.html#datafusion.Expr.schema_name">schema_name() (datafusion.Expr method)</a>
<ul>
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.Expr.schema_name">(datafusion.expr.Expr method)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/index.html#datafusion.Catalog.schema_names">schema_names() (datafusion.Catalog method)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Catalog.schema_names">(datafusion.catalog.Catalog method)</a>
+</li>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.CatalogProvider.schema_names">(datafusion.catalog.CatalogProvider method)</a>
+</li>
+ </ul></li>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.SchemaProvider">SchemaProvider (class in datafusion.catalog)</a>
+</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.select">select() (datafusion.dataframe.DataFrame method)</a>
</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.select_columns">select_columns() (datafusion.dataframe.DataFrame method)</a>
@@ -2916,6 +2978,8 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.signum">(in module datafusion.functions)</a>
</li>
</ul></li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.SimilarTo">SimilarTo (in module datafusion.expr)</a>
</li>
<li><a href="autoapi/datafusion/index.html#datafusion.Expr.sin">sin() (datafusion.Expr method)</a>
@@ -2926,8 +2990,6 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.sin">(in module datafusion.functions)</a>
</li>
</ul></li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/index.html#datafusion.Expr.sinh">sinh() (datafusion.Expr method)</a>
<ul>
@@ -3058,16 +3120,26 @@
<li><a href="autoapi/datafusion/index.html#datafusion.Table.table">(datafusion.Table attribute)</a>
</li>
</ul></li>
- <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Database.table">table() (datafusion.catalog.Database method)</a>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Schema.table">table() (datafusion.catalog.Schema method)</a>
<ul>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.SchemaProvider.table">(datafusion.catalog.SchemaProvider method)</a>
+</li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.table">(datafusion.context.SessionContext method)</a>
</li>
- <li><a href="autoapi/datafusion/index.html#datafusion.Database.table">(datafusion.Database method)</a>
+ </ul></li>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.SchemaProvider.table_exist">table_exist() (datafusion.catalog.SchemaProvider method)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.table_exist">(datafusion.context.SessionContext method)</a>
</li>
</ul></li>
- <li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.table_exist">table_exist() (datafusion.context.SessionContext method)</a>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Schema.table_names">table_names() (datafusion.catalog.Schema method)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.SchemaProvider.table_names">(datafusion.catalog.SchemaProvider method)</a>
</li>
+ </ul></li>
<li><a href="autoapi/datafusion/index.html#datafusion.TableFunction">TableFunction (class in datafusion)</a>
<ul>
diff --git a/objects.inv b/objects.inv
index bcf04d74..7ff725e4 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/searchindex.js b/searchindex.js
index 45dd0dd0..775c1852 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"API Reference": [[18, null]], "Additional Resources": [[36, "additional-resources"]], "Aggregate Functions": [[23, "aggregate-functions"], [30, "aggregate-functions"], [32, "aggregate-functions"]], "Aggregation": [[23, null]], "Alternative Approach": [[19, "alternative-approach"]], "Arrays": [[25, "arrays"]], "Arrow": [[38, null]], "Attributes": [[4, "attributes"], [7, "attributes"], [12, "attributes"], [17, "attributes"]], "Available Functions": [[32, "av [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"API Reference": [[18, null]], "Additional Resources": [[36, "additional-resources"]], "Aggregate Functions": [[23, "aggregate-functions"], [30, "aggregate-functions"], [32, "aggregate-functions"]], "Aggregation": [[23, null]], "Alternative Approach": [[19, "alternative-approach"]], "Arrays": [[25, "arrays"]], "Arrow": [[38, null]], "Attributes": [[4, "attributes"], [7, "attributes"], [12, "attributes"], [17, "attributes"]], "Available Functions": [[32, "av [...]
\ No newline at end of file
diff --git a/user-guide/common-operations/aggregations.html b/user-guide/common-operations/aggregations.html
index 185063c3..c98246d9 100644
--- a/user-guide/common-operations/aggregations.html
+++ b/user-guide/common-operations/aggregations.html
@@ -610,15 +610,15 @@ Pokemon set. Since there will be many entries of <code class="docutils literal n
<span class="o">+----------+--------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">Type</span> <span class="mi">2</span> <span class="n">List</span> <span class="o">|</span>
<span class="o">+----------+--------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Bug</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Grass</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Ground</span><span class="p">,</span> <span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Electric</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Steel</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Bug</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Grass</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Electric</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Steel</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Fairy</span> <span class="o">|</span> <span class="p">[]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Normal</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="p">,</span> <span class="n">Fairy</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ice</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Dragon</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Fighting</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Dark</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Dragon</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span class="n">Fighting</span><span class="p">,</span> <span class="p">,</span> <span class="n">Dark</span><span class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ground</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Rock</span><span class="p">]</span> <span class="o">|</span>
<span class="o">+----------+--------------------------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span class="o">.</span>
@@ -634,15 +634,15 @@ entries entirely removed. The second is we can use the <code class="docutils lit
<span class="o">+----------+------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">Type</span> <span class="mi">2</span> <span class="n">List</span> <span class="o">|</span>
<span class="o">+----------+------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Bug</span> <span class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span class="p">,</span> <span class="n">Grass</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Bug</span> <span class="o">|</span> <span class="p">[</span><span class="n">Grass</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Electric</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Steel</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Electric</span> <span class="o">|</span> <span class="p">[</span><span class="n">Steel</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Normal</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Fairy</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ice</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Dragon</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span class="p">,</span> <span class="n">Fighting</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span class="n">Dark</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Rock</span> <span class="o">|</span> <span class="p">[</span><span class="n">Water</span><span class="p">,</span> <span class="n">Ground</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span> <span class="p">[</span><span class="n">Dragon</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Dark</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span class="n">Fighting</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Rock</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Water</span><span class="p">,</span> <span class="n">Ground</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ghost</span> <span class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
<span class="o">+----------+------------------------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span class="o">.</span>
@@ -653,15 +653,15 @@ entries entirely removed. The second is we can use the <code class="docutils lit
<span class="o">+----------+------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">Type</span> <span class="mi">2</span> <span class="n">List</span> <span class="o">|</span>
<span class="o">+----------+------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Bug</span> <span class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span class="p">,</span> <span class="n">Grass</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Bug</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Grass</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Electric</span> <span class="o">|</span> <span class="p">[</span><span class="n">Steel</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Fairy</span> <span class="o">|</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Normal</span> <span class="o">|</span> <span class="p">[</span><span class="n">Fairy</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Ice</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Normal</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Fairy</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Ice</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span> <span class="p">[</span><span class="n">Dragon</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Ice</span><span class="p">,</span> <span class="n">Dark</span><span class="p">,</span> <span class="n">Fighting</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Fighting</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span class="n">Dark</span><span class="p">,</span> <span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ground</span> <span class="o">|</span> <span class="p">[</span><span class="n">Rock</span><span class="p">]</span> <span class="o">|</span>
<span class="o">+----------+------------------------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span class="o">.</span>
diff --git a/user-guide/common-operations/basic-info.html b/user-guide/common-operations/basic-info.html
index 86df4abd..09923d23 100644
--- a/user-guide/common-operations/basic-info.html
+++ b/user-guide/common-operations/basic-info.html
@@ -446,11 +446,11 @@
<span class="o">+-----+---------+--------+--------+</span>
<span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span> <span class="n">names</span> <span class="o">|</span> <span class="n">random</span> <span class="o">|</span> <span class="n">groups</span> <span class="o">|</span>
<span class="o">+-----+---------+--------+--------+</span>
-<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">python</span> <span class="o">|</span> <span class="mi">88</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mi">45</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">3</span> <span class="o">|</span> <span class="n">java</span> <span class="o">|</span> <span class="mi">976</span> <span class="o">|</span> <span class="n">B</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">4</span> <span class="o">|</span> <span class="n">haskell</span> <span class="o">|</span> <span class="mi">74</span> <span class="o">|</span> <span class="n">C</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span class="mi">179</span> <span class="o">|</span> <span class="n">B</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">python</span> <span class="o">|</span> <span class="mi">328</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mi">739</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">3</span> <span class="o">|</span> <span class="n">java</span> <span class="o">|</span> <span class="mi">777</span> <span class="o">|</span> <span class="n">B</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">4</span> <span class="o">|</span> <span class="n">haskell</span> <span class="o">|</span> <span class="mi">491</span> <span class="o">|</span> <span class="n">C</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span class="mi">778</span> <span class="o">|</span> <span class="n">B</span> <span class="o">|</span>
<span class="o">+-----+---------+--------+--------+</span>
</pre></div>
</div>
@@ -461,8 +461,8 @@
<span class="o">+-----+--------+--------+--------+</span>
<span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span> <span class="n">names</span> <span class="o">|</span> <span class="n">random</span> <span class="o">|</span> <span class="n">groups</span> <span class="o">|</span>
<span class="o">+-----+--------+--------+--------+</span>
-<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">python</span> <span class="o">|</span> <span class="mi">88</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mi">45</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">python</span> <span class="o">|</span> <span class="mi">328</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mi">739</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
<span class="o">+-----+--------+--------+--------+</span>
</pre></div>
</div>
@@ -480,11 +480,11 @@ passing them to an Arrow table, and then converting them to a pandas DataFrame.<
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">to_pandas</span><span class="p">()</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">8</span><span class="p">]:</span>
<span class="n">nrs</span> <span class="n">names</span> <span class="n">random</span> <span class="n">groups</span>
-<span class="mi">0</span> <span class="mi">1</span> <span class="n">python</span> <span class="mi">88</span> <span class="n">A</span>
-<span class="mi">1</span> <span class="mi">2</span> <span class="n">ruby</span> <span class="mi">45</span> <span class="n">A</span>
-<span class="mi">2</span> <span class="mi">3</span> <span class="n">java</span> <span class="mi">976</span> <span class="n">B</span>
-<span class="mi">3</span> <span class="mi">4</span> <span class="n">haskell</span> <span class="mi">74</span> <span class="n">C</span>
-<span class="mi">4</span> <span class="mi">5</span> <span class="n">go</span> <span class="mi">179</span> <span class="n">B</span>
+<span class="mi">0</span> <span class="mi">1</span> <span class="n">python</span> <span class="mi">328</span> <span class="n">A</span>
+<span class="mi">1</span> <span class="mi">2</span> <span class="n">ruby</span> <span class="mi">739</span> <span class="n">A</span>
+<span class="mi">2</span> <span class="mi">3</span> <span class="n">java</span> <span class="mi">777</span> <span class="n">B</span>
+<span class="mi">3</span> <span class="mi">4</span> <span class="n">haskell</span> <span class="mi">491</span> <span class="n">C</span>
+<span class="mi">4</span> <span class="mi">5</span> <span class="n">go</span> <span class="mi">778</span> <span class="n">B</span>
</pre></div>
</div>
<p><a class="reference internal" href="../../autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.describe" title="datafusion.dataframe.DataFrame.describe"><code class="xref py py-func docutils literal notranslate"><span class="pre">describe()</span></code></a> shows a quick statistic summary of your data:</p>
@@ -496,11 +496,11 @@ passing them to an Arrow table, and then converting them to a pandas DataFrame.<
<span class="o">+------------+--------------------+-------+-------------------+--------+</span>
<span class="o">|</span> <span class="n">count</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="mi">5</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">null_count</span> <span class="o">|</span> <span class="mf">0.0</span> <span class="o">|</span> <span class="mi">0</span> <span class="o">|</span> <span class="mf">0.0</span> <span class="o">|</span> <span class="mi">0</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">mean</span> <span class="o">|</span> <span class="mf">3.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">272.4</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">std</span> <span class="o">|</span> <span class="mf">1.5811388300841898</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">396.5013240835395</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
-<span class="o">|</span> <span class="nb">min</span> <span class="o">|</span> <span class="mf">1.0</span> <span class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span class="mf">45.0</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
-<span class="o">|</span> <span class="nb">max</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mf">976.0</span> <span class="o">|</span> <span class="n">C</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">median</span> <span class="o">|</span> <span class="mf">3.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">88.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">mean</span> <span class="o">|</span> <span class="mf">3.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">622.6</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">std</span> <span class="o">|</span> <span class="mf">1.5811388300841898</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">203.4976658342793</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="nb">min</span> <span class="o">|</span> <span class="mf">1.0</span> <span class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span class="mf">328.0</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="nb">max</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mf">778.0</span> <span class="o">|</span> <span class="n">C</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">median</span> <span class="o">|</span> <span class="mf">3.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">739.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
<span class="o">+------------+--------------------+-------+-------------------+--------+</span>
</pre></div>
</div>
diff --git a/user-guide/common-operations/functions.html b/user-guide/common-operations/functions.html
index 095d0da5..3e3d57e4 100644
--- a/user-guide/common-operations/functions.html
+++ b/user-guide/common-operations/functions.html
@@ -556,16 +556,16 @@ In here we will cover some of the more popular use cases. If you want to view al
<span class="o">+--------------------------------+</span>
<span class="o">|</span> <span class="n">now</span><span class="p">()</span> <span class="o">|</span>
<span class="o">+--------------------------------+</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">25</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mf">04.167557812</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">25</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mf">04.167557812</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">25</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mf">04.167557812</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">25</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mf">04.167557812</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">25</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mf">04.167557812</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">25</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mf">04.167557812</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">25</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mf">04.167557812</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">25</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mf">04.167557812</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">25</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mf">04.167557812</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">25</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mf">04.167557812</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">02</span><span class="n">T12</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">39.194013948</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">02</span><span class="n">T12</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">39.194013948</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">02</span><span class="n">T12</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">39.194013948</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">02</span><span class="n">T12</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">39.194013948</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">02</span><span class="n">T12</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">39.194013948</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">02</span><span class="n">T12</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">39.194013948</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">02</span><span class="n">T12</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">39.194013948</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">02</span><span class="n">T12</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">39.194013948</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">02</span><span class="n">T12</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">39.194013948</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">02</span><span class="n">T12</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">39.194013948</span><span class="n">Z</span> <span class="o">|</span>
<span class="o">+--------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span class="o">.</span>
</pre></div>
diff --git a/user-guide/data-sources.html b/user-guide/data-sources.html
index 99431fc6..a2a08fc2 100644
--- a/user-guide/data-sources.html
+++ b/user-guide/data-sources.html
@@ -407,40 +407,59 @@
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#local-file">
- Local file
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#create-in-memory">
- Create in-memory
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#object-store">
- Object Store
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#other-dataframe-libraries">
- Other DataFrame Libraries
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#delta-lake">
- Delta Lake
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#iceberg">
- Iceberg
+ <li class="toc-h1 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#">
+ Data Sources
</a>
+ <ul class="visible nav section-nav flex-column">
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#local-file">
+ Local file
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#create-in-memory">
+ Create in-memory
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#object-store">
+ Object Store
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#other-dataframe-libraries">
+ Other DataFrame Libraries
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#delta-lake">
+ Delta Lake
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#iceberg">
+ Iceberg
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#custom-table-provider">
+ Custom Table Provider
+ </a>
+ </li>
+ </ul>
</li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#custom-table-provider">
- Custom Table Provider
+ <li class="toc-h1 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#catalog">
+ Catalog
</a>
+ <ul class="visible nav section-nav flex-column">
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#user-defined-catalog-and-schema">
+ User Defined Catalog and Schema
+ </a>
+ </li>
+ </ul>
</li>
</ul>
@@ -667,6 +686,54 @@ section. This is an advanced topic, but a
<a class="reference external" href="https://github.com/apache/datafusion-python/tree/main/examples/ffi-table-provider">user example</a>
is provided in the DataFusion repository.</p>
</section>
+</section>
+<section id="catalog">
+<h1>Catalog<a class="headerlink" href="#catalog" title="Link to this heading">¶</a></h1>
+<p>A common technique for organizing tables is using a three level hierarchical approach. DataFusion
+supports this form of organizing using the <a class="reference internal" href="../autoapi/datafusion/catalog/index.html#datafusion.catalog.Catalog" title="datafusion.catalog.Catalog"><code class="xref py py-class docutils literal notranslate"><span class="pre">Catalog</span></code></a>,
+<a class="reference internal" href="../autoapi/datafusion/catalog/index.html#datafusion.catalog.Schema" title="datafusion.catalog.Schema"><code class="xref py py-class docutils literal notranslate"><span class="pre">Schema</span></code></a>, and <a class="reference internal" href="../autoapi/datafusion/catalog/index.html#datafusion.catalog.Table" title="datafusion.catalog.Table"><code class="xref py py-class docutils literal notranslate"><span class="pre">Table</span></code></a>. By default,
+a <a class="reference internal" href="../autoapi/datafusion/context/index.html#datafusion.context.SessionContext" title="datafusion.context.SessionContext"><code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code></a> comes with a single Catalog and a single Schema
+with the names <code class="docutils literal notranslate"><span class="pre">datafusion</span></code> and <code class="docutils literal notranslate"><span class="pre">default</span></code>, respectively.</p>
+<p>The default implementation uses an in-memory approach to the catalog and schema. We have support
+for adding additional in-memory catalogs and schemas. This can be done like in the following
+example:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">datafusion.catalog</span><span class="w"> </span><span class="kn">import</span> <span class="n">Catalog</span><span class="p">,</span> <span class="n">Schema</span>
+
+<span class="n">my_catalog</span> <span class="o">=</span> <span class="n">Catalog</span><span class="o">.</span><span class="n">memory_catalog</span><span class="p">()</span>
+<span class="n">my_schema</span> <span class="o">=</span> <span class="n">Schema</span><span class="o">.</span><span class="n">memory_schema</span><span class="p">()</span>
+
+<span class="n">my_catalog</span><span class="o">.</span><span class="n">register_schema</span><span class="p">(</span><span class="s2">"my_schema_name"</span><span class="p">,</span> <span class="n">my_schema</span><span class="p">)</span>
+
+<span class="n">ctx</span><span class="o">.</span><span class="n">register_catalog</span><span class="p">(</span><span class="s2">"my_catalog_name"</span><span class="p">,</span> <span class="n">my_catalog</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>You could then register tables in <code class="docutils literal notranslate"><span class="pre">my_schema</span></code> and access them either through the DataFrame
+API or via sql commands such as <code class="docutils literal notranslate"><span class="pre">"SELECT</span> <span class="pre">*</span> <span class="pre">from</span> <span class="pre">my_catalog_name.my_schema_name.my_table"</span></code>.</p>
+<section id="user-defined-catalog-and-schema">
+<h2>User Defined Catalog and Schema<a class="headerlink" href="#user-defined-catalog-and-schema" title="Link to this heading">¶</a></h2>
+<p>If the in-memory catalogs are insufficient for your uses, there are two approaches you can take
+to implementing a custom catalog and/or schema. In the below discussion, we describe how to
+implement these for a Catalog, but the approach to implementing for a Schema is nearly
+identical.</p>
+<p>DataFusion supports Catalogs written in either Rust or Python. If you write a Catalog in Rust,
+you will need to export it as a Python library via PyO3. There is a complete example of a
+catalog implemented this way in the
+<a class="reference external" href="https://github.com/apache/datafusion-python/tree/main/examples/">examples folder</a>
+of our repository. Writing catalog providers in Rust provides typically can lead to significant
+performance improvements over the Python based approach.</p>
+<p>To implement a Catalog in Python, you will need to inherit from the abstract base class
+<a class="reference internal" href="../autoapi/datafusion/catalog/index.html#datafusion.catalog.CatalogProvider" title="datafusion.catalog.CatalogProvider"><code class="xref py py-class docutils literal notranslate"><span class="pre">CatalogProvider</span></code></a>. There are examples in the
+<a class="reference external" href="https://github.com/apache/datafusion-python/tree/main/python/tests">unit tests</a> of
+implementing a basic Catalog in Python where we simply keep a dictionary of the
+registered Schemas.</p>
+<p>One important note for developers is that when we have a Catalog defined in Python, we have
+two different ways of accessing this Catalog. First, we register the catalog with a Rust
+wrapper. This allows for any rust based code to call the Python functions as necessary.
+Second, if the user access the Catalog via the Python API, we identify this and return back
+the original Python object that implements the Catalog. This is an important distinction
+for developers because we do <em>not</em> return a Python wrapper around the Rust wrapper of the
+original Python object.</p>
+</section>
</section>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org