You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by gi...@apache.org on 2023/09/18 14:17:55 UTC
[arrow-datafusion-python] branch asf-site updated: Publish built docs triggered by 54d17771fc2814339f94e0871401ee946d7c913b
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion-python.git
The following commit(s) were added to refs/heads/asf-site by this push:
new cf7bdd5 Publish built docs triggered by 54d17771fc2814339f94e0871401ee946d7c913b
cf7bdd5 is described below
commit cf7bdd50c5e8dc6092d9db47ea7031f85a779480
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Mon Sep 18 14:17:50 2023 +0000
Publish built docs triggered by 54d17771fc2814339f94e0871401ee946d7c913b
---
_sources/api.rst.txt | 1 -
_sources/contributor-guide/introduction.rst.txt | 94 +++
.../generated/datafusion.SessionContext.rst.txt | 3 +
_sources/index.rst.txt | 263 ++------
_sources/user-guide/basics.rst.txt | 89 +++
.../common-operations/aggregations.rst.txt | 60 ++
.../common-operations/basic-info.rst.txt | 61 ++
.../user-guide/common-operations/functions.rst.txt | 117 ++++
.../common-operations/index.rst.txt} | 20 +-
.../user-guide/common-operations/joins.rst.txt | 104 ++++
.../common-operations/select-and-filter.rst.txt | 67 +++
.../common-operations/udf-and-udfa.rst.txt | 85 +++
.../user-guide/common-operations/windows.rst.txt | 93 +++
.../introduction.rst.txt} | 35 +-
.../{api.rst.txt => user-guide/io/avro.rst.txt} | 22 +-
.../{api.rst.txt => user-guide/io/csv.rst.txt} | 28 +-
.../{api.rst.txt => user-guide/io/index.rst.txt} | 18 +-
.../{api.rst.txt => user-guide/io/json.rst.txt} | 21 +-
.../{api.rst.txt => user-guide/io/parquet.rst.txt} | 28 +-
_sources/{api.rst.txt => user-guide/sql.rst.txt} | 31 +-
api.html | 172 +++++-
api/dataframe.html | 166 +++++-
api/execution_context.html | 166 +++++-
api/expression.html | 166 +++++-
api/functions.html | 166 +++++-
api/object_store.html | 166 +++++-
contributor-guide/introduction.html | 497 ++++++++++++++++
generated/datafusion.DataFrame.html | 170 +++++-
generated/datafusion.Expr.html | 166 +++++-
generated/datafusion.RuntimeConfig.html | 166 +++++-
generated/datafusion.SessionConfig.html | 166 +++++-
generated/datafusion.SessionContext.html | 249 +++++++-
generated/datafusion.functions.functions.html | 406 ++++++++++++-
.../datafusion.object_store.object_store.html | 166 +++++-
genindex.html | 232 +++++++-
index.html | 456 +++++++-------
objects.inv | Bin 2023 -> 2498 bytes
py-modindex.html | 166 +++++-
search.html | 166 +++++-
searchindex.js | 2 +-
user-guide/basics.html | 481 +++++++++++++++
user-guide/common-operations/aggregations.html | 491 +++++++++++++++
user-guide/common-operations/basic-info.html | 500 ++++++++++++++++
user-guide/common-operations/functions.html | 660 +++++++++++++++++++++
user-guide/common-operations/index.html | 446 ++++++++++++++
user-guide/common-operations/joins.html | 567 ++++++++++++++++++
.../common-operations/select-and-filter.html | 529 +++++++++++++++++
user-guide/common-operations/udf-and-udfa.html | 493 +++++++++++++++
user-guide/common-operations/windows.html | 540 +++++++++++++++++
api/dataframe.html => user-guide/introduction.html | 237 ++++++--
user-guide/io/avro.html | 429 ++++++++++++++
user-guide/io/csv.html | 433 ++++++++++++++
user-guide/io/index.html | 429 ++++++++++++++
user-guide/io/json.html | 429 ++++++++++++++
user-guide/io/parquet.html | 433 ++++++++++++++
user-guide/sql.html | 456 ++++++++++++++
56 files changed, 11944 insertions(+), 859 deletions(-)
diff --git a/_sources/api.rst.txt b/_sources/api.rst.txt
index a5d6543..d9f4a09 100644
--- a/_sources/api.rst.txt
+++ b/_sources/api.rst.txt
@@ -24,7 +24,6 @@ API Reference
.. toctree::
:maxdepth: 2
- api/config
api/dataframe
api/execution_context
api/expression
diff --git a/_sources/contributor-guide/introduction.rst.txt b/_sources/contributor-guide/introduction.rst.txt
new file mode 100644
index 0000000..6de2b87
--- /dev/null
+++ b/_sources/contributor-guide/introduction.rst.txt
@@ -0,0 +1,94 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+Introduction
+============
+We welcome and encourage contributions of all kinds, such as:
+
+1. Tickets with issue reports of feature requests
+2. Documentation improvements
+3. Code, both PR and (especially) PR Review.
+
+In addition to submitting new PRs, we have a healthy tradition of community members reviewing each other’s PRs.
+Doing so is a great way to help the community as well as get more familiar with Rust and the relevant codebases.
+
+How to develop
+--------------
+
+This assumes that you have rust and cargo installed. We use the workflow recommended by `pyo3 <https://github.com/PyO3/pyo3>`_ and `maturin <https://github.com/PyO3/maturin>`_.
+
+Bootstrap:
+
+.. code-block:: shell
+
+ # fetch this repo
+ git clone git@github.com:apache/arrow-datafusion-python.git
+ # prepare development environment (used to build wheel / install in development)
+ python3 -m venv venv
+ # activate the venv
+ source venv/bin/activate
+ # update pip itself if necessary
+ python -m pip install -U pip
+ # install dependencies (for Python 3.8+)
+ python -m pip install -r requirements-310.txt
+
+The tests rely on test data in git submodules.
+
+.. code-block:: shell
+
+ git submodule init
+ git submodule update
+
+
+Whenever rust code changes (your changes or via `git pull`):
+
+.. code-block:: shell
+
+ # make sure you activate the venv using "source venv/bin/activate" first
+ maturin develop
+ python -m pytest
+
+Running & Installing pre-commit hooks
+-------------------------------------
+
+arrow-datafusion-python takes advantage of `pre-commit <https://pre-commit.com/>`_ to assist developers with code linting to help reduce the number of commits that ultimately fail in CI due to linter errors. Using the pre-commit hooks is optional for the developer but certainly helpful for keeping PRs clean and concise.
+
+Our pre-commit hooks can be installed by running :code:`pre-commit install`, which will install the configurations in your ARROW_DATAFUSION_PYTHON_ROOT/.github directory and run each time you perform a commit, failing to complete the commit if an offending lint is found allowing you to make changes locally before pushing.
+
+The pre-commit hooks can also be run adhoc without installing them by simply running :code:`pre-commit run --all-files`
+
+
+Update Dependencies
+-------------------
+
+To change test dependencies, change the `requirements.in` and run
+
+.. code-block:: shell
+
+ # install pip-tools (this can be done only once), also consider running in venv
+ python -m pip install pip-tools
+ python -m piptools compile --generate-hashes -o requirements-310.txt
+
+
+To update dependencies, run with `-U`
+
+.. code-block:: shell
+
+ python -m piptools compile -U --generate-hashes -o requirements-310.txt
+
+
+More details about pip-tools `here <https://github.com/jazzband/pip-tools>`_
diff --git a/_sources/generated/datafusion.SessionContext.rst.txt b/_sources/generated/datafusion.SessionContext.rst.txt
index 8ef3db6..923a385 100644
--- a/_sources/generated/datafusion.SessionContext.rst.txt
+++ b/_sources/generated/datafusion.SessionContext.rst.txt
@@ -29,8 +29,11 @@
~SessionContext.read_csv
~SessionContext.read_json
~SessionContext.read_parquet
+ ~SessionContext.read_table
+ ~SessionContext.register_avro
~SessionContext.register_csv
~SessionContext.register_dataset
+ ~SessionContext.register_json
~SessionContext.register_object_store
~SessionContext.register_parquet
~SessionContext.register_record_batches
diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt
index 78f44ea..155b0cf 100644
--- a/_sources/index.rst.txt
+++ b/_sources/index.rst.txt
@@ -31,12 +31,17 @@ Its query engine, DataFusion, is written in `Rust <https://www.rust-lang.org>`_,
Technically, zero-copy is achieved via the `c data interface <https://arrow.apache.org/docs/format/CDataInterface.html>`_.
-How to use it
-=============
+Install
+-------
-Simple usage:
+.. code-block:: shell
+
+ pip install datafusion
-.. code-block:: python
+Example
+-------
+
+.. ipython:: python
import datafusion
from datafusion import col
@@ -50,7 +55,7 @@ Simple usage:
[pyarrow.array([1, 2, 3]), pyarrow.array([4, 5, 6])],
names=["a", "b"],
)
- df = ctx.create_dataframe([[batch]])
+ df = ctx.create_dataframe([[batch]], name="batch_array")
# create a new statement
df = df.select(
@@ -58,234 +63,44 @@ Simple usage:
col("a") - col("b"),
)
- # execute and collect the first (and only) batch
- result = df.collect()[0]
-
- assert result.column(0) == pyarrow.array([5, 7, 9])
- assert result.column(1) == pyarrow.array([-3, -3, -3])
-
-
-We can also execute a query against data stored in CSV
-
-.. code-block:: bash
-
- echo "a,b\n1,4\n2,5\n3,6" > example.csv
-
-
-.. code-block:: python
-
- import datafusion
- from datafusion import col
- import pyarrow
-
- # create a context
- ctx = datafusion.SessionContext()
-
- # register a CSV
- ctx.register_csv('example', 'example.csv')
-
- # create a new statement
- df = ctx.table('example').select(
- col("a") + col("b"),
- col("a") - col("b"),
- )
-
- # execute and collect the first (and only) batch
- result = df.collect()[0]
-
- assert result.column(0) == pyarrow.array([5, 7, 9])
- assert result.column(1) == pyarrow.array([-3, -3, -3])
-
-
-And how to execute a query against a CSV using SQL:
-
-
-.. code-block:: python
-
- import datafusion
- from datafusion import col
- import pyarrow
-
- # create a context
- ctx = datafusion.SessionContext()
-
- # register a CSV
- ctx.register_csv('example', 'example.csv')
-
- # create a new statement via SQL
- df = ctx.sql("SELECT a+b, a-b FROM example")
-
- # execute and collect the first (and only) batch
- result = df.collect()[0]
-
- assert result.column(0) == pyarrow.array([5, 7, 9])
- assert result.column(1) == pyarrow.array([-3, -3, -3])
-
-
-
-UDFs
-----
-
-.. code-block:: python
-
- import pyarrow
- from datafusion import udf
-
- def is_null(array: pyarrow.Array) -> pyarrow.Array:
- return array.is_null()
-
- is_null_arr = udf(is_null, [pyarrow.int64()], pyarrow.bool_(), 'stable')
-
- # create a context
- ctx = datafusion.SessionContext()
-
- # create a RecordBatch and a new DataFrame from it
- batch = pyarrow.RecordBatch.from_arrays(
- [pyarrow.array([1, 2, 3]), pyarrow.array([4, 5, 6])],
- names=["a", "b"],
- )
- df = ctx.create_dataframe([[batch]])
-
- df = df.select(is_null_arr(col("a")))
-
- result = df.collect()[0]
-
- assert result.column(0) == pyarrow.array([False] * 3)
-
+ df
-UDAF
-----
-
-.. code-block:: python
-
- import pyarrow
- import pyarrow.compute
- import datafusion
- from datafusion import udaf, Accumulator
- from datafusion import col
-
-
- class MyAccumulator(Accumulator):
- """
- Interface of a user-defined accumulation.
- """
- def __init__(self):
- self._sum = pyarrow.scalar(0.0)
-
- def update(self, values: pyarrow.Array) -> None:
- # not nice since pyarrow scalars can't be summed yet. This breaks on `None`
- self._sum = pyarrow.scalar(self._sum.as_py() + pyarrow.compute.sum(values).as_py())
-
- def merge(self, states: pyarrow.Array) -> None:
- # not nice since pyarrow scalars can't be summed yet. This breaks on `None`
- self._sum = pyarrow.scalar(self._sum.as_py() + pyarrow.compute.sum(states).as_py())
-
- def state(self) -> pyarrow.Array:
- return pyarrow.array([self._sum.as_py()])
-
- def evaluate(self) -> pyarrow.Scalar:
- return self._sum
-
- # create a context
- ctx = datafusion.SessionContext()
-
- # create a RecordBatch and a new DataFrame from it
- batch = pyarrow.RecordBatch.from_arrays(
- [pyarrow.array([1, 2, 3]), pyarrow.array([4, 5, 6])],
- names=["a", "b"],
- )
- df = ctx.create_dataframe([[batch]])
-
- my_udaf = udaf(MyAccumulator, pyarrow.float64(), pyarrow.float64(), [pyarrow.float64()], 'stable')
-
- df = df.aggregate(
- [],
- [my_udaf(col("a"))]
- )
-
- result = df.collect()[0]
-
- assert result.column(0) == pyarrow.array([6.0])
-
-How to install (from pip)
-=========================
-
-.. code-block:: shell
-
- pip install datafusion
-
-You can verify the installation by running:
-
-.. code-block:: python
-
- >>> import datafusion
- >>> datafusion.__version__
- '0.6.0'
-
-
-How to develop
-==============
-
-This assumes that you have rust and cargo installed. We use the workflow recommended by `pyo3 <https://github.com/PyO3/pyo3>`_ and `maturin <https://github.com/PyO3/maturin>`_.
-
-Bootstrap:
-
-.. code-block:: shell
-
- # fetch this repo
- git clone git@github.com:apache/arrow-datafusion-python.git
- # prepare development environment (used to build wheel / install in development)
- python3 -m venv venv
- # activate the venv
- source venv/bin/activate
- # update pip itself if necessary
- python -m pip install -U pip
- # install dependencies (for Python 3.8+)
- python -m pip install -r requirements-310.txt
-
-The tests rely on test data in git submodules.
-
-.. code-block:: shell
-
- git submodule init
- git submodule update
-
-
-Whenever rust code changes (your changes or via `git pull`):
-
-.. code-block:: shell
-
- # make sure you activate the venv using "source venv/bin/activate" first
- maturin develop
- python -m pytest
-
-
-How to update dependencies
-==========================
-
-To change test dependencies, change the `requirements.in` and run
-
-.. code-block:: shell
-
- # install pip-tools (this can be done only once), also consider running in venv
- python -m pip install pip-tools
- python -m piptools compile --generate-hashes -o requirements-310.txt
+.. _toc.links:
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: LINKS
-To update dependencies, run with `-U`
+ Github and Issue Tracker <https://github.com/apache/arrow-datafusion-python>
+ Rust's API Docs <https://docs.rs/datafusion/latest/datafusion/>
+ Code of conduct <https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md>
-.. code-block:: shell
-
- python -m piptools compile -U --generate-hashes -o requirements-310.txt
+.. _toc.guide:
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: USER GUIDE
+ user-guide/introduction
+ user-guide/basics
+ user-guide/common-operations/index
+ user-guide/io/index
+ user-guide/sql
-More details about pip-tools `here <https://github.com/jazzband/pip-tools>`_
+.. _toc.contributor_guide:
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: CONTRIBUTOR GUIDE
-API reference
-=============
+ contributor-guide/introduction
+.. _toc.api:
.. toctree::
- :maxdepth: 2
+ :hidden:
+ :maxdepth: 1
+ :caption: API
api
diff --git a/_sources/user-guide/basics.rst.txt b/_sources/user-guide/basics.rst.txt
new file mode 100644
index 0000000..438b231
--- /dev/null
+++ b/_sources/user-guide/basics.rst.txt
@@ -0,0 +1,89 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+Concepts
+========
+
+In this section, we will cover a basic example to introduce a few key concepts.
+
+.. code-block:: python
+
+ import datafusion
+ from datafusion import col
+ import pyarrow
+
+ # create a context
+ ctx = datafusion.SessionContext()
+
+ # create a RecordBatch and a new DataFrame from it
+ batch = pyarrow.RecordBatch.from_arrays(
+ [pyarrow.array([1, 2, 3]), pyarrow.array([4, 5, 6])],
+ names=["a", "b"],
+ )
+ df = ctx.create_dataframe([[batch]])
+
+ # create a new statement
+ df = df.select(
+ col("a") + col("b"),
+ col("a") - col("b"),
+ )
+
+ # execute and collect the first (and only) batch
+ result = df.collect()[0]
+
+The first statement group:
+
+.. code-block:: python
+
+ # create a context
+ ctx = datafusion.SessionContext()
+
+creates a :code:`SessionContext`, that is, the main interface for executing queries with DataFusion. It maintains the state
+of the connection between a user and an instance of the DataFusion engine. Additionally it provides the following functionality:
+
+- Create a DataFrame from a CSV or Parquet data source.
+- Register a CSV or Parquet data source as a table that can be referenced from a SQL query.
+- Register a custom data source that can be referenced from a SQL query.
+- Execute a SQL query
+
+The second statement group creates a :code:`DataFrame`,
+
+.. code-block:: python
+
+ # create a RecordBatch and a new DataFrame from it
+ batch = pyarrow.RecordBatch.from_arrays(
+ [pyarrow.array([1, 2, 3]), pyarrow.array([4, 5, 6])],
+ names=["a", "b"],
+ )
+ df = ctx.create_dataframe([[batch]])
+
+A DataFrame refers to a (logical) set of rows that share the same column names, similar to a `Pandas DataFrame <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html>`_.
+DataFrames are typically created by calling a method on :code:`SessionContext`, such as :code:`read_csv`, and can then be modified by
+calling the transformation methods, such as :meth:`.DataFrame.filter`, :meth:`.DataFrame.select`, :meth:`.DataFrame.aggregate`,
+and :meth:`.DataFrame.limit` to build up a query definition.
+
+The third statement uses :code:`Expressions` to build up a query definition.
+
+.. code-block:: python
+
+ df = df.select(
+ col("a") + col("b"),
+ col("a") - col("b"),
+ )
+
+Finally the :code:`collect` method converts the logical plan represented by the DataFrame into a physical plan and execute it,
+collecting all results into a list of `RecordBatch <https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatch.html>`_.
\ No newline at end of file
diff --git a/_sources/user-guide/common-operations/aggregations.rst.txt b/_sources/user-guide/common-operations/aggregations.rst.txt
new file mode 100644
index 0000000..235d644
--- /dev/null
+++ b/_sources/user-guide/common-operations/aggregations.rst.txt
@@ -0,0 +1,60 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+Aggregation
+============
+
+An aggregate or aggregation is a function where the values of multiple rows are processed together to form a single summary value.
+For performing an aggregation, DataFusion provides the :meth:`.DataFrame.aggregate`
+
+.. ipython:: python
+
+ from datafusion import SessionContext
+ from datafusion import column, lit
+ from datafusion import functions as f
+ import random
+
+ ctx = SessionContext()
+ df = ctx.from_pydict(
+ {
+ "a": ["foo", "bar", "foo", "bar", "foo", "bar", "foo", "foo"],
+ "b": ["one", "one", "two", "three", "two", "two", "one", "three"],
+ "c": [random.randint(0, 100) for _ in range(8)],
+ "d": [random.random() for _ in range(8)],
+ },
+ name="foo_bar"
+ )
+
+ col_a = column("a")
+ col_b = column("b")
+ col_c = column("c")
+ col_d = column("d")
+
+ df.aggregate([], [f.approx_distinct(col_c), f.approx_median(col_d), f.approx_percentile_cont(col_d, lit(0.5))])
+
+When the :code:`group_by` list is empty the aggregation is done over the whole :class:`.DataFrame`. For grouping
+the :code:`group_by` list must contain at least one column
+
+.. ipython:: python
+
+ df.aggregate([col_a], [f.sum(col_c), f.max(col_d), f.min(col_d)])
+
+More than one column can be used for grouping
+
+.. ipython:: python
+
+ df.aggregate([col_a, col_b], [f.sum(col_c), f.max(col_d), f.min(col_d)])
diff --git a/_sources/user-guide/common-operations/basic-info.rst.txt b/_sources/user-guide/common-operations/basic-info.rst.txt
new file mode 100644
index 0000000..424e1cc
--- /dev/null
+++ b/_sources/user-guide/common-operations/basic-info.rst.txt
@@ -0,0 +1,61 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+Basic Operations
+================
+
+In this section, you will learn how to display essential details of DataFrames using specific functions.
+
+.. ipython:: python
+
+ from datafusion import SessionContext
+ import random
+
+ ctx = SessionContext()
+ df = ctx.from_pydict({
+ "nrs": [1, 2, 3, 4, 5],
+ "names": ["python", "ruby", "java", "haskell", "go"],
+ "random": random.sample(range(1000), 5),
+ "groups": ["A", "A", "B", "C", "B"],
+ })
+ df
+
+Use :meth:`.DataFrame.limit` to view the top rows of the frame:
+
+.. ipython:: python
+
+ df.limit(2)
+
+Display the columns of the DataFrame using :meth:`.DataFrame.schema`:
+
+.. ipython:: python
+
+ df.schema()
+
+The method :meth:`.DataFrame.to_pandas` uses pyarrow to convert to pandas DataFrame, by collecting the batches,
+passing them to an Arrow table, and then converting them to a pandas DataFrame.
+
+.. ipython:: python
+
+ df.to_pandas()
+
+:meth:`.DataFrame.describe` shows a quick statistic summary of your data:
+
+.. ipython:: python
+
+ df.describe()
+
diff --git a/_sources/user-guide/common-operations/functions.rst.txt b/_sources/user-guide/common-operations/functions.rst.txt
new file mode 100644
index 0000000..7e5c592
--- /dev/null
+++ b/_sources/user-guide/common-operations/functions.rst.txt
@@ -0,0 +1,117 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+Functions
+=========
+
+DataFusion provides a large number of built-in functions for performing complex queries without requiring user-defined functions.
+In here we will cover some of the more popular use cases. If you want to view all the functions go to the :ref:`Functions` API Reference.
+
+We'll use the pokemon dataset in the following examples.
+
+.. ipython:: python
+
+ import urllib.request
+ from datafusion import SessionContext
+
+ urllib.request.urlretrieve(
+ "https://gist.githubusercontent.com/ritchie46/cac6b337ea52281aa23c049250a4ff03/raw/89a957ff3919d90e6ef2d34235e6bf22304f3366/pokemon.csv",
+ "pokemon.csv",
+ )
+
+ ctx = SessionContext()
+ ctx.register_csv("pokemon", "pokemon.csv")
+ df = ctx.table("pokemon")
+
+Mathematical
+------------
+
+DataFusion offers mathematical functions such as :func:`.pow` or :func:`.log`
+
+.. ipython:: python
+
+ from datafusion import col, literal
+ from datafusion import functions as f
+
+ df.select(
+ f.pow(col('"Attack"'), literal(2)) - f.pow(col('"Defense"'), literal(2))
+ ).limit(10)
+
+
+Conditional
+-----------
+
+There 3 conditional functions in DataFusion :func:`.coalesce`, :func:`.nullif` and :func:`.case` (not available in Python)
+
+.. ipython:: python
+
+ df.select(
+ f.coalesce(col('"Type 1"'), col('"Type 2"')).alias("dominant_type")
+ ).limit(10)
+
+Temporal
+--------
+
+For selecting the current time use :func:`.now`
+
+.. ipython:: python
+
+ df.select(f.now())
+
+Convert to timestamps using :func:`.to_timestamp`
+
+.. ipython:: python
+
+ df.select(f.to_timestamp(col('"Total"')).alias("timestamp"))
+
+String
+------
+
+In the field of data science, working with textual data is a common task. To make string manipulation easier,
+DataFusion offers a range of helpful options.
+
+.. ipython:: python
+
+ df.select(
+ f.char_length(col('"Name"')).alias("len"),
+ f.lower(col('"Name"')).alias("lower"),
+ f.left(col('"Name"'), literal(4)).alias("code")
+ )
+
+This also includes the functions for regular expressions :func:`.regexp_replace` and :func:`.regexp_match`
+
+.. ipython:: python
+
+ df.select(
+ f.regexp_match(col('"Name"'), literal("Char")).alias("dragons"),
+ f.regexp_replace(col('"Name"'), literal("saur"), literal("fleur")).alias("flowers")
+ )
+
+
+Other
+-----
+
+The function :func:`.in_list` allows to check a column for the presence of multiple values:
+
+.. ipython:: python
+
+ types = [literal("Grass"), literal("Fire"), literal("Water")]
+ (
+ df.select(f.in_list(col('"Type 1"'), types, negated=False).alias("basic_types"))
+ .limit(20)
+ .to_pandas()
+ )
diff --git a/_sources/api.rst.txt b/_sources/user-guide/common-operations/index.rst.txt
similarity index 83%
copy from _sources/api.rst.txt
copy to _sources/user-guide/common-operations/index.rst.txt
index a5d6543..950afb9 100644
--- a/_sources/api.rst.txt
+++ b/_sources/user-guide/common-operations/index.rst.txt
@@ -15,18 +15,16 @@
.. specific language governing permissions and limitations
.. under the License.
-.. _api:
-
-*************
-API Reference
-*************
+Common Operations
+=================
.. toctree::
:maxdepth: 2
- api/config
- api/dataframe
- api/execution_context
- api/expression
- api/functions
- api/object_store
+ basic-info
+ select-and-filter
+ joins
+ functions
+ aggregations
+ windows
+ udf-and-udfa
diff --git a/_sources/user-guide/common-operations/joins.rst.txt b/_sources/user-guide/common-operations/joins.rst.txt
new file mode 100644
index 0000000..1282031
--- /dev/null
+++ b/_sources/user-guide/common-operations/joins.rst.txt
@@ -0,0 +1,104 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+Joins
+=====
+
+DataFusion supports the following join variants via the method :meth:`.DataFrame.join`
+
+- Inner Join
+- Left Join
+- Right Join
+- Full Join
+- Left Semi Join
+- Left Anti Join
+
+For the examples in this section we'll use the following two DataFrames
+
+.. ipython:: python
+
+ from datafusion import SessionContext
+
+ ctx = SessionContext()
+
+ left = ctx.from_pydict(
+ {
+ "customer_id": [1, 2, 3],
+ "customer": ["Alice", "Bob", "Charlie"],
+ }
+ )
+
+ right = ctx.from_pylist([
+ {"id": 1, "name": "CityCabs"},
+ {"id": 2, "name": "MetroRide"},
+ {"id": 5, "name": "UrbanGo"},
+ ])
+
+Inner Join
+----------
+
+When using an inner join, only rows containing the common values between the two join columns present in both DataFrames
+will be included in the resulting DataFrame.
+
+.. ipython:: python
+
+ left.join(right, join_keys=(["customer_id"], ["id"]), how="inner")
+
+The parameter :code:`join_keys` specifies the columns from the left DataFrame and right DataFrame that contains the values
+that should match.
+
+Left Join
+---------
+
+A left join combines rows from two DataFrames using the key columns. It returns all rows from the left DataFrame and
+matching rows from the right DataFrame. If there's no match in the right DataFrame, it returns null
+values for the corresponding columns.
+
+.. ipython:: python
+
+ left.join(right, join_keys=(["customer_id"], ["id"]), how="left")
+
+Full Join
+---------
+
+A full join merges rows from two tables based on a related column, returning all rows from both tables, even if there
+is no match. Unmatched rows will have null values.
+
+.. ipython:: python
+
+ left.join(right, join_keys=(["customer_id"], ["id"]), how="full")
+
+Left Semi Join
+--------------
+
+A left semi join retrieves matching rows from the left table while
+omitting duplicates with multiple matches in the right table.
+
+.. ipython:: python
+
+ left.join(right, join_keys=(["customer_id"], ["id"]), how="semi")
+
+Left Anti Join
+--------------
+
+A left anti join shows all rows from the left table without any matching rows in the right table,
+based on a the specified matching columns. It excludes rows from the left table that have at least one matching row in
+the right table.
+
+.. ipython:: python
+
+ left.join(right, join_keys=(["customer_id"], ["id"]), how="anti")
\ No newline at end of file
diff --git a/_sources/user-guide/common-operations/select-and-filter.rst.txt b/_sources/user-guide/common-operations/select-and-filter.rst.txt
new file mode 100644
index 0000000..8ede230
--- /dev/null
+++ b/_sources/user-guide/common-operations/select-and-filter.rst.txt
@@ -0,0 +1,67 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+Column Selections
+=================
+
+Use :meth:`.DataFrame.select_columns` for basic column selection.
+
+DataFusion can work with several file types, to start simple we can use a subset of the
+`TLC Trip Record Data <https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page>`_
+
+.. ipython:: python
+
+ import urllib.request
+ from datafusion import SessionContext
+
+ urllib.request.urlretrieve("https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2021-01.parquet",
+ "yellow_trip_data.parquet")
+
+ ctx = SessionContext()
+ df = ctx.read_parquet("yellow_trip_data.parquet")
+ df.select_columns("trip_distance", "passenger_count")
+
+For mathematical or logical operations use :func:`.col` to select columns, and give meaningful names to the resulting
+operations using :func:`.alias`
+
+
+.. ipython:: python
+
+ from datafusion import col, lit
+ df.select((col("tip_amount") + col("tolls_amount")).alias("tips_plus_tolls"))
+
+.. warning::
+
+ Please be aware that all identifiers are effectively made lower-case in SQL, so if your file has capital letters
+ (ex: Name) you must put your column name in double quotes or the selection won’t work. As an alternative for simple
+ column selection use :meth:`.DataFrame.select_columns` without double quotes
+
+For selecting columns with capital letters use ``'"VendorID"'``
+
+.. ipython:: python
+
+ df.select(col('"VendorID"'))
+
+
+To combine it with literal values use the :func:`.lit`
+
+.. ipython:: python
+
+ large_trip_distance = col("trip_distance") > lit(5.0)
+ low_passenger_count = col("passenger_count") < lit(4)
+ df.select((large_trip_distance & low_passenger_count).alias("lonely_trips"))
+
diff --git a/_sources/user-guide/common-operations/udf-and-udfa.rst.txt b/_sources/user-guide/common-operations/udf-and-udfa.rst.txt
new file mode 100644
index 0000000..62d249c
--- /dev/null
+++ b/_sources/user-guide/common-operations/udf-and-udfa.rst.txt
@@ -0,0 +1,85 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+User Defined Functions
+======================
+
+DataFusion provides powerful expressions and functions, reducing the need for custom Python functions.
+However you can still incorporate your own functions, i.e. User-Defined Functions (UDFs), with the :func:`.udf` function.
+
+.. ipython:: python
+
+ import pyarrow
+ import datafusion
+ from datafusion import udf, col
+
+ def is_null(array: pyarrow.Array) -> pyarrow.Array:
+ return array.is_null()
+
+ is_null_arr = udf(is_null, [pyarrow.int64()], pyarrow.bool_(), 'stable')
+
+ ctx = datafusion.SessionContext()
+
+ batch = pyarrow.RecordBatch.from_arrays(
+ [pyarrow.array([1, 2, 3]), pyarrow.array([4, 5, 6])],
+ names=["a", "b"],
+ )
+ df = ctx.create_dataframe([[batch]], name="batch_array")
+
+ df.select(is_null_arr(col("a"))).to_pandas()
+
+Additionally the :func:`.udaf` function allows you to define User-Defined Aggregate Functions (UDAFs)
+
+.. code-block:: python
+
+ import pyarrow
+ import pyarrow.compute
+ import datafusion
+ from datafusion import col, udaf, Accumulator
+
+ class MyAccumulator(Accumulator):
+ """
+ Interface of a user-defined accumulation.
+ """
+ def __init__(self):
+ self._sum = pyarrow.scalar(0.0)
+
+ def update(self, values: pyarrow.Array) -> None:
+ # not nice since pyarrow scalars can't be summed yet. This breaks on `None`
+ self._sum = pyarrow.scalar(self._sum.as_py() + pyarrow.compute.sum(values).as_py())
+
+ def merge(self, states: pyarrow.Array) -> None:
+ # not nice since pyarrow scalars can't be summed yet. This breaks on `None`
+ self._sum = pyarrow.scalar(self._sum.as_py() + pyarrow.compute.sum(states).as_py())
+
+ def state(self) -> pyarrow.Array:
+ return pyarrow.array([self._sum.as_py()])
+
+ def evaluate(self) -> pyarrow.Scalar:
+ return self._sum
+
+ ctx = datafusion.SessionContext()
+ df = ctx.from_pydict(
+ {
+ "a": [1, 2, 3],
+ "b": [4, 5, 6],
+ }
+ )
+
+ my_udaf = udaf(MyAccumulator, pyarrow.float64(), pyarrow.float64(), [pyarrow.float64()], 'stable')
+
+ df.aggregate([],[my_udaf(col("a"))])
diff --git a/_sources/user-guide/common-operations/windows.rst.txt b/_sources/user-guide/common-operations/windows.rst.txt
new file mode 100644
index 0000000..f884c7e
--- /dev/null
+++ b/_sources/user-guide/common-operations/windows.rst.txt
@@ -0,0 +1,93 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+Window Functions
+================
+
+In this section you will learn about window functions. A window function utilizes values from one or multiple rows to
+produce a result for each individual row, unlike an aggregate function that provides a single value for multiple rows.
+
+The functionality of window functions in DataFusion is supported by the dedicated :func:`.window` function.
+
+We'll use the pokemon dataset (from Ritchie Vink) in the following examples.
+
+.. ipython:: python
+
+ import urllib.request
+ from datafusion import SessionContext
+ from datafusion import col
+ from datafusion import functions as f
+
+ urllib.request.urlretrieve(
+ "https://gist.githubusercontent.com/ritchie46/cac6b337ea52281aa23c049250a4ff03/raw/89a957ff3919d90e6ef2d34235e6bf22304f3366/pokemon.csv",
+ "pokemon.csv",
+ )
+
+ ctx = SessionContext()
+ df = ctx.read_csv("pokemon.csv")
+
+Here is an example that shows how to compare each pokemons’s attack power with the average attack power in its :code:`"Type 1"`
+
+.. ipython:: python
+
+ df.select(
+ col('"Name"'),
+ col('"Attack"'),
+ f.alias(
+ f.window("avg", [col('"Attack"')], partition_by=[col('"Type 1"')]),
+ "Average Attack",
+ )
+ )
+
+You can also control the order in which rows are processed by window functions by providing
+a list of :func:`.order_by` functions for the :code:`order_by` parameter.
+
+.. ipython:: python
+
+ df.select(
+ col('"Name"'),
+ col('"Attack"'),
+ f.alias(
+ f.window(
+ "rank",
+ [],
+ partition_by=[col('"Type 1"')],
+ order_by=[f.order_by(col('"Attack"'))],
+ ),
+ "rank",
+ ),
+ )
+
+The possible window functions are:
+
+1. Rank Functions
+ - rank
+ - dense_rank
+ - row_number
+ - ntile
+
+2. Analytical Functions
+ - cume_dist
+ - percent_rank
+ - lag
+ - lead
+ - first_value
+ - last_value
+ - nth_value
+
+3. Aggregate Functions
+ - All aggregate functions can be used as window functions.
diff --git a/_sources/api.rst.txt b/_sources/user-guide/introduction.rst.txt
similarity index 58%
copy from _sources/api.rst.txt
copy to _sources/user-guide/introduction.rst.txt
index a5d6543..8abb911 100644
--- a/_sources/api.rst.txt
+++ b/_sources/user-guide/introduction.rst.txt
@@ -15,18 +15,29 @@
.. specific language governing permissions and limitations
.. under the License.
-.. _api:
+.. _guide:
+
+Introduction
+============
+
+Welcome to the User Guide for the Python bindings of Arrow DataFusion. This guide aims to provide an introduction to
+DataFusion through various examples and highlight the most effective ways of using it.
+
+Installation
+------------
+
+DataFusion is a Python library and, as such, can be installed via pip from `PyPI <https://pypi.org/project/datafusion>`__.
+
+.. code-block:: shell
+
+ pip install datafusion
+
+You can verify the installation by running:
+
+.. ipython:: python
+
+ import datafusion
+ datafusion.__version__
-*************
-API Reference
-*************
-.. toctree::
- :maxdepth: 2
- api/config
- api/dataframe
- api/execution_context
- api/expression
- api/functions
- api/object_store
diff --git a/_sources/api.rst.txt b/_sources/user-guide/io/avro.rst.txt
similarity index 72%
copy from _sources/api.rst.txt
copy to _sources/user-guide/io/avro.rst.txt
index a5d6543..85d546e 100644
--- a/_sources/api.rst.txt
+++ b/_sources/user-guide/io/avro.rst.txt
@@ -15,18 +15,16 @@
.. specific language governing permissions and limitations
.. under the License.
-.. _api:
+Avro
+====
-*************
-API Reference
-*************
+`Avro <https://avro.apache.org/>`_ is a serialization format for record data. Reading an avro file is very straightforward
+with :meth:`.SessionContext.read_avro`
-.. toctree::
- :maxdepth: 2
+.. code-block:: python
- api/config
- api/dataframe
- api/execution_context
- api/expression
- api/functions
- api/object_store
+
+ from datafusion import SessionContext
+
+ ctx = SessionContext()
+ df = ctx.read_avro("file.avro")
\ No newline at end of file
diff --git a/_sources/api.rst.txt b/_sources/user-guide/io/csv.rst.txt
similarity index 68%
copy from _sources/api.rst.txt
copy to _sources/user-guide/io/csv.rst.txt
index a5d6543..3f95c54 100644
--- a/_sources/api.rst.txt
+++ b/_sources/user-guide/io/csv.rst.txt
@@ -15,18 +15,22 @@
.. specific language governing permissions and limitations
.. under the License.
-.. _api:
+CSV
+===
-*************
-API Reference
-*************
+Reading a csv is very straightforward with :meth:`.SessionContext.read_csv`
-.. toctree::
- :maxdepth: 2
+.. code-block:: python
- api/config
- api/dataframe
- api/execution_context
- api/expression
- api/functions
- api/object_store
+
+ from datafusion import SessionContext
+
+ ctx = SessionContext()
+ df = ctx.read_csv("file.csv")
+
+An alternative is to use :meth:`.SessionContext.register_csv`
+
+.. code-block:: python
+
+ ctx.register_csv("file", "file.csv")
+ df = ctx.table("file")
\ No newline at end of file
diff --git a/_sources/api.rst.txt b/_sources/user-guide/io/index.rst.txt
similarity index 83%
copy from _sources/api.rst.txt
copy to _sources/user-guide/io/index.rst.txt
index a5d6543..af08240 100644
--- a/_sources/api.rst.txt
+++ b/_sources/user-guide/io/index.rst.txt
@@ -15,18 +15,14 @@
.. specific language governing permissions and limitations
.. under the License.
-.. _api:
-
-*************
-API Reference
-*************
+IO
+==
.. toctree::
:maxdepth: 2
- api/config
- api/dataframe
- api/execution_context
- api/expression
- api/functions
- api/object_store
+ csv
+ parquet
+ json
+ avro
+
diff --git a/_sources/api.rst.txt b/_sources/user-guide/io/json.rst.txt
similarity index 69%
copy from _sources/api.rst.txt
copy to _sources/user-guide/io/json.rst.txt
index a5d6543..5949a03 100644
--- a/_sources/api.rst.txt
+++ b/_sources/user-guide/io/json.rst.txt
@@ -15,18 +15,15 @@
.. specific language governing permissions and limitations
.. under the License.
-.. _api:
+JSON
+====
+`JSON <https://www.json.org/json-en.html>`_ (JavaScript Object Notation) is a lightweight data-interchange format.
+When it comes to reading a JSON file, using :meth:`.SessionContext.read_json` is a simple and easy
-*************
-API Reference
-*************
+.. code-block:: python
-.. toctree::
- :maxdepth: 2
- api/config
- api/dataframe
- api/execution_context
- api/expression
- api/functions
- api/object_store
+ from datafusion import SessionContext
+
+ ctx = SessionContext()
+ df = ctx.read_avro("file.json")
\ No newline at end of file
diff --git a/_sources/api.rst.txt b/_sources/user-guide/io/parquet.rst.txt
similarity index 65%
copy from _sources/api.rst.txt
copy to _sources/user-guide/io/parquet.rst.txt
index a5d6543..78bba30 100644
--- a/_sources/api.rst.txt
+++ b/_sources/user-guide/io/parquet.rst.txt
@@ -15,18 +15,22 @@
.. specific language governing permissions and limitations
.. under the License.
-.. _api:
+Parquet
+=======
-*************
-API Reference
-*************
+It is quite simple to read a parquet file using the :meth:`.SessionContext.read_parquet` function.
-.. toctree::
- :maxdepth: 2
+.. code-block:: python
- api/config
- api/dataframe
- api/execution_context
- api/expression
- api/functions
- api/object_store
+
+ from datafusion import SessionContext
+
+ ctx = SessionContext()
+ df = ctx.read_parquet("file.parquet")
+
+An alternative is to use :meth:`.SessionContext.register_parquet`
+
+.. code-block:: python
+
+ ctx.register_parquet("file", "file.parquet")
+ df = ctx.table("file")
\ No newline at end of file
diff --git a/_sources/api.rst.txt b/_sources/user-guide/sql.rst.txt
similarity index 59%
copy from _sources/api.rst.txt
copy to _sources/user-guide/sql.rst.txt
index a5d6543..6fa7f0c 100644
--- a/_sources/api.rst.txt
+++ b/_sources/user-guide/sql.rst.txt
@@ -15,18 +15,25 @@
.. specific language governing permissions and limitations
.. under the License.
-.. _api:
+SQL
+===
-*************
-API Reference
-*************
+DataFusion also offers a SQL API, read the full reference `here <https://arrow.apache.org/datafusion/user-guide/sql/index.html>`_
-.. toctree::
- :maxdepth: 2
+.. ipython:: python
- api/config
- api/dataframe
- api/execution_context
- api/expression
- api/functions
- api/object_store
+ import datafusion
+ from datafusion import col
+ import pyarrow
+
+ # create a context
+ ctx = datafusion.SessionContext()
+
+ # register a CSV
+ ctx.register_csv('pokemon', 'pokemon.csv')
+
+ # create a new statement via SQL
+ df = ctx.sql('SELECT "Attack"+"Defense", "Attack"-"Defense" FROM pokemon')
+
+ # collect and convert to pandas DataFrame
+ df.to_pandas()
\ No newline at end of file
diff --git a/api.html b/api.html
index 8292c7c..a50fcf7 100644
--- a/api.html
+++ b/api.html
@@ -38,7 +38,7 @@
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="DataFrame" href="api/dataframe.html" />
- <link rel="prev" title="DataFusion in Python" href="index.html" />
+ <link rel="prev" title="Introduction" href="contributor-guide/introduction.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -72,13 +72,153 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="current nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="user-guide/common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="current reference internal" href="#">
API Reference
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
- <label for="toctree-checkbox-1">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -87,8 +227,8 @@
<a class="reference internal" href="api/dataframe.html">
DataFrame
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -104,8 +244,8 @@
<a class="reference internal" href="api/execution_context.html">
SessionContext
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -131,8 +271,8 @@
<a class="reference internal" href="api/expression.html">
Expr
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -148,8 +288,8 @@
<a class="reference internal" href="api/functions.html">
Functions
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -165,8 +305,8 @@
<a class="reference internal" href="api/object_store.html">
ObjectStore
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -263,11 +403,11 @@
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="index.html" title="previous page">
+ <a class='left-prev' id="prev-link" href="contributor-guide/introduction.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">DataFusion in Python</p>
+ <p class="prev-next-title">Introduction</p>
</div>
</a>
<a class='right-next' id="next-link" href="api/dataframe.html" title="next page">
diff --git a/api/dataframe.html b/api/dataframe.html
index 078cf23..f608942 100644
--- a/api/dataframe.html
+++ b/api/dataframe.html
@@ -72,13 +72,153 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="current nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="../api.html">
API Reference
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
- <label for="toctree-checkbox-1">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -87,8 +227,8 @@
<a class="current reference internal" href="#">
DataFrame
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -104,8 +244,8 @@
<a class="reference internal" href="execution_context.html">
SessionContext
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -131,8 +271,8 @@
<a class="reference internal" href="expression.html">
Expr
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -148,8 +288,8 @@
<a class="reference internal" href="functions.html">
Functions
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -165,8 +305,8 @@
<a class="reference internal" href="object_store.html">
ObjectStore
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
diff --git a/api/execution_context.html b/api/execution_context.html
index 0d8c411..d7c01cc 100644
--- a/api/execution_context.html
+++ b/api/execution_context.html
@@ -72,13 +72,153 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="current nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="../api.html">
API Reference
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
- <label for="toctree-checkbox-1">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -87,8 +227,8 @@
<a class="reference internal" href="dataframe.html">
DataFrame
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -104,8 +244,8 @@
<a class="current reference internal" href="#">
SessionContext
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -131,8 +271,8 @@
<a class="reference internal" href="expression.html">
Expr
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -148,8 +288,8 @@
<a class="reference internal" href="functions.html">
Functions
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -165,8 +305,8 @@
<a class="reference internal" href="object_store.html">
ObjectStore
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
diff --git a/api/expression.html b/api/expression.html
index f3d1db9..3cfcbbc 100644
--- a/api/expression.html
+++ b/api/expression.html
@@ -72,13 +72,153 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="current nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="../api.html">
API Reference
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
- <label for="toctree-checkbox-1">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -87,8 +227,8 @@
<a class="reference internal" href="dataframe.html">
DataFrame
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -104,8 +244,8 @@
<a class="reference internal" href="execution_context.html">
SessionContext
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -131,8 +271,8 @@
<a class="current reference internal" href="#">
Expr
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -148,8 +288,8 @@
<a class="reference internal" href="functions.html">
Functions
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -165,8 +305,8 @@
<a class="reference internal" href="object_store.html">
ObjectStore
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
diff --git a/api/functions.html b/api/functions.html
index 922a94f..5b2df91 100644
--- a/api/functions.html
+++ b/api/functions.html
@@ -72,13 +72,153 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="current nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="../api.html">
API Reference
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
- <label for="toctree-checkbox-1">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -87,8 +227,8 @@
<a class="reference internal" href="dataframe.html">
DataFrame
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -104,8 +244,8 @@
<a class="reference internal" href="execution_context.html">
SessionContext
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -131,8 +271,8 @@
<a class="reference internal" href="expression.html">
Expr
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -148,8 +288,8 @@
<a class="current reference internal" href="#">
Functions
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -165,8 +305,8 @@
<a class="reference internal" href="object_store.html">
ObjectStore
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
diff --git a/api/object_store.html b/api/object_store.html
index 8e4ca13..ac36fae 100644
--- a/api/object_store.html
+++ b/api/object_store.html
@@ -72,13 +72,153 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="current nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="../api.html">
API Reference
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
- <label for="toctree-checkbox-1">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -87,8 +227,8 @@
<a class="reference internal" href="dataframe.html">
DataFrame
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -104,8 +244,8 @@
<a class="reference internal" href="execution_context.html">
SessionContext
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -131,8 +271,8 @@
<a class="reference internal" href="expression.html">
Expr
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -148,8 +288,8 @@
<a class="reference internal" href="functions.html">
Functions
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -165,8 +305,8 @@
<a class="current reference internal" href="#">
ObjectStore
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
diff --git a/contributor-guide/introduction.html b/contributor-guide/introduction.html
new file mode 100644
index 0000000..7bba934
--- /dev/null
+++ b/contributor-guide/introduction.html
@@ -0,0 +1,497 @@
+
+<!DOCTYPE html>
+
+<html lang="en">
+ <head>
+ <meta charset="utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
+
+ <title>Introduction — Arrow DataFusion documentation</title>
+
+ <link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+<link href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+
+
+ <link rel="stylesheet"
+ href="../_static/vendor/fontawesome/5.13.0/css/all.min.css">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
+
+
+
+
+
+ <link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
+ <link rel="stylesheet" type="text/css" href="../_static/styles/pydata-sphinx-theme.css" />
+ <link rel="stylesheet" type="text/css" href="../_static/theme_overrides.css" />
+
+ <link rel="preload" as="script" href="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
+
+ <script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
+ <script src="../_static/jquery.js"></script>
+ <script src="../_static/underscore.js"></script>
+ <script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
+ <script src="../_static/doctools.js"></script>
+ <script src="../_static/sphinx_highlight.js"></script>
+ <link rel="index" title="Index" href="../genindex.html" />
+ <link rel="search" title="Search" href="../search.html" />
+ <link rel="next" title="API Reference" href="../api.html" />
+ <link rel="prev" title="SQL" href="../user-guide/sql.html" />
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
+ <meta name="docsearch:language" content="en">
+
+
+ <!-- Google Analytics -->
+
+ </head>
+ <body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
+
+ <div class="container-fluid" id="banner"></div>
+
+
+
+
+ <div class="container-xl">
+ <div class="row">
+
+
+ <!-- Only show if we have sidebars configured, else just a small margin -->
+ <div class="col-12 col-md-3 bd-sidebar">
+ <div class="sidebar-start-items">
+<a class="navbar-brand" href="../index.html">
+ <img src="../_static/images/DataFusion-Logo-Background-White.png" class="logo" alt="logo">
+</a>
+
+<form class="bd-search d-flex align-items-center" action="../search.html" method="get">
+ <i class="icon fas fa-search"></i>
+ <input type="search" class="form-control" name="q" id="search-input" placeholder="Search the docs ..." aria-label="Search the docs ..." autocomplete="off" >
+</form>
+
+<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
+ <div class="bd-toc-item active">
+
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
+ <li class="toctree-l1 current active">
+ <a class="current reference internal" href="#">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../api.html">
+ API Reference
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../api/dataframe.html">
+ DataFrame
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../generated/datafusion.DataFrame.html">
+ datafusion.DataFrame
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../api/execution_context.html">
+ SessionContext
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../generated/datafusion.SessionConfig.html">
+ datafusion.SessionConfig
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../generated/datafusion.RuntimeConfig.html">
+ datafusion.RuntimeConfig
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../generated/datafusion.SessionContext.html">
+ datafusion.SessionContext
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../api/expression.html">
+ Expr
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../generated/datafusion.Expr.html">
+ datafusion.Expr
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../api/functions.html">
+ Functions
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../generated/datafusion.functions.functions.html">
+ datafusion.functions.functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../api/object_store.html">
+ ObjectStore
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../generated/datafusion.object_store.object_store.html">
+ datafusion.object_store.object_store
+ </a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+</ul>
+
+
+ </div>
+</nav>
+ </div>
+ <div class="sidebar-end-items">
+ </div>
+ </div>
+
+
+
+
+ <div class="d-none d-xl-block col-xl-2 bd-toc">
+
+
+ <div class="toc-item">
+
+<div class="tocsection onthispage pt-5 pb-3">
+ <i class="fas fa-list"></i> On this page
+</div>
+
+<nav id="bd-toc-nav">
+ <ul class="visible nav section-nav flex-column">
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#how-to-develop">
+ How to develop
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#running-installing-pre-commit-hooks">
+ Running & Installing pre-commit hooks
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#update-dependencies">
+ Update Dependencies
+ </a>
+ </li>
+</ul>
+
+</nav>
+ </div>
+
+ <div class="toc-item">
+
+
+<div class="tocsection editthispage">
+ <a href="https://github.com/apache/arrow-datafusion-python/edit/main/docs/source/contributor-guide/introduction.rst">
+ <i class="fas fa-pencil-alt"></i> Edit this page
+ </a>
+</div>
+
+ </div>
+
+
+ </div>
+
+
+
+
+
+
+ <main class="col-12 col-md-9 col-xl-7 py-md-5 pl-md-5 pr-md-4 bd-content" role="main">
+
+ <div>
+
+ <section id="introduction">
+<h1>Introduction<a class="headerlink" href="#introduction" title="Permalink to this heading">¶</a></h1>
+<p>We welcome and encourage contributions of all kinds, such as:</p>
+<ol class="arabic simple">
+<li><p>Tickets with issue reports of feature requests</p></li>
+<li><p>Documentation improvements</p></li>
+<li><p>Code, both PR and (especially) PR Review.</p></li>
+</ol>
+<p>In addition to submitting new PRs, we have a healthy tradition of community members reviewing each other’s PRs.
+Doing so is a great way to help the community as well as get more familiar with Rust and the relevant codebases.</p>
+<section id="how-to-develop">
+<h2>How to develop<a class="headerlink" href="#how-to-develop" title="Permalink to this heading">¶</a></h2>
+<p>This assumes that you have rust and cargo installed. We use the workflow recommended by <a class="reference external" href="https://github.com/PyO3/pyo3">pyo3</a> and <a class="reference external" href="https://github.com/PyO3/maturin">maturin</a>.</p>
+<p>Bootstrap:</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="c1"># fetch this repo</span>
+git<span class="w"> </span>clone<span class="w"> </span>git@github.com:apache/arrow-datafusion-python.git
+<span class="c1"># prepare development environment (used to build wheel / install in development)</span>
+python3<span class="w"> </span>-m<span class="w"> </span>venv<span class="w"> </span>venv
+<span class="c1"># activate the venv</span>
+<span class="nb">source</span><span class="w"> </span>venv/bin/activate
+<span class="c1"># update pip itself if necessary</span>
+python<span class="w"> </span>-m<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>-U<span class="w"> </span>pip
+<span class="c1"># install dependencies (for Python 3.8+)</span>
+python<span class="w"> </span>-m<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>-r<span class="w"> </span>requirements-310.txt
+</pre></div>
+</div>
+<p>The tests rely on test data in git submodules.</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>git<span class="w"> </span>submodule<span class="w"> </span>init
+git<span class="w"> </span>submodule<span class="w"> </span>update
+</pre></div>
+</div>
+<p>Whenever rust code changes (your changes or via <cite>git pull</cite>):</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="c1"># make sure you activate the venv using "source venv/bin/activate" first</span>
+maturin<span class="w"> </span>develop
+python<span class="w"> </span>-m<span class="w"> </span>pytest
+</pre></div>
+</div>
+</section>
+<section id="running-installing-pre-commit-hooks">
+<h2>Running & Installing pre-commit hooks<a class="headerlink" href="#running-installing-pre-commit-hooks" title="Permalink to this heading">¶</a></h2>
+<p>arrow-datafusion-python takes advantage of <a class="reference external" href="https://pre-commit.com/">pre-commit</a> to assist developers with code linting to help reduce the number of commits that ultimately fail in CI due to linter errors. Using the pre-commit hooks is optional for the developer but certainly helpful for keeping PRs clean and concise.</p>
+<p>Our pre-commit hooks can be installed by running <code class="code docutils literal notranslate"><span class="pre">pre-commit</span> <span class="pre">install</span></code>, which will install the configurations in your ARROW_DATAFUSION_PYTHON_ROOT/.github directory and run each time you perform a commit, failing to complete the commit if an offending lint is found allowing you to make changes locally before pushing.</p>
+<p>The pre-commit hooks can also be run adhoc without installing them by simply running <code class="code docutils literal notranslate"><span class="pre">pre-commit</span> <span class="pre">run</span> <span class="pre">--all-files</span></code></p>
+</section>
+<section id="update-dependencies">
+<h2>Update Dependencies<a class="headerlink" href="#update-dependencies" title="Permalink to this heading">¶</a></h2>
+<p>To change test dependencies, change the <cite>requirements.in</cite> and run</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="c1"># install pip-tools (this can be done only once), also consider running in venv</span>
+python<span class="w"> </span>-m<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>pip-tools
+python<span class="w"> </span>-m<span class="w"> </span>piptools<span class="w"> </span>compile<span class="w"> </span>--generate-hashes<span class="w"> </span>-o<span class="w"> </span>requirements-310.txt
+</pre></div>
+</div>
+<p>To update dependencies, run with <cite>-U</cite></p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>python<span class="w"> </span>-m<span class="w"> </span>piptools<span class="w"> </span>compile<span class="w"> </span>-U<span class="w"> </span>--generate-hashes<span class="w"> </span>-o<span class="w"> </span>requirements-310.txt
+</pre></div>
+</div>
+<p>More details about pip-tools <a class="reference external" href="https://github.com/jazzband/pip-tools">here</a></p>
+</section>
+</section>
+
+
+ </div>
+
+
+ <!-- Previous / next buttons -->
+<div class='prev-next-area'>
+ <a class='left-prev' id="prev-link" href="../user-guide/sql.html" title="previous page">
+ <i class="fas fa-angle-left"></i>
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">previous</p>
+ <p class="prev-next-title">SQL</p>
+ </div>
+ </a>
+ <a class='right-next' id="next-link" href="../api.html" title="next page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">API Reference</p>
+ </div>
+ <i class="fas fa-angle-right"></i>
+ </a>
+</div>
+
+ </main>
+
+
+ </div>
+ </div>
+
+ <script src="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
+ <footer class="footer mt-5 mt-md-0">
+ <div class="container">
+
+ <div class="footer-item">
+ <p class="copyright">
+ © Copyright 2022, Apache Software Foundation.<br>
+</p>
+ </div>
+
+ <div class="footer-item">
+ <p class="sphinx-version">
+Created using <a href="http://sphinx-doc.org/">Sphinx</a> 5.3.0.<br>
+</p>
+ </div>
+
+ </div>
+</footer>
+ </body>
+</html>
\ No newline at end of file
diff --git a/generated/datafusion.DataFrame.html b/generated/datafusion.DataFrame.html
index 0323a1e..1b81fb0 100644
--- a/generated/datafusion.DataFrame.html
+++ b/generated/datafusion.DataFrame.html
@@ -72,13 +72,153 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="current nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="../api.html">
API Reference
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
- <label for="toctree-checkbox-1">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -87,8 +227,8 @@
<a class="reference internal" href="../api/dataframe.html">
DataFrame
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -104,8 +244,8 @@
<a class="reference internal" href="../api/execution_context.html">
SessionContext
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -131,8 +271,8 @@
<a class="reference internal" href="../api/expression.html">
Expr
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -148,8 +288,8 @@
<a class="reference internal" href="../api/functions.html">
Functions
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -165,8 +305,8 @@
<a class="reference internal" href="../api/object_store.html">
ObjectStore
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -689,7 +829,7 @@ The actual execution of a plan runs natively on Rust and Arrow on a multi-thread
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.DataFrame.write_json" title="datafusion.DataFrame.write_json"><code class="xref py py-obj docutils literal notranslate"><span class="pre">write_json</span></code></a>(path)</p></td>
<td><p>Executes a query and writes the results to a partitioned JSON file.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.DataFrame.write_parquet" title="datafusion.DataFrame.write_parquet"><code class="xref py py-obj docutils literal notranslate"><span class="pre">write_parquet</span></code></a>(path)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.DataFrame.write_parquet" title="datafusion.DataFrame.write_parquet"><code class="xref py py-obj docutils literal notranslate"><span class="pre">write_parquet</span></code></a>(path[, compression, ...])</p></td>
<td><p>Write a <cite>DataFrame</cite> to a Parquet file.</p></td>
</tr>
</tbody>
@@ -902,7 +1042,7 @@ renamed does not exist.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.DataFrame.write_parquet">
-<span class="sig-name descname"><span class="pre">write_parquet</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">path</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.DataFrame.write_parquet" title="Permalink to this definition">¶</a></dt>
+<span class="sig-name descname"><span class="pre">write_parquet</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">compression</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'uncompressed'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">compression_level</span></span><spa [...]
<dd><p>Write a <cite>DataFrame</cite> to a Parquet file.</p>
</dd></dl>
diff --git a/generated/datafusion.Expr.html b/generated/datafusion.Expr.html
index 9b259e1..67fbbc1 100644
--- a/generated/datafusion.Expr.html
+++ b/generated/datafusion.Expr.html
@@ -72,13 +72,153 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="current nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="../api.html">
API Reference
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
- <label for="toctree-checkbox-1">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -87,8 +227,8 @@
<a class="reference internal" href="../api/dataframe.html">
DataFrame
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -104,8 +244,8 @@
<a class="reference internal" href="../api/execution_context.html">
SessionContext
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -131,8 +271,8 @@
<a class="reference internal" href="../api/expression.html">
Expr
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -148,8 +288,8 @@
<a class="reference internal" href="../api/functions.html">
Functions
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -165,8 +305,8 @@
<a class="reference internal" href="../api/object_store.html">
ObjectStore
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
diff --git a/generated/datafusion.RuntimeConfig.html b/generated/datafusion.RuntimeConfig.html
index 3c76756..c175fb3 100644
--- a/generated/datafusion.RuntimeConfig.html
+++ b/generated/datafusion.RuntimeConfig.html
@@ -72,13 +72,153 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="current nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="../api.html">
API Reference
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
- <label for="toctree-checkbox-1">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -87,8 +227,8 @@
<a class="reference internal" href="../api/dataframe.html">
DataFrame
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -104,8 +244,8 @@
<a class="reference internal" href="../api/execution_context.html">
SessionContext
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -131,8 +271,8 @@
<a class="reference internal" href="../api/expression.html">
Expr
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -148,8 +288,8 @@
<a class="reference internal" href="../api/functions.html">
Functions
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -165,8 +305,8 @@
<a class="reference internal" href="../api/object_store.html">
ObjectStore
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
diff --git a/generated/datafusion.SessionConfig.html b/generated/datafusion.SessionConfig.html
index dc3dc20..f3ed915 100644
--- a/generated/datafusion.SessionConfig.html
+++ b/generated/datafusion.SessionConfig.html
@@ -72,13 +72,153 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="current nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="../api.html">
API Reference
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
- <label for="toctree-checkbox-1">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -87,8 +227,8 @@
<a class="reference internal" href="../api/dataframe.html">
DataFrame
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -104,8 +244,8 @@
<a class="reference internal" href="../api/execution_context.html">
SessionContext
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -131,8 +271,8 @@
<a class="reference internal" href="../api/expression.html">
Expr
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -148,8 +288,8 @@
<a class="reference internal" href="../api/functions.html">
Functions
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -165,8 +305,8 @@
<a class="reference internal" href="../api/object_store.html">
ObjectStore
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
diff --git a/generated/datafusion.SessionContext.html b/generated/datafusion.SessionContext.html
index 6fc3a5c..aaac844 100644
--- a/generated/datafusion.SessionContext.html
+++ b/generated/datafusion.SessionContext.html
@@ -72,13 +72,153 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="current nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="../api.html">
API Reference
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
- <label for="toctree-checkbox-1">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -87,8 +227,8 @@
<a class="reference internal" href="../api/dataframe.html">
DataFrame
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -104,8 +244,8 @@
<a class="reference internal" href="../api/execution_context.html">
SessionContext
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -131,8 +271,8 @@
<a class="reference internal" href="../api/expression.html">
Expr
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -148,8 +288,8 @@
<a class="reference internal" href="../api/functions.html">
Functions
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -165,8 +305,8 @@
<a class="reference internal" href="../api/object_store.html">
ObjectStore
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -357,6 +497,24 @@
</code>
</a>
</li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.SessionContext.read_table">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ SessionContext.read_table()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.SessionContext.register_avro">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ SessionContext.register_avro()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.SessionContext.register_csv">
<code class="docutils literal notranslate">
@@ -375,6 +533,15 @@
</code>
</a>
</li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.SessionContext.register_json">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ SessionContext.register_json()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.SessionContext.register_object_store">
<code class="docutils literal notranslate">
@@ -569,43 +736,52 @@ multi-threaded execution engine to perform the execution.</p>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.SessionContext.read_parquet" title="datafusion.SessionContext.read_parquet"><code class="xref py py-obj docutils literal notranslate"><span class="pre">read_parquet</span></code></a>(path[, table_partition_cols, ...])</p></td>
<td><p></p></td>
</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.SessionContext.read_table" title="datafusion.SessionContext.read_table"><code class="xref py py-obj docutils literal notranslate"><span class="pre">read_table</span></code></a>(table)</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_avro" title="datafusion.SessionContext.register_avro"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_avro</span></code></a>(name, path[, schema, ...])</p></td>
+<td><p></p></td>
+</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_csv" title="datafusion.SessionContext.register_csv"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_csv</span></code></a>(name, path[, schema, ...])</p></td>
<td><p></p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_dataset" title="datafusion.SessionContext.register_dataset"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_dataset</span></code></a>(name, dataset)</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_object_store" title="datafusion.SessionContext.register_object_store"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_object_store</span></code></a>(scheme, store[, host])</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_json" title="datafusion.SessionContext.register_json"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_json</span></code></a>(name, path[, schema, ...])</p></td>
+<td><p></p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_object_store" title="datafusion.SessionContext.register_object_store"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_object_store</span></code></a>(scheme, store[, host])</p></td>
<td><p>Register a an object store with the given name</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_parquet" title="datafusion.SessionContext.register_parquet"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_parquet</span></code></a>(name, path[, ...])</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_parquet" title="datafusion.SessionContext.register_parquet"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_parquet</span></code></a>(name, path[, ...])</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_record_batches" title="datafusion.SessionContext.register_record_batches"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_record_batches</span></code></a>(name, partitions)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_record_batches" title="datafusion.SessionContext.register_record_batches"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_record_batches</span></code></a>(name, partitions)</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_table" title="datafusion.SessionContext.register_table"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_table</span></code></a>(name, table)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_table" title="datafusion.SessionContext.register_table"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_table</span></code></a>(name, table)</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_udaf" title="datafusion.SessionContext.register_udaf"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_udaf</span></code></a>(udaf)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_udaf" title="datafusion.SessionContext.register_udaf"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_udaf</span></code></a>(udaf)</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_udf" title="datafusion.SessionContext.register_udf"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_udf</span></code></a>(udf)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.SessionContext.register_udf" title="datafusion.SessionContext.register_udf"><code class="xref py py-obj docutils literal notranslate"><span class="pre">register_udf</span></code></a>(udf)</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.SessionContext.session_id" title="datafusion.SessionContext.session_id"><code class="xref py py-obj docutils literal notranslate"><span class="pre">session_id</span></code></a>()</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.SessionContext.session_id" title="datafusion.SessionContext.session_id"><code class="xref py py-obj docutils literal notranslate"><span class="pre">session_id</span></code></a>()</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.SessionContext.sql" title="datafusion.SessionContext.sql"><code class="xref py py-obj docutils literal notranslate"><span class="pre">sql</span></code></a>(query)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.SessionContext.sql" title="datafusion.SessionContext.sql"><code class="xref py py-obj docutils literal notranslate"><span class="pre">sql</span></code></a>(query)</p></td>
<td><p>Returns a PyDataFrame whose plan corresponds to the SQL statement.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.SessionContext.table" title="datafusion.SessionContext.table"><code class="xref py py-obj docutils literal notranslate"><span class="pre">table</span></code></a>(name)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.SessionContext.table" title="datafusion.SessionContext.table"><code class="xref py py-obj docutils literal notranslate"><span class="pre">table</span></code></a>(name)</p></td>
<td><p></p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.SessionContext.table_exist" title="datafusion.SessionContext.table_exist"><code class="xref py py-obj docutils literal notranslate"><span class="pre">table_exist</span></code></a>(name)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.SessionContext.table_exist" title="datafusion.SessionContext.table_exist"><code class="xref py py-obj docutils literal notranslate"><span class="pre">table_exist</span></code></a>(name)</p></td>
<td><p></p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.SessionContext.tables" title="datafusion.SessionContext.tables"><code class="xref py py-obj docutils literal notranslate"><span class="pre">tables</span></code></a>()</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.SessionContext.tables" title="datafusion.SessionContext.tables"><code class="xref py py-obj docutils literal notranslate"><span class="pre">tables</span></code></a>()</p></td>
<td><p></p></td>
</tr>
</tbody>
@@ -679,22 +855,32 @@ multi-threaded execution engine to perform the execution.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.SessionContext.read_csv">
-<span class="sig-name descname"><span class="pre">read_csv</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">schema</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">has_header</span></span><span class="o"><span class="pr [...]
+<span class="sig-name descname"><span class="pre">read_csv</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">schema</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">has_header</span></span><span class="o"><span class="pr [...]
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.SessionContext.read_json">
-<span class="sig-name descname"><span class="pre">read_json</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">schema</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">schema_infer_max_records</span></span><span class="o"> [...]
+<span class="sig-name descname"><span class="pre">read_json</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">schema</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">schema_infer_max_records</span></span><span class="o"> [...]
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.SessionContext.read_parquet">
-<span class="sig-name descname"><span class="pre">read_parquet</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">table_partition_cols</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">Ellipsis</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">parquet_pruning</span></span><spa [...]
+<span class="sig-name descname"><span class="pre">read_parquet</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">table_partition_cols</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">Ellipsis</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">parquet_pruning</span></span><spa [...]
+<dd></dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.SessionContext.read_table">
+<span class="sig-name descname"><span class="pre">read_table</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">table</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.SessionContext.read_table" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.SessionContext.register_avro">
+<span class="sig-name descname"><span class="pre">register_avro</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">schema</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><sp [...]
<dd></dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.SessionContext.register_csv">
-<span class="sig-name descname"><span class="pre">register_csv</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">schema</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><spa [...]
+<span class="sig-name descname"><span class="pre">register_csv</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">schema</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><spa [...]
<dd></dd></dl>
<dl class="py method">
@@ -702,6 +888,11 @@ multi-threaded execution engine to perform the execution.</p>
<span class="sig-name descname"><span class="pre">register_dataset</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dataset</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.SessionContext.register_dataset" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.SessionContext.register_json">
+<span class="sig-name descname"><span class="pre">register_json</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">schema</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><sp [...]
+<dd></dd></dl>
+
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.SessionContext.register_object_store">
<span class="sig-name descname"><span class="pre">register_object_store</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">scheme</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">store</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">host</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-p [...]
@@ -710,7 +901,7 @@ multi-threaded execution engine to perform the execution.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.SessionContext.register_parquet">
-<span class="sig-name descname"><span class="pre">register_parquet</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">table_partition_cols</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">Ellipsis</span></span></em>, <em [...]
+<span class="sig-name descname"><span class="pre">register_parquet</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">table_partition_cols</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">Ellipsis</span></span></em>, <em [...]
<dd></dd></dl>
<dl class="py method">
diff --git a/generated/datafusion.functions.functions.html b/generated/datafusion.functions.functions.html
index 56e506f..9fbeecd 100644
--- a/generated/datafusion.functions.functions.html
+++ b/generated/datafusion.functions.functions.html
@@ -72,13 +72,153 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="current nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="../api.html">
API Reference
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
- <label for="toctree-checkbox-1">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -87,8 +227,8 @@
<a class="reference internal" href="../api/dataframe.html">
DataFrame
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -104,8 +244,8 @@
<a class="reference internal" href="../api/execution_context.html">
SessionContext
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -131,8 +271,8 @@
<a class="reference internal" href="../api/expression.html">
Expr
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -148,8 +288,8 @@
<a class="reference internal" href="../api/functions.html">
Functions
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -165,8 +305,8 @@
<a class="reference internal" href="../api/object_store.html">
ObjectStore
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -222,6 +362,15 @@
</code>
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.acosh">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ acosh()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.functions.functions.alias">
<code class="docutils literal notranslate">
@@ -312,6 +461,15 @@
</code>
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.asinh">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ asinh()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.functions.functions.atan">
<code class="docutils literal notranslate">
@@ -330,6 +488,15 @@
</code>
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.atanh">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ atanh()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.functions.functions.avg">
<code class="docutils literal notranslate">
@@ -357,6 +524,24 @@
</code>
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.case">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ case()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.cbrt">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ cbrt()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.functions.functions.ceil">
<code class="docutils literal notranslate">
@@ -447,6 +632,15 @@
</code>
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.cosh">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ cosh()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.functions.functions.count">
<code class="docutils literal notranslate">
@@ -555,6 +749,15 @@
</code>
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.degrees">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ degrees()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.functions.functions.digest">
<code class="docutils literal notranslate">
@@ -573,6 +776,15 @@
</code>
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.factorial">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ factorial()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.functions.functions.floor">
<code class="docutils literal notranslate">
@@ -591,6 +803,15 @@
</code>
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.gcd">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ gcd()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.functions.functions.grouping">
<code class="docutils literal notranslate">
@@ -618,6 +839,33 @@
</code>
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.isnan">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ isnan()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.iszero">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ iszero()
+ </span>
+ </code>
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.lcm">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ lcm()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.functions.functions.left">
<code class="docutils literal notranslate">
@@ -753,6 +1001,15 @@
</code>
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.nanvl">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ nanvl()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.functions.functions.now">
<code class="docutils literal notranslate">
@@ -789,6 +1046,15 @@
</code>
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.pi">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ pi()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.functions.functions.pow">
<code class="docutils literal notranslate">
@@ -807,6 +1073,15 @@
</code>
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.radians">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ radians()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.functions.functions.random">
<code class="docutils literal notranslate">
@@ -951,6 +1226,15 @@
</code>
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.sinh">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ sinh()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.functions.functions.split_part">
<code class="docutils literal notranslate">
@@ -1050,6 +1334,15 @@
</code>
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.functions.functions.tanh">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ tanh()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.functions.functions.to_hex">
<code class="docutils literal notranslate">
@@ -1216,6 +1509,11 @@
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">acos</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.acos" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.acosh">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">acosh</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.acosh" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.functions.alias">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">alias</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expr</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">name</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.alias" title="Permalink to this defi [...]
@@ -1268,6 +1566,11 @@
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">asin</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.asin" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.asinh">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">asinh</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.asinh" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.functions.atan">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">atan</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.atan" title="Permalink to this definition">¶</a></dt>
@@ -1278,6 +1581,11 @@
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">atan2</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.atan2" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.atanh">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">atanh</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.atanh" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.functions.avg">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">avg</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">distinct</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span [...]
@@ -1295,6 +1603,17 @@
<dd><p>Removes the longest string containing only characters in characters (a space by default) from the start and end of string.</p>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.case">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">case</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expr</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.case" title="Permalink to this definition">¶</a></dt>
+<dd><p>Create a CASE WHEN statement with literal WHEN expressions for comparison to the base expression.</p>
+</dd></dl>
+
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.cbrt">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">cbrt</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.cbrt" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.functions.ceil">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">ceil</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.ceil" title="Permalink to this definition">¶</a></dt>
@@ -1353,6 +1672,11 @@ Other NULL arguments are ignored.</p>
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">cos</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.cos" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.cosh">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">cosh</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.cosh" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.functions.count">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">count</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">distinct</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><sp [...]
@@ -1414,6 +1738,11 @@ Other NULL arguments are ignored.</p>
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">datetrunc</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.datetrunc" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.degrees">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">degrees</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.degrees" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.functions.digest">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">digest</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">value</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">method</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.digest" title="Permalink to this [...]
@@ -1426,6 +1755,11 @@ Standard algorithms are md5, sha224, sha256, sha384, sha512, blake2s, blake2b, a
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">exp</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.exp" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.factorial">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">factorial</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.factorial" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.functions.floor">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">floor</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.floor" title="Permalink to this definition">¶</a></dt>
@@ -1436,6 +1770,11 @@ Standard algorithms are md5, sha224, sha256, sha384, sha512, blake2s, blake2b, a
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">from_unixtime</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.from_unixtime" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.gcd">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">gcd</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.gcd" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.functions.grouping">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">grouping</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">distinct</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"> [...]
@@ -1452,6 +1791,21 @@ Standard algorithms are md5, sha224, sha256, sha384, sha512, blake2s, blake2b, a
<dd><p>Converts the first letter of each word to upper case and the rest to lower case. Words are sequences of alphanumeric characters separated by non-alphanumeric characters.</p>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.isnan">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">isnan</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.isnan" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.iszero">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">iszero</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.iszero" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.lcm">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">lcm</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.lcm" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.functions.left">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">left</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.left" title="Permalink to this definition">¶</a></dt>
@@ -1532,6 +1886,12 @@ Standard algorithms are md5, sha224, sha256, sha384, sha512, blake2s, blake2b, a
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">min</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">distinct</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span [...]
<dd></dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.nanvl">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">nanvl</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.nanvl" title="Permalink to this definition">¶</a></dt>
+<dd><p>Returns x if x is not NaN otherwise returns y.</p>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.functions.now">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">now</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.now" title="Permalink to this definition">¶</a></dt>
@@ -1554,6 +1914,11 @@ Standard algorithms are md5, sha224, sha256, sha384, sha512, blake2s, blake2b, a
<dd><p>Creates a new Sort Expr</p>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.pi">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">pi</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.pi" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.functions.pow">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">pow</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.pow" title="Permalink to this definition">¶</a></dt>
@@ -1564,6 +1929,11 @@ Standard algorithms are md5, sha224, sha256, sha384, sha512, blake2s, blake2b, a
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">power</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.power" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.radians">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">radians</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.radians" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.functions.random">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">random</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.random" title="Permalink to this definition">¶</a></dt>
@@ -1651,6 +2021,11 @@ Standard algorithms are md5, sha224, sha256, sha384, sha512, blake2s, blake2b, a
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">sin</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.sin" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.sinh">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">sinh</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.sinh" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.functions.split_part">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">split_part</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.split_part" title="Permalink to this definition">¶</a></dt>
@@ -1710,6 +2085,11 @@ Standard algorithms are md5, sha224, sha256, sha384, sha512, blake2s, blake2b, a
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">tan</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.tan" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.functions.tanh">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">tanh</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.tanh" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.functions.to_hex">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.functions.</span></span><span class="sig-name descname"><span class="pre">to_hex</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#datafusion.functions.functions.to_hex" title="Permalink to this definition">¶</a></dt>
diff --git a/generated/datafusion.object_store.object_store.html b/generated/datafusion.object_store.object_store.html
index cc0be25..a9a0616 100644
--- a/generated/datafusion.object_store.object_store.html
+++ b/generated/datafusion.object_store.object_store.html
@@ -71,13 +71,153 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="current nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="../api.html">
API Reference
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
- <label for="toctree-checkbox-1">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -86,8 +226,8 @@
<a class="reference internal" href="../api/dataframe.html">
DataFrame
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -103,8 +243,8 @@
<a class="reference internal" href="../api/execution_context.html">
SessionContext
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -130,8 +270,8 @@
<a class="reference internal" href="../api/expression.html">
Expr
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -147,8 +287,8 @@
<a class="reference internal" href="../api/functions.html">
Functions
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -164,8 +304,8 @@
<a class="reference internal" href="../api/object_store.html">
ObjectStore
</a>
- <input checked="" class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
diff --git a/genindex.html b/genindex.html
index 3b74a3a..f7bf0af 100644
--- a/genindex.html
+++ b/genindex.html
@@ -69,23 +69,163 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
<li class="toctree-l1 has-children">
- <a class="reference internal" href="api.html">
- API Reference
+ <a class="reference internal" href="user-guide/common-operations/index.html">
+ Common Operations
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
<i class="fas fa-chevron-down">
</i>
</label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="api.html">
+ API Reference
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
<ul>
<li class="toctree-l2 has-children">
<a class="reference internal" href="api/dataframe.html">
DataFrame
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -101,8 +241,8 @@
<a class="reference internal" href="api/execution_context.html">
SessionContext
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -128,8 +268,8 @@
<a class="reference internal" href="api/expression.html">
Expr
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -145,8 +285,8 @@
<a class="reference internal" href="api/functions.html">
Functions
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -162,8 +302,8 @@
<a class="reference internal" href="api/object_store.html">
ObjectStore
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -254,6 +394,8 @@
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.abs">abs() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.acos">acos() (in module datafusion.functions.functions)</a>
+</li>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.acosh">acosh() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.DataFrame.html#datafusion.DataFrame.aggregate">aggregate() (datafusion.DataFrame method)</a>
</li>
@@ -284,10 +426,14 @@
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.ascii">ascii() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.asin">asin() (in module datafusion.functions.functions)</a>
+</li>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.asinh">asinh() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.atan">atan() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.atan2">atan2() (in module datafusion.functions.functions)</a>
+</li>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.atanh">atanh() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.avg">avg() (in module datafusion.functions.functions)</a>
</li>
@@ -312,10 +458,14 @@
<li><a href="generated/datafusion.DataFrame.html#datafusion.DataFrame.cache">cache() (datafusion.DataFrame method)</a>
</li>
<li><a href="generated/datafusion.Expr.html#datafusion.Expr.canonical_name">canonical_name() (datafusion.Expr method)</a>
+</li>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.case">case() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.Expr.html#datafusion.Expr.cast">cast() (datafusion.Expr method)</a>
</li>
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.catalog">catalog() (datafusion.SessionContext method)</a>
+</li>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.cbrt">cbrt() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.ceil">ceil() (in module datafusion.functions.functions)</a>
</li>
@@ -334,11 +484,11 @@
<li><a href="generated/datafusion.DataFrame.html#datafusion.DataFrame.collect_partitioned">collect_partitioned() (datafusion.DataFrame method)</a>
</li>
<li><a href="generated/datafusion.Expr.html#datafusion.Expr.column">column() (datafusion.Expr static method)</a>
-</li>
- <li><a href="generated/datafusion.Expr.html#datafusion.Expr.column_name">column_name() (datafusion.Expr method)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="generated/datafusion.Expr.html#datafusion.Expr.column_name">column_name() (datafusion.Expr method)</a>
+</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.concat">concat() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.concat_ws">concat_ws() (in module datafusion.functions.functions)</a>
@@ -346,6 +496,8 @@
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.corr">corr() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.cos">cos() (in module datafusion.functions.functions)</a>
+</li>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.cosh">cosh() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.DataFrame.html#datafusion.DataFrame.count">count() (datafusion.DataFrame method)</a>
@@ -395,13 +547,15 @@
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.date_part">date_part() (in module datafusion.functions.functions)</a>
</li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.date_trunc">date_trunc() (in module datafusion.functions.functions)</a>
</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.datepart">datepart() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.datetrunc">datetrunc() (in module datafusion.functions.functions)</a>
+</li>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.degrees">degrees() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.deregister_table">deregister_table() (datafusion.SessionContext method)</a>
</li>
@@ -441,16 +595,18 @@
<h2 id="F">F</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.factorial">factorial() (in module datafusion.functions.functions)</a>
+</li>
<li><a href="generated/datafusion.DataFrame.html#datafusion.DataFrame.filter">filter() (datafusion.DataFrame method)</a>
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.floor">floor() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.from_arrow_table">from_arrow_table() (datafusion.SessionContext method)</a>
-</li>
- <li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.from_pandas">from_pandas() (datafusion.SessionContext method)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.from_pandas">from_pandas() (datafusion.SessionContext method)</a>
+</li>
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.from_polars">from_polars() (datafusion.SessionContext method)</a>
</li>
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.from_pydict">from_pydict() (datafusion.SessionContext method)</a>
@@ -465,10 +621,12 @@
<h2 id="G">G</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
- <li><a href="generated/datafusion.object_store.object_store.html#datafusion.object_store.object_store.GoogleCloud">GoogleCloud (class in datafusion.object_store.object_store)</a>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.gcd">gcd() (in module datafusion.functions.functions)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="generated/datafusion.object_store.object_store.html#datafusion.object_store.object_store.GoogleCloud">GoogleCloud (class in datafusion.object_store.object_store)</a>
+</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.grouping">grouping() (in module datafusion.functions.functions)</a>
</li>
</ul></td>
@@ -481,11 +639,15 @@
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.initcap">initcap() (in module datafusion.functions.functions)</a>
</li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="generated/datafusion.DataFrame.html#datafusion.DataFrame.intersect">intersect() (datafusion.DataFrame method)</a>
</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="generated/datafusion.Expr.html#datafusion.Expr.is_null">is_null() (datafusion.Expr method)</a>
+</li>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.isnan">isnan() (in module datafusion.functions.functions)</a>
+</li>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.iszero">iszero() (in module datafusion.functions.functions)</a>
</li>
</ul></td>
</tr></table>
@@ -501,6 +663,8 @@
<h2 id="L">L</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.lcm">lcm() (in module datafusion.functions.functions)</a>
+</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.left">left() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.length">length() (in module datafusion.functions.functions)</a>
@@ -566,10 +730,12 @@
<h2 id="N">N</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
- <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.now">now() (in module datafusion.functions.functions)</a>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.nanvl">nanvl() (in module datafusion.functions.functions)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.now">now() (in module datafusion.functions.functions)</a>
+</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.nullif">nullif() (in module datafusion.functions.functions)</a>
</li>
</ul></td>
@@ -592,6 +758,8 @@
<h2 id="P">P</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.pi">pi() (in module datafusion.functions.functions)</a>
+</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.pow">pow() (in module datafusion.functions.functions)</a>
</li>
</ul></td>
@@ -606,6 +774,8 @@
<h2 id="R">R</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.radians">radians() (in module datafusion.functions.functions)</a>
+</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.random">random() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.read_avro">read_avro() (datafusion.SessionContext method)</a>
@@ -615,14 +785,20 @@
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.read_json">read_json() (datafusion.SessionContext method)</a>
</li>
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.read_parquet">read_parquet() (datafusion.SessionContext method)</a>
+</li>
+ <li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.read_table">read_table() (datafusion.SessionContext method)</a>
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.regexp_match">regexp_match() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.regexp_replace">regexp_replace() (in module datafusion.functions.functions)</a>
+</li>
+ <li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.register_avro">register_avro() (datafusion.SessionContext method)</a>
</li>
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.register_csv">register_csv() (datafusion.SessionContext method)</a>
</li>
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.register_dataset">register_dataset() (datafusion.SessionContext method)</a>
+</li>
+ <li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.register_json">register_json() (datafusion.SessionContext method)</a>
</li>
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.register_object_store">register_object_store() (datafusion.SessionContext method)</a>
</li>
@@ -630,12 +806,12 @@
</li>
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.register_record_batches">register_record_batches() (datafusion.SessionContext method)</a>
</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.register_table">register_table() (datafusion.SessionContext method)</a>
</li>
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.register_udaf">register_udaf() (datafusion.SessionContext method)</a>
</li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.register_udf">register_udf() (datafusion.SessionContext method)</a>
</li>
<li><a href="generated/datafusion.DataFrame.html#datafusion.DataFrame.repartition">repartition() (datafusion.DataFrame method)</a>
@@ -695,10 +871,12 @@
<li><a href="generated/datafusion.DataFrame.html#datafusion.DataFrame.show">show() (datafusion.DataFrame method)</a>
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.signum">signum() (in module datafusion.functions.functions)</a>
+</li>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.sin">sin() (in module datafusion.functions.functions)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
- <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.sin">sin() (in module datafusion.functions.functions)</a>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.sinh">sinh() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.DataFrame.html#datafusion.DataFrame.sort">sort() (datafusion.DataFrame method)</a>
@@ -741,6 +919,8 @@
<li><a href="generated/datafusion.SessionContext.html#datafusion.SessionContext.tables">tables() (datafusion.SessionContext method)</a>
</li>
<li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.tan">tan() (in module datafusion.functions.functions)</a>
+</li>
+ <li><a href="generated/datafusion.functions.functions.html#datafusion.functions.functions.tanh">tanh() (in module datafusion.functions.functions)</a>
</li>
<li><a href="generated/datafusion.DataFrame.html#datafusion.DataFrame.to_arrow_table">to_arrow_table() (datafusion.DataFrame method)</a>
</li>
diff --git a/index.html b/index.html
index e4d4566..298202b 100644
--- a/index.html
+++ b/index.html
@@ -37,7 +37,7 @@
<script src="_static/sphinx_highlight.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
- <link rel="next" title="API Reference" href="api.html" />
+ <link rel="next" title="Introduction" href="user-guide/introduction.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -71,23 +71,163 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
<li class="toctree-l1 has-children">
- <a class="reference internal" href="api.html">
- API Reference
+ <a class="reference internal" href="user-guide/common-operations/index.html">
+ Common Operations
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
<i class="fas fa-chevron-down">
</i>
</label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="api.html">
+ API Reference
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
<ul>
<li class="toctree-l2 has-children">
<a class="reference internal" href="api/dataframe.html">
DataFrame
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -103,8 +243,8 @@
<a class="reference internal" href="api/execution_context.html">
SessionContext
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -130,8 +270,8 @@
<a class="reference internal" href="api/expression.html">
Expr
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -147,8 +287,8 @@
<a class="reference internal" href="api/functions.html">
Functions
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -164,8 +304,8 @@
<a class="reference internal" href="api/object_store.html">
ObjectStore
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -204,41 +344,16 @@
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#how-to-use-it">
- How to use it
- </a>
- <ul class="nav section-nav flex-column">
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#udfs">
- UDFs
- </a>
- </li>
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#udaf">
- UDAF
- </a>
- </li>
- </ul>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#how-to-install-from-pip">
- How to install (from pip)
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#how-to-develop">
- How to develop
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#how-to-update-dependencies">
- How to update dependencies
+ <a class="reference internal nav-link" href="#install">
+ Install
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#api-reference">
- API reference
+ <a class="reference internal nav-link" href="#example">
+ Example
</a>
+ <ul class="nav section-nav flex-column">
+ </ul>
</li>
</ul>
@@ -276,233 +391,58 @@
<p>The major advantage of this library over other execution engines is that this library achieves zero-copy between Python and its execution engine: there is no cost in using UDFs, UDAFs, and collecting the results to Python apart from having to lock the GIL when running those operations.</p>
<p>Its query engine, DataFusion, is written in <a class="reference external" href="https://www.rust-lang.org">Rust</a>, which makes strong assumptions about thread safety and lack of memory leaks.</p>
<p>Technically, zero-copy is achieved via the <a class="reference external" href="https://arrow.apache.org/docs/format/CDataInterface.html">c data interface</a>.</p>
-<section id="how-to-use-it">
-<h2>How to use it<a class="headerlink" href="#how-to-use-it" title="Permalink to this heading">¶</a></h2>
-<p>Simple usage:</p>
-<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">datafusion</span>
-<span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">col</span>
-<span class="kn">import</span> <span class="nn">pyarrow</span>
-
-<span class="c1"># create a context</span>
-<span class="n">ctx</span> <span class="o">=</span> <span class="n">datafusion</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
-
-<span class="c1"># create a RecordBatch and a new DataFrame from it</span>
-<span class="n">batch</span> <span class="o">=</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">RecordBatch</span><span class="o">.</span><span class="n">from_arrays</span><span class="p">(</span>
- <span class="p">[</span><span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]),</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span [...]
- <span class="n">names</span><span class="o">=</span><span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">],</span>
-<span class="p">)</span>
-<span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">create_dataframe</span><span class="p">([[</span><span class="n">batch</span><span class="p">]])</span>
-
-<span class="c1"># create a new statement</span>
-<span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
- <span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">+</span> <span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span>
- <span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">-</span> <span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span>
-<span class="p">)</span>
-
-<span class="c1"># execute and collect the first (and only) batch</span>
-<span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">collect</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
-
-<span class="k">assert</span> <span class="n">result</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">5</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">9</span><span class="p">])</span>
-<span class="k">assert</span> <span class="n">result</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">==</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="o">-</span>< [...]
-</pre></div>
-</div>
-<p>We can also execute a query against data stored in CSV</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nb">echo</span><span class="w"> </span><span class="s2">"a,b\n1,4\n2,5\n3,6"</span><span class="w"> </span>><span class="w"> </span>example.csv
-</pre></div>
-</div>
-<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">datafusion</span>
-<span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">col</span>
-<span class="kn">import</span> <span class="nn">pyarrow</span>
-
-<span class="c1"># create a context</span>
-<span class="n">ctx</span> <span class="o">=</span> <span class="n">datafusion</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
-
-<span class="c1"># register a CSV</span>
-<span class="n">ctx</span><span class="o">.</span><span class="n">register_csv</span><span class="p">(</span><span class="s1">'example'</span><span class="p">,</span> <span class="s1">'example.csv'</span><span class="p">)</span>
-
-<span class="c1"># create a new statement</span>
-<span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">table</span><span class="p">(</span><span class="s1">'example'</span><span class="p">)</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
- <span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">+</span> <span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span>
- <span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">-</span> <span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span>
-<span class="p">)</span>
-
-<span class="c1"># execute and collect the first (and only) batch</span>
-<span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">collect</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
-
-<span class="k">assert</span> <span class="n">result</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">5</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">9</span><span class="p">])</span>
-<span class="k">assert</span> <span class="n">result</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">==</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="o">-</span>< [...]
-</pre></div>
-</div>
-<p>And how to execute a query against a CSV using SQL:</p>
-<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">datafusion</span>
-<span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">col</span>
-<span class="kn">import</span> <span class="nn">pyarrow</span>
-
-<span class="c1"># create a context</span>
-<span class="n">ctx</span> <span class="o">=</span> <span class="n">datafusion</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
-
-<span class="c1"># register a CSV</span>
-<span class="n">ctx</span><span class="o">.</span><span class="n">register_csv</span><span class="p">(</span><span class="s1">'example'</span><span class="p">,</span> <span class="s1">'example.csv'</span><span class="p">)</span>
-
-<span class="c1"># create a new statement via SQL</span>
-<span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span><span class="s2">"SELECT a+b, a-b FROM example"</span><span class="p">)</span>
-
-<span class="c1"># execute and collect the first (and only) batch</span>
-<span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">collect</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
-
-<span class="k">assert</span> <span class="n">result</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">5</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">9</span><span class="p">])</span>
-<span class="k">assert</span> <span class="n">result</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">==</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="o">-</span>< [...]
-</pre></div>
-</div>
-<section id="udfs">
-<h3>UDFs<a class="headerlink" href="#udfs" title="Permalink to this heading">¶</a></h3>
-<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pyarrow</span>
-<span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">udf</span>
-
-<span class="k">def</span> <span class="nf">is_null</span><span class="p">(</span><span class="n">array</span><span class="p">:</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">Array</span><span class="p">)</span> <span class="o">-></span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">Array</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">array</span><span class="o">.</span><span class="n">is_null</span><span class="p">()</span>
-
-<span class="n">is_null_arr</span> <span class="o">=</span> <span class="n">udf</span><span class="p">(</span><span class="n">is_null</span><span class="p">,</span> <span class="p">[</span><span class="n">pyarrow</span><span class="o">.</span><span class="n">int64</span><span class="p">()],</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">bool_</span><span class="p">(),</span> <span class="s1">'stable'</span><span class="p">)</span>
-
-<span class="c1"># create a context</span>
-<span class="n">ctx</span> <span class="o">=</span> <span class="n">datafusion</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
-
-<span class="c1"># create a RecordBatch and a new DataFrame from it</span>
-<span class="n">batch</span> <span class="o">=</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">RecordBatch</span><span class="o">.</span><span class="n">from_arrays</span><span class="p">(</span>
- <span class="p">[</span><span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]),</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span [...]
- <span class="n">names</span><span class="o">=</span><span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">],</span>
-<span class="p">)</span>
-<span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">create_dataframe</span><span class="p">([[</span><span class="n">batch</span><span class="p">]])</span>
-
-<span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">is_null_arr</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)))</span>
-
-<span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">collect</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
-
-<span class="k">assert</span> <span class="n">result</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="kc">False</span><span class="p">]</span> <span class="o">*</span> <span class="mi">3</span><span class="p">)</span>
+<section id="install">
+<h2>Install<a class="headerlink" href="#install" title="Permalink to this heading">¶</a></h2>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>pip<span class="w"> </span>install<span class="w"> </span>datafusion
</pre></div>
</div>
</section>
-<section id="udaf">
-<h3>UDAF<a class="headerlink" href="#udaf" title="Permalink to this heading">¶</a></h3>
-<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pyarrow</span>
-<span class="kn">import</span> <span class="nn">pyarrow.compute</span>
-<span class="kn">import</span> <span class="nn">datafusion</span>
-<span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">udaf</span><span class="p">,</span> <span class="n">Accumulator</span>
-<span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">col</span>
-
-
-<span class="k">class</span> <span class="nc">MyAccumulator</span><span class="p">(</span><span class="n">Accumulator</span><span class="p">):</span>
-<span class="w"> </span><span class="sd">"""</span>
-<span class="sd"> Interface of a user-defined accumulation.</span>
-<span class="sd"> """</span>
- <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_sum</span> <span class="o">=</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">scalar</span><span class="p">(</span><span class="mf">0.0</span><span class="p">)</span>
+<section id="example">
+<h2>Example<a class="headerlink" href="#example" title="Permalink to this heading">¶</a></h2>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="kn">import</span> <span class="nn">datafusion</span>
- <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">values</span><span class="p">:</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">Array</span><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
- <span class="c1"># not nice since pyarrow scalars can't be summed yet. This breaks on `None`</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_sum</span> <span class="o">=</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">scalar</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_sum</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span> <span class="o">+</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">compute</span><span [...]
+<span class="n">In</span> <span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">col</span>
- <span class="k">def</span> <span class="nf">merge</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">states</span><span class="p">:</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">Array</span><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
- <span class="c1"># not nice since pyarrow scalars can't be summed yet. This breaks on `None`</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_sum</span> <span class="o">=</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">scalar</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_sum</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span> <span class="o">+</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">compute</span><span [...]
-
- <span class="k">def</span> <span class="nf">state</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">Array</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="bp">self</span><span class="o">.</span><span class="n">_sum</span><span class="o">.</span><span class="n">as_py</span><span class="p">()])</span>
-
- <span class="k">def</span> <span class="nf">evaluate</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">Scalar</span><span class="p">:</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_sum</span>
+<span class="n">In</span> <span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="kn">import</span> <span class="nn">pyarrow</span>
<span class="c1"># create a context</span>
-<span class="n">ctx</span> <span class="o">=</span> <span class="n">datafusion</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="n">In</span> <span class="p">[</span><span class="mi">4</span><span class="p">]:</span> <span class="n">ctx</span> <span class="o">=</span> <span class="n">datafusion</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
<span class="c1"># create a RecordBatch and a new DataFrame from it</span>
-<span class="n">batch</span> <span class="o">=</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">RecordBatch</span><span class="o">.</span><span class="n">from_arrays</span><span class="p">(</span>
- <span class="p">[</span><span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]),</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span [...]
- <span class="n">names</span><span class="o">=</span><span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">],</span>
-<span class="p">)</span>
-<span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">create_dataframe</span><span class="p">([[</span><span class="n">batch</span><span class="p">]])</span>
-
-<span class="n">my_udaf</span> <span class="o">=</span> <span class="n">udaf</span><span class="p">(</span><span class="n">MyAccumulator</span><span class="p">,</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">float64</span><span class="p">(),</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">float64</span><span class="p">(),</span> <span class="p">[</span><span class="n">pyarrow</span><span class="o">.</span><span class="n">float64< [...]
+<span class="n">In</span> <span class="p">[</span><span class="mi">5</span><span class="p">]:</span> <span class="n">batch</span> <span class="o">=</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">RecordBatch</span><span class="o">.</span><span class="n">from_arrays</span><span class="p">(</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">[</span><span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]),</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">4</span><span class="p">,</ [...]
+ <span class="o">...</span><span class="p">:</span> <span class="n">names</span><span class="o">=</span><span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">],</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">)</span>
+ <span class="o">...</span><span class="p">:</span>
-<span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">aggregate</span><span class="p">(</span>
- <span class="p">[],</span>
- <span class="p">[</span><span class="n">my_udaf</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">))]</span>
-<span class="p">)</span>
+<span class="n">In</span> <span class="p">[</span><span class="mi">6</span><span class="p">]:</span> <span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">create_dataframe</span><span class="p">([[</span><span class="n">batch</span><span class="p">]],</span> <span class="n">name</span><span class="o">=</span><span class="s2">"batch_array"</span><span class="p">)</span>
-<span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">collect</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
-
-<span class="k">assert</span> <span class="n">result</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">6.0</span><span class="p">])</span>
-</pre></div>
-</div>
-</section>
-</section>
-<section id="how-to-install-from-pip">
-<h2>How to install (from pip)<a class="headerlink" href="#how-to-install-from-pip" title="Permalink to this heading">¶</a></h2>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>pip<span class="w"> </span>install<span class="w"> </span>datafusion
-</pre></div>
-</div>
-<p>You can verify the installation by running:</p>
-<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">datafusion</span>
-<span class="gp">>>> </span><span class="n">datafusion</span><span class="o">.</span><span class="n">__version__</span>
-<span class="go">'0.6.0'</span>
-</pre></div>
-</div>
-</section>
-<section id="how-to-develop">
-<h2>How to develop<a class="headerlink" href="#how-to-develop" title="Permalink to this heading">¶</a></h2>
-<p>This assumes that you have rust and cargo installed. We use the workflow recommended by <a class="reference external" href="https://github.com/PyO3/pyo3">pyo3</a> and <a class="reference external" href="https://github.com/PyO3/maturin">maturin</a>.</p>
-<p>Bootstrap:</p>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="c1"># fetch this repo</span>
-git<span class="w"> </span>clone<span class="w"> </span>git@github.com:apache/arrow-datafusion-python.git
-<span class="c1"># prepare development environment (used to build wheel / install in development)</span>
-python3<span class="w"> </span>-m<span class="w"> </span>venv<span class="w"> </span>venv
-<span class="c1"># activate the venv</span>
-<span class="nb">source</span><span class="w"> </span>venv/bin/activate
-<span class="c1"># update pip itself if necessary</span>
-python<span class="w"> </span>-m<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>-U<span class="w"> </span>pip
-<span class="c1"># install dependencies (for Python 3.8+)</span>
-python<span class="w"> </span>-m<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>-r<span class="w"> </span>requirements-310.txt
-</pre></div>
-</div>
-<p>The tests rely on test data in git submodules.</p>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>git<span class="w"> </span>submodule<span class="w"> </span>init
-git<span class="w"> </span>submodule<span class="w"> </span>update
+<span class="c1"># create a new statement</span>
+<span class="n">In</span> <span class="p">[</span><span class="mi">7</span><span class="p">]:</span> <span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+ <span class="o">...</span><span class="p">:</span> <span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">+</span> <span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span>
+ <span class="o">...</span><span class="p">:</span> <span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">-</span> <span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">)</span>
+ <span class="o">...</span><span class="p">:</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="n">df</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">8</span><span class="p">]:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+-------------------------------+-------------------------------+</span>
+<span class="o">|</span> <span class="n">batch_array</span><span class="o">.</span><span class="n">a</span> <span class="o">+</span> <span class="n">batch_array</span><span class="o">.</span><span class="n">b</span> <span class="o">|</span> <span class="n">batch_array</span><span class="o">.</span><span class="n">a</span> <span class="o">-</span> <span class="n">batch_array</span><span class="o">.</span><span class="n">b</span> <span class="o">|</span>
+<span class="o">+-------------------------------+-------------------------------+</span>
+<span class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span class="o">-</span><span class="mi">3</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">7</span> <span class="o">|</span> <span class="o">-</span><span class="mi">3</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">9</span> <span class="o">|</span> <span class="o">-</span><span class="mi">3</span> <span class="o">|</span>
+<span class="o">+-------------------------------+-------------------------------+</span>
</pre></div>
</div>
-<p>Whenever rust code changes (your changes or via <cite>git pull</cite>):</p>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="c1"># make sure you activate the venv using "source venv/bin/activate" first</span>
-maturin<span class="w"> </span>develop
-python<span class="w"> </span>-m<span class="w"> </span>pytest
-</pre></div>
+<div class="toctree-wrapper compound" id="toc-links">
</div>
-</section>
-<section id="how-to-update-dependencies">
-<h2>How to update dependencies<a class="headerlink" href="#how-to-update-dependencies" title="Permalink to this heading">¶</a></h2>
-<p>To change test dependencies, change the <cite>requirements.in</cite> and run</p>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="c1"># install pip-tools (this can be done only once), also consider running in venv</span>
-python<span class="w"> </span>-m<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>pip-tools
-python<span class="w"> </span>-m<span class="w"> </span>piptools<span class="w"> </span>compile<span class="w"> </span>--generate-hashes<span class="w"> </span>-o<span class="w"> </span>requirements-310.txt
-</pre></div>
+<div class="toctree-wrapper compound" id="toc-guide">
</div>
-<p>To update dependencies, run with <cite>-U</cite></p>
-<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>python<span class="w"> </span>-m<span class="w"> </span>piptools<span class="w"> </span>compile<span class="w"> </span>-U<span class="w"> </span>--generate-hashes<span class="w"> </span>-o<span class="w"> </span>requirements-310.txt
-</pre></div>
+<div class="toctree-wrapper compound" id="toc-contributor-guide">
</div>
-<p>More details about pip-tools <a class="reference external" href="https://github.com/jazzband/pip-tools">here</a></p>
-</section>
-<section id="api-reference">
-<h2>API reference<a class="headerlink" href="#api-reference" title="Permalink to this heading">¶</a></h2>
-<div class="toctree-wrapper compound">
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="api.html">API Reference</a><ul>
-<li class="toctree-l2"><a class="reference internal" href="api/dataframe.html">DataFrame</a></li>
-<li class="toctree-l2"><a class="reference internal" href="api/execution_context.html">SessionContext</a></li>
-<li class="toctree-l2"><a class="reference internal" href="api/expression.html">Expr</a></li>
-<li class="toctree-l2"><a class="reference internal" href="api/functions.html">Functions</a></li>
-<li class="toctree-l2"><a class="reference internal" href="api/object_store.html">ObjectStore</a></li>
-</ul>
-</li>
-</ul>
+<div class="toctree-wrapper compound" id="toc-api">
</div>
</section>
</section>
@@ -513,10 +453,10 @@ python<span class="w"> </span>-m<span class="w"> </span>piptools<span class="w">
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='right-next' id="next-link" href="api.html" title="next page">
+ <a class='right-next' id="next-link" href="user-guide/introduction.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">API Reference</p>
+ <p class="prev-next-title">Introduction</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/objects.inv b/objects.inv
index bd78dc9..c566419 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/py-modindex.html b/py-modindex.html
index b46fd01..39031d6 100644
--- a/py-modindex.html
+++ b/py-modindex.html
@@ -72,23 +72,163 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
<li class="toctree-l1 has-children">
- <a class="reference internal" href="api.html">
- API Reference
+ <a class="reference internal" href="user-guide/common-operations/index.html">
+ Common Operations
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
<i class="fas fa-chevron-down">
</i>
</label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="api.html">
+ API Reference
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
<ul>
<li class="toctree-l2 has-children">
<a class="reference internal" href="api/dataframe.html">
DataFrame
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -104,8 +244,8 @@
<a class="reference internal" href="api/execution_context.html">
SessionContext
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -131,8 +271,8 @@
<a class="reference internal" href="api/expression.html">
Expr
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -148,8 +288,8 @@
<a class="reference internal" href="api/functions.html">
Functions
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -165,8 +305,8 @@
<a class="reference internal" href="api/object_store.html">
ObjectStore
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
diff --git a/search.html b/search.html
index a6d9bb5..e78a7a0 100644
--- a/search.html
+++ b/search.html
@@ -75,23 +75,163 @@
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
- <ul class="nav bd-sidenav">
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/basics.html">
+ Concepts
+ </a>
+ </li>
<li class="toctree-l1 has-children">
- <a class="reference internal" href="api.html">
- API Reference
+ <a class="reference internal" href="user-guide/common-operations/index.html">
+ Common Operations
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
<i class="fas fa-chevron-down">
</i>
</label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="user-guide/io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="user-guide/io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="api.html">
+ API Reference
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
<ul>
<li class="toctree-l2 has-children">
<a class="reference internal" href="api/dataframe.html">
DataFrame
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
- <label for="toctree-checkbox-2">
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -107,8 +247,8 @@
<a class="reference internal" href="api/execution_context.html">
SessionContext
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
- <label for="toctree-checkbox-3">
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -134,8 +274,8 @@
<a class="reference internal" href="api/expression.html">
Expr
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
- <label for="toctree-checkbox-4">
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -151,8 +291,8 @@
<a class="reference internal" href="api/functions.html">
Functions
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
- <label for="toctree-checkbox-5">
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
<i class="fas fa-chevron-down">
</i>
</label>
@@ -168,8 +308,8 @@
<a class="reference internal" href="api/object_store.html">
ObjectStore
</a>
- <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
- <label for="toctree-checkbox-6">
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
<i class="fas fa-chevron-down">
</i>
</label>
diff --git a/searchindex.js b/searchindex.js
index a6a7cde..4c0750a 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"docnames": ["api", "api/dataframe", "api/execution_context", "api/expression", "api/functions", "api/object_store", "generated/datafusion.DataFrame", "generated/datafusion.Expr", "generated/datafusion.RuntimeConfig", "generated/datafusion.SessionConfig", "generated/datafusion.SessionContext", "generated/datafusion.functions.functions", "generated/datafusion.object_store.object_store", "index"], "filenames": ["api.rst", "api/dataframe.rst", "api/execution_context.rst", " [...]
\ No newline at end of file
+Search.setIndex({"docnames": ["api", "api/dataframe", "api/execution_context", "api/expression", "api/functions", "api/object_store", "contributor-guide/introduction", "generated/datafusion.DataFrame", "generated/datafusion.Expr", "generated/datafusion.RuntimeConfig", "generated/datafusion.SessionConfig", "generated/datafusion.SessionContext", "generated/datafusion.functions.functions", "generated/datafusion.object_store.object_store", "index", "user-guide/basics", "user-guide/common-ope [...]
\ No newline at end of file
diff --git a/user-guide/basics.html b/user-guide/basics.html
new file mode 100644
index 0000000..ec5fa52
--- /dev/null
+++ b/user-guide/basics.html
@@ -0,0 +1,481 @@
+
+<!DOCTYPE html>
+
+<html lang="en">
+ <head>
+ <meta charset="utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
+
+ <title>Concepts — Arrow DataFusion documentation</title>
+
+ <link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+<link href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+
+
+ <link rel="stylesheet"
+ href="../_static/vendor/fontawesome/5.13.0/css/all.min.css">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
+
+
+
+
+
+ <link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
+ <link rel="stylesheet" type="text/css" href="../_static/styles/pydata-sphinx-theme.css" />
+ <link rel="stylesheet" type="text/css" href="../_static/theme_overrides.css" />
+
+ <link rel="preload" as="script" href="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
+
+ <script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
+ <script src="../_static/jquery.js"></script>
+ <script src="../_static/underscore.js"></script>
+ <script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
+ <script src="../_static/doctools.js"></script>
+ <script src="../_static/sphinx_highlight.js"></script>
+ <link rel="index" title="Index" href="../genindex.html" />
+ <link rel="search" title="Search" href="../search.html" />
+ <link rel="next" title="Common Operations" href="common-operations/index.html" />
+ <link rel="prev" title="Introduction" href="introduction.html" />
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
+ <meta name="docsearch:language" content="en">
+
+
+ <!-- Google Analytics -->
+
+ </head>
+ <body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
+
+ <div class="container-fluid" id="banner"></div>
+
+
+
+
+ <div class="container-xl">
+ <div class="row">
+
+
+ <!-- Only show if we have sidebars configured, else just a small margin -->
+ <div class="col-12 col-md-3 bd-sidebar">
+ <div class="sidebar-start-items">
+<a class="navbar-brand" href="../index.html">
+ <img src="../_static/images/DataFusion-Logo-Background-White.png" class="logo" alt="logo">
+</a>
+
+<form class="bd-search d-flex align-items-center" action="../search.html" method="get">
+ <i class="icon fas fa-search"></i>
+ <input type="search" class="form-control" name="q" id="search-input" placeholder="Search the docs ..." aria-label="Search the docs ..." autocomplete="off" >
+</form>
+
+<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
+ <div class="bd-toc-item active">
+
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1 current active">
+ <a class="current reference internal" href="#">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="common-operations/index.html">
+ Common Operations
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="common-operations/basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="common-operations/select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="common-operations/joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="common-operations/functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="common-operations/aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="common-operations/windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="common-operations/udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../api.html">
+ API Reference
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../api/dataframe.html">
+ DataFrame
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../generated/datafusion.DataFrame.html">
+ datafusion.DataFrame
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../api/execution_context.html">
+ SessionContext
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../generated/datafusion.SessionConfig.html">
+ datafusion.SessionConfig
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../generated/datafusion.RuntimeConfig.html">
+ datafusion.RuntimeConfig
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../generated/datafusion.SessionContext.html">
+ datafusion.SessionContext
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../api/expression.html">
+ Expr
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../generated/datafusion.Expr.html">
+ datafusion.Expr
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../api/functions.html">
+ Functions
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../generated/datafusion.functions.functions.html">
+ datafusion.functions.functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../api/object_store.html">
+ ObjectStore
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../generated/datafusion.object_store.object_store.html">
+ datafusion.object_store.object_store
+ </a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+</ul>
+
+
+ </div>
+</nav>
+ </div>
+ <div class="sidebar-end-items">
+ </div>
+ </div>
+
+
+
+
+ <div class="d-none d-xl-block col-xl-2 bd-toc">
+
+
+ <div class="toc-item">
+
+
+<nav id="bd-toc-nav">
+
+</nav>
+ </div>
+
+ <div class="toc-item">
+
+
+<div class="tocsection editthispage">
+ <a href="https://github.com/apache/arrow-datafusion-python/edit/main/docs/source/user-guide/basics.rst">
+ <i class="fas fa-pencil-alt"></i> Edit this page
+ </a>
+</div>
+
+ </div>
+
+
+ </div>
+
+
+
+
+
+
+ <main class="col-12 col-md-9 col-xl-7 py-md-5 pl-md-5 pr-md-4 bd-content" role="main">
+
+ <div>
+
+ <section id="concepts">
+<h1>Concepts<a class="headerlink" href="#concepts" title="Permalink to this heading">¶</a></h1>
+<p>In this section, we will cover a basic example to introduce a few key concepts.</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">datafusion</span>
+<span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">col</span>
+<span class="kn">import</span> <span class="nn">pyarrow</span>
+
+<span class="c1"># create a context</span>
+<span class="n">ctx</span> <span class="o">=</span> <span class="n">datafusion</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+
+<span class="c1"># create a RecordBatch and a new DataFrame from it</span>
+<span class="n">batch</span> <span class="o">=</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">RecordBatch</span><span class="o">.</span><span class="n">from_arrays</span><span class="p">(</span>
+ <span class="p">[</span><span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]),</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span [...]
+ <span class="n">names</span><span class="o">=</span><span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">],</span>
+<span class="p">)</span>
+<span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">create_dataframe</span><span class="p">([[</span><span class="n">batch</span><span class="p">]])</span>
+
+<span class="c1"># create a new statement</span>
+<span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+ <span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">+</span> <span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span>
+ <span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">-</span> <span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span>
+<span class="p">)</span>
+
+<span class="c1"># execute and collect the first (and only) batch</span>
+<span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">collect</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
+</pre></div>
+</div>
+<p>The first statement group:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="c1"># create a context</span>
+<span class="n">ctx</span> <span class="o">=</span> <span class="n">datafusion</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+</pre></div>
+</div>
+<p>creates a <code class="code docutils literal notranslate"><span class="pre">SessionContext</span></code>, that is, the main interface for executing queries with DataFusion. It maintains the state
+of the connection between a user and an instance of the DataFusion engine. Additionally it provides the following functionality:</p>
+<ul class="simple">
+<li><p>Create a DataFrame from a CSV or Parquet data source.</p></li>
+<li><p>Register a CSV or Parquet data source as a table that can be referenced from a SQL query.</p></li>
+<li><p>Register a custom data source that can be referenced from a SQL query.</p></li>
+<li><p>Execute a SQL query</p></li>
+</ul>
+<p>The second statement group creates a <code class="code docutils literal notranslate"><span class="pre">DataFrame</span></code>,</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="c1"># create a RecordBatch and a new DataFrame from it</span>
+<span class="n">batch</span> <span class="o">=</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">RecordBatch</span><span class="o">.</span><span class="n">from_arrays</span><span class="p">(</span>
+ <span class="p">[</span><span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]),</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span [...]
+ <span class="n">names</span><span class="o">=</span><span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">],</span>
+<span class="p">)</span>
+<span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">create_dataframe</span><span class="p">([[</span><span class="n">batch</span><span class="p">]])</span>
+</pre></div>
+</div>
+<p>A DataFrame refers to a (logical) set of rows that share the same column names, similar to a <a class="reference external" href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html">Pandas DataFrame</a>.
+DataFrames are typically created by calling a method on <code class="code docutils literal notranslate"><span class="pre">SessionContext</span></code>, such as <code class="code docutils literal notranslate"><span class="pre">read_csv</span></code>, and can then be modified by
+calling the transformation methods, such as <a class="reference internal" href="../generated/datafusion.DataFrame.html#datafusion.DataFrame.filter" title="datafusion.DataFrame.filter"><code class="xref py py-meth docutils literal notranslate"><span class="pre">DataFrame.filter()</span></code></a>, <a class="reference internal" href="../generated/datafusion.DataFrame.html#datafusion.DataFrame.select" title="datafusion.DataFrame.select"><code class="xref py py-meth docutils literal notrans [...]
+and <a class="reference internal" href="../generated/datafusion.DataFrame.html#datafusion.DataFrame.limit" title="datafusion.DataFrame.limit"><code class="xref py py-meth docutils literal notranslate"><span class="pre">DataFrame.limit()</span></code></a> to build up a query definition.</p>
+<p>The third statement uses <code class="code docutils literal notranslate"><span class="pre">Expressions</span></code> to build up a query definition.</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+ <span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">+</span> <span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span>
+ <span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">-</span> <span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span>
+<span class="p">)</span>
+</pre></div>
+</div>
+<p>Finally the <code class="code docutils literal notranslate"><span class="pre">collect</span></code> method converts the logical plan represented by the DataFrame into a physical plan and execute it,
+collecting all results into a list of <a class="reference external" href="https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatch.html">RecordBatch</a>.</p>
+</section>
+
+
+ </div>
+
+
+ <!-- Previous / next buttons -->
+<div class='prev-next-area'>
+ <a class='left-prev' id="prev-link" href="introduction.html" title="previous page">
+ <i class="fas fa-angle-left"></i>
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">previous</p>
+ <p class="prev-next-title">Introduction</p>
+ </div>
+ </a>
+ <a class='right-next' id="next-link" href="common-operations/index.html" title="next page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">Common Operations</p>
+ </div>
+ <i class="fas fa-angle-right"></i>
+ </a>
+</div>
+
+ </main>
+
+
+ </div>
+ </div>
+
+ <script src="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
+ <footer class="footer mt-5 mt-md-0">
+ <div class="container">
+
+ <div class="footer-item">
+ <p class="copyright">
+ © Copyright 2022, Apache Software Foundation.<br>
+</p>
+ </div>
+
+ <div class="footer-item">
+ <p class="sphinx-version">
+Created using <a href="http://sphinx-doc.org/">Sphinx</a> 5.3.0.<br>
+</p>
+ </div>
+
+ </div>
+</footer>
+ </body>
+</html>
\ No newline at end of file
diff --git a/user-guide/common-operations/aggregations.html b/user-guide/common-operations/aggregations.html
new file mode 100644
index 0000000..0fa4f49
--- /dev/null
+++ b/user-guide/common-operations/aggregations.html
@@ -0,0 +1,491 @@
+
+<!DOCTYPE html>
+
+<html lang="en">
+ <head>
+ <meta charset="utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
+
+ <title>Aggregation — Arrow DataFusion documentation</title>
+
+ <link href="../../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+<link href="../../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+
+
+ <link rel="stylesheet"
+ href="../../_static/vendor/fontawesome/5.13.0/css/all.min.css">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
+
+
+
+
+
+ <link rel="stylesheet" type="text/css" href="../../_static/pygments.css" />
+ <link rel="stylesheet" type="text/css" href="../../_static/styles/pydata-sphinx-theme.css" />
+ <link rel="stylesheet" type="text/css" href="../../_static/theme_overrides.css" />
+
+ <link rel="preload" as="script" href="../../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
+
+ <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script>
+ <script src="../../_static/jquery.js"></script>
+ <script src="../../_static/underscore.js"></script>
+ <script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script>
+ <script src="../../_static/doctools.js"></script>
+ <script src="../../_static/sphinx_highlight.js"></script>
+ <link rel="index" title="Index" href="../../genindex.html" />
+ <link rel="search" title="Search" href="../../search.html" />
+ <link rel="next" title="Window Functions" href="windows.html" />
+ <link rel="prev" title="Functions" href="functions.html" />
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
+ <meta name="docsearch:language" content="en">
+
+
+ <!-- Google Analytics -->
+
+ </head>
+ <body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
+
+ <div class="container-fluid" id="banner"></div>
+
+
+
+
+ <div class="container-xl">
+ <div class="row">
+
+
+ <!-- Only show if we have sidebars configured, else just a small margin -->
+ <div class="col-12 col-md-3 bd-sidebar">
+ <div class="sidebar-start-items">
+<a class="navbar-brand" href="../../index.html">
+ <img src="../../_static/images/DataFusion-Logo-Background-White.png" class="logo" alt="logo">
+</a>
+
+<form class="bd-search d-flex align-items-center" action="../../search.html" method="get">
+ <i class="icon fas fa-search"></i>
+ <input type="search" class="form-control" name="q" id="search-input" placeholder="Search the docs ..." aria-label="Search the docs ..." autocomplete="off" >
+</form>
+
+<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
+ <div class="bd-toc-item active">
+
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 current active has-children">
+ <a class="reference internal" href="index.html">
+ Common Operations
+ </a>
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul class="current">
+ <li class="toctree-l2">
+ <a class="reference internal" href="basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2 current active">
+ <a class="current reference internal" href="#">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../../api.html">
+ API Reference
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/dataframe.html">
+ DataFrame
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.DataFrame.html">
+ datafusion.DataFrame
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/execution_context.html">
+ SessionContext
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.SessionConfig.html">
+ datafusion.SessionConfig
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.RuntimeConfig.html">
+ datafusion.RuntimeConfig
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.SessionContext.html">
+ datafusion.SessionContext
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/expression.html">
+ Expr
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.Expr.html">
+ datafusion.Expr
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/functions.html">
+ Functions
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.functions.functions.html">
+ datafusion.functions.functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/object_store.html">
+ ObjectStore
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.object_store.object_store.html">
+ datafusion.object_store.object_store
+ </a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+</ul>
+
+
+ </div>
+</nav>
+ </div>
+ <div class="sidebar-end-items">
+ </div>
+ </div>
+
+
+
+
+ <div class="d-none d-xl-block col-xl-2 bd-toc">
+
+
+ <div class="toc-item">
+
+
+<nav id="bd-toc-nav">
+
+</nav>
+ </div>
+
+ <div class="toc-item">
+
+
+<div class="tocsection editthispage">
+ <a href="https://github.com/apache/arrow-datafusion-python/edit/main/docs/source/user-guide/common-operations/aggregations.rst">
+ <i class="fas fa-pencil-alt"></i> Edit this page
+ </a>
+</div>
+
+ </div>
+
+
+ </div>
+
+
+
+
+
+
+ <main class="col-12 col-md-9 col-xl-7 py-md-5 pl-md-5 pr-md-4 bd-content" role="main">
+
+ <div>
+
+ <section id="aggregation">
+<h1>Aggregation<a class="headerlink" href="#aggregation" title="Permalink to this heading">¶</a></h1>
+<p>An aggregate or aggregation is a function where the values of multiple rows are processed together to form a single summary value.
+For performing an aggregation, DataFusion provides the <a class="reference internal" href="../../generated/datafusion.DataFrame.html#datafusion.DataFrame.aggregate" title="datafusion.DataFrame.aggregate"><code class="xref py py-meth docutils literal notranslate"><span class="pre">DataFrame.aggregate()</span></code></a></p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">SessionContext</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">column</span><span class="p">,</span> <span class="n">lit</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">functions</span> <span class="k">as</span> <span class="n">f</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">4</span><span class="p">]:</span> <span class="kn">import</span> <span class="nn">random</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">5</span><span class="p">]:</span> <span class="n">ctx</span> <span class="o">=</span> <span class="n">SessionContext</span><span class="p">()</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">6</span><span class="p">]:</span> <span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">(</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">{</span>
+ <span class="o">...</span><span class="p">:</span> <span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"foo"</span><span class="p">,</span> <span class="s2">"bar"</span><span class="p">,</span> <span class="s2">"foo"</span><span class="p">,</span> <span class="s2">"bar"</span><span class="p">,</span> <span class="s2">"foo"</span><span class="p">,</span> <span class="s2">"b [...]
+ <span class="o">...</span><span class="p">:</span> <span class="s2">"b"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"one"</span><span class="p">,</span> <span class="s2">"one"</span><span class="p">,</span> <span class="s2">"two"</span><span class="p">,</span> <span class="s2">"three"</span><span class="p">,</span> <span class="s2">"two"</span><span class="p">,</span> <span class="s2">" [...]
+ <span class="o">...</span><span class="p">:</span> <span class="s2">"c"</span><span class="p">:</span> <span class="p">[</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><sp [...]
+ <span class="o">...</span><span class="p">:</span> <span class="s2">"d"</span><span class="p">:</span> <span class="p">[</span><span class="n">random</span><span class="o">.</span><span class="n">random</span><span class="p">()</span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">8</span><span class="p">)],</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">},</span>
+ <span class="o">...</span><span class="p">:</span> <span class="n">name</span><span class="o">=</span><span class="s2">"foo_bar"</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">)</span>
+ <span class="o">...</span><span class="p">:</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">7</span><span class="p">]:</span> <span class="n">col_a</span> <span class="o">=</span> <span class="n">column</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="n">col_b</span> <span class="o">=</span> <span class="n">column</span><span class="p">(</span><span class="s2">"b"</span><span class="p">)</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">9</span><span class="p">]:</span> <span class="n">col_c</span> <span class="o">=</span> <span class="n">column</span><span class="p">(</span><span class="s2">"c"</span><span class="p">)</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">10</span><span class="p">]:</span> <span class="n">col_d</span> <span class="o">=</span> <span class="n">column</span><span class="p">(</span><span class="s2">"d"</span><span class="p">)</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">11</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">aggregate</span><span class="p">([],</span> <span class="p">[</span><span class="n">f</span><span class="o">.</span><span class="n">approx_distinct</span><span class="p">(</span><span class="n">col_c</span><span class="p">),</span> <span class="n">f</span><span class="o">.</span><span class="n">approx_median</span><span cla [...]
+<span class="n">Out</span><span class="p">[</span><span class="mi">11</span><span class="p">]:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+----------------------------+--------------------------+------------------------------------------------+</span>
+<span class="o">|</span> <span class="n">APPROX_DISTINCT</span><span class="p">(</span><span class="n">foo_bar</span><span class="o">.</span><span class="n">c</span><span class="p">)</span> <span class="o">|</span> <span class="n">APPROX_MEDIAN</span><span class="p">(</span><span class="n">foo_bar</span><span class="o">.</span><span class="n">d</span><span class="p">)</span> <span class="o">|</span> <span class="n">APPROX_PERCENTILE_CONT</span><span class="p">(</span><span class="n">foo_ [...]
+<span class="o">+----------------------------+--------------------------+------------------------------------------------+</span>
+<span class="o">|</span> <span class="mi">7</span> <span class="o">|</span> <span class="mf">0.5938667947211614</span> <span class="o">|</span> <span class="mf">0.5938667947211614</span> <span class="o">|</span>
+<span class="o">+----------------------------+--------------------------+------------------------------------------------+</span>
+</pre></div>
+</div>
+<p>When the <code class="code docutils literal notranslate"><span class="pre">group_by</span></code> list is empty the aggregation is done over the whole <a class="reference internal" href="../../generated/datafusion.DataFrame.html#datafusion.DataFrame" title="datafusion.DataFrame"><code class="xref py py-class docutils literal notranslate"><span class="pre">DataFrame</span></code></a>. For grouping
+the <code class="code docutils literal notranslate"><span class="pre">group_by</span></code> list must contain at least one column</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">12</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">aggregate</span><span class="p">([</span><span class="n">col_a</span><span class="p">],</span> <span class="p">[</span><span class="n">f</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">col_c</s [...]
+<span class="n">Out</span><span class="p">[</span><span class="mi">12</span><span class="p">]:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+-----+----------------+--------------------+--------------------+</span>
+<span class="o">|</span> <span class="n">a</span> <span class="o">|</span> <span class="n">SUM</span><span class="p">(</span><span class="n">foo_bar</span><span class="o">.</span><span class="n">c</span><span class="p">)</span> <span class="o">|</span> <span class="n">MAX</span><span class="p">(</span><span class="n">foo_bar</span><span class="o">.</span><span class="n">d</span><span class="p">)</span> <span class="o">|</span> <span class="n">MIN</span><span class="p">(</span><span [...]
+<span class="o">+-----+----------------+--------------------+--------------------+</span>
+<span class="o">|</span> <span class="n">foo</span> <span class="o">|</span> <span class="mi">240</span> <span class="o">|</span> <span class="mf">0.7051103057099627</span> <span class="o">|</span> <span class="mf">0.1781661139664239</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">bar</span> <span class="o">|</span> <span class="mi">53</span> <span class="o">|</span> <span class="mf">0.8023162636291106</span> <span class="o">|</span> <span class="mf">0.4150143008021163</span> <span class="o">|</span>
+<span class="o">+-----+----------------+--------------------+--------------------+</span>
+</pre></div>
+</div>
+<p>More than one column can be used for grouping</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">13</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">aggregate</span><span class="p">([</span><span class="n">col_a</span><span class="p">,</span> <span class="n">col_b</span><span class="p">],</span> <span class="p">[</span><span class="n">f</span><span class="o">.</span><span class="n">sum</ [...]
+<span class="n">Out</span><span class="p">[</span><span class="mi">13</span><span class="p">]:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+-----+-------+----------------+---------------------+---------------------+</span>
+<span class="o">|</span> <span class="n">a</span> <span class="o">|</span> <span class="n">b</span> <span class="o">|</span> <span class="n">SUM</span><span class="p">(</span><span class="n">foo_bar</span><span class="o">.</span><span class="n">c</span><span class="p">)</span> <span class="o">|</span> <span class="n">MAX</span><span class="p">(</span><span class="n">foo_bar</span><span class="o">.</span><span class="n">d</span><span class="p">)</span> <span class="o">|</span> [...]
+<span class="o">+-----+-------+----------------+---------------------+---------------------+</span>
+<span class="o">|</span> <span class="n">foo</span> <span class="o">|</span> <span class="n">one</span> <span class="o">|</span> <span class="mi">102</span> <span class="o">|</span> <span class="mf">0.6988692102870115</span> <span class="o">|</span> <span class="mf">0.18702584635568487</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">bar</span> <span class="o">|</span> <span class="n">one</span> <span class="o">|</span> <span class="mi">7</span> <span class="o">|</span> <span class="mf">0.8023162636291106</span> <span class="o">|</span> <span class="mf">0.8023162636291106</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">foo</span> <span class="o">|</span> <span class="n">two</span> <span class="o">|</span> <span class="mi">68</span> <span class="o">|</span> <span class="mf">0.7051103057099627</span> <span class="o">|</span> <span class="mf">0.1781661139664239</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">bar</span> <span class="o">|</span> <span class="n">three</span> <span class="o">|</span> <span class="mi">39</span> <span class="o">|</span> <span class="mf">0.6648305220923852</span> <span class="o">|</span> <span class="mf">0.6648305220923852</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">bar</span> <span class="o">|</span> <span class="n">two</span> <span class="o">|</span> <span class="mi">7</span> <span class="o">|</span> <span class="mf">0.4150143008021163</span> <span class="o">|</span> <span class="mf">0.4150143008021163</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">foo</span> <span class="o">|</span> <span class="n">three</span> <span class="o">|</span> <span class="mi">70</span> <span class="o">|</span> <span class="mf">0.37779846732681843</span> <span class="o">|</span> <span class="mf">0.37779846732681843</span> <span class="o">|</span>
+<span class="o">+-----+-------+----------------+---------------------+---------------------+</span>
+</pre></div>
+</div>
+</section>
+
+
+ </div>
+
+
+ <!-- Previous / next buttons -->
+<div class='prev-next-area'>
+ <a class='left-prev' id="prev-link" href="functions.html" title="previous page">
+ <i class="fas fa-angle-left"></i>
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">previous</p>
+ <p class="prev-next-title">Functions</p>
+ </div>
+ </a>
+ <a class='right-next' id="next-link" href="windows.html" title="next page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">Window Functions</p>
+ </div>
+ <i class="fas fa-angle-right"></i>
+ </a>
+</div>
+
+ </main>
+
+
+ </div>
+ </div>
+
+ <script src="../../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
+ <footer class="footer mt-5 mt-md-0">
+ <div class="container">
+
+ <div class="footer-item">
+ <p class="copyright">
+ © Copyright 2022, Apache Software Foundation.<br>
+</p>
+ </div>
+
+ <div class="footer-item">
+ <p class="sphinx-version">
+Created using <a href="http://sphinx-doc.org/">Sphinx</a> 5.3.0.<br>
+</p>
+ </div>
+
+ </div>
+</footer>
+ </body>
+</html>
\ No newline at end of file
diff --git a/user-guide/common-operations/basic-info.html b/user-guide/common-operations/basic-info.html
new file mode 100644
index 0000000..55433bf
--- /dev/null
+++ b/user-guide/common-operations/basic-info.html
@@ -0,0 +1,500 @@
+
+<!DOCTYPE html>
+
+<html lang="en">
+ <head>
+ <meta charset="utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
+
+ <title>Basic Operations — Arrow DataFusion documentation</title>
+
+ <link href="../../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+<link href="../../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+
+
+ <link rel="stylesheet"
+ href="../../_static/vendor/fontawesome/5.13.0/css/all.min.css">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
+
+
+
+
+
+ <link rel="stylesheet" type="text/css" href="../../_static/pygments.css" />
+ <link rel="stylesheet" type="text/css" href="../../_static/styles/pydata-sphinx-theme.css" />
+ <link rel="stylesheet" type="text/css" href="../../_static/theme_overrides.css" />
+
+ <link rel="preload" as="script" href="../../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
+
+ <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script>
+ <script src="../../_static/jquery.js"></script>
+ <script src="../../_static/underscore.js"></script>
+ <script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script>
+ <script src="../../_static/doctools.js"></script>
+ <script src="../../_static/sphinx_highlight.js"></script>
+ <link rel="index" title="Index" href="../../genindex.html" />
+ <link rel="search" title="Search" href="../../search.html" />
+ <link rel="next" title="Column Selections" href="select-and-filter.html" />
+ <link rel="prev" title="Common Operations" href="index.html" />
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
+ <meta name="docsearch:language" content="en">
+
+
+ <!-- Google Analytics -->
+
+ </head>
+ <body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
+
+ <div class="container-fluid" id="banner"></div>
+
+
+
+
+ <div class="container-xl">
+ <div class="row">
+
+
+ <!-- Only show if we have sidebars configured, else just a small margin -->
+ <div class="col-12 col-md-3 bd-sidebar">
+ <div class="sidebar-start-items">
+<a class="navbar-brand" href="../../index.html">
+ <img src="../../_static/images/DataFusion-Logo-Background-White.png" class="logo" alt="logo">
+</a>
+
+<form class="bd-search d-flex align-items-center" action="../../search.html" method="get">
+ <i class="icon fas fa-search"></i>
+ <input type="search" class="form-control" name="q" id="search-input" placeholder="Search the docs ..." aria-label="Search the docs ..." autocomplete="off" >
+</form>
+
+<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
+ <div class="bd-toc-item active">
+
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 current active has-children">
+ <a class="reference internal" href="index.html">
+ Common Operations
+ </a>
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul class="current">
+ <li class="toctree-l2 current active">
+ <a class="current reference internal" href="#">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../../api.html">
+ API Reference
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/dataframe.html">
+ DataFrame
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.DataFrame.html">
+ datafusion.DataFrame
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/execution_context.html">
+ SessionContext
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.SessionConfig.html">
+ datafusion.SessionConfig
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.RuntimeConfig.html">
+ datafusion.RuntimeConfig
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.SessionContext.html">
+ datafusion.SessionContext
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/expression.html">
+ Expr
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.Expr.html">
+ datafusion.Expr
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/functions.html">
+ Functions
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.functions.functions.html">
+ datafusion.functions.functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/object_store.html">
+ ObjectStore
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.object_store.object_store.html">
+ datafusion.object_store.object_store
+ </a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+</ul>
+
+
+ </div>
+</nav>
+ </div>
+ <div class="sidebar-end-items">
+ </div>
+ </div>
+
+
+
+
+ <div class="d-none d-xl-block col-xl-2 bd-toc">
+
+
+ <div class="toc-item">
+
+
+<nav id="bd-toc-nav">
+
+</nav>
+ </div>
+
+ <div class="toc-item">
+
+
+<div class="tocsection editthispage">
+ <a href="https://github.com/apache/arrow-datafusion-python/edit/main/docs/source/user-guide/common-operations/basic-info.rst">
+ <i class="fas fa-pencil-alt"></i> Edit this page
+ </a>
+</div>
+
+ </div>
+
+
+ </div>
+
+
+
+
+
+
+ <main class="col-12 col-md-9 col-xl-7 py-md-5 pl-md-5 pr-md-4 bd-content" role="main">
+
+ <div>
+
+ <section id="basic-operations">
+<h1>Basic Operations<a class="headerlink" href="#basic-operations" title="Permalink to this heading">¶</a></h1>
+<p>In this section, you will learn how to display essential details of DataFrames using specific functions.</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">SessionContext</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="kn">import</span> <span class="nn">random</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="n">ctx</span> <span class="o">=</span> <span class="n">SessionContext</span><span class="p">()</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">4</span><span class="p">]:</span> <span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span>
+ <span class="o">...</span><span class="p">:</span> <span class="s2">"nrs"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">],</span>
+ <span class="o">...</span><span class="p">:</span> <span class="s2">"names"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"python"</span><span class="p">,</span> <span class="s2">"ruby"</span><span class="p">,</span> <span class="s2">"java"</span><span class="p">,</span> <span class="s2">"haskell"</span><span class="p">,</span> <span class="s2">"go"</span><span class="p">],</span>
+ <span class="o">...</span><span class="p">:</span> <span class="s2">"random"</span><span class="p">:</span> <span class="n">random</span><span class="o">.</span><span class="n">sample</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">1000</span><span class="p">),</span> <span class="mi">5</span><span class="p">),</span>
+ <span class="o">...</span><span class="p">:</span> <span class="s2">"groups"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"A"</span><span class="p">,</span> <span class="s2">"A"</span><span class="p">,</span> <span class="s2">"B"</span><span class="p">,</span> <span class="s2">"C"</span><span class="p">,</span> <span class="s2">"B"</span><span class="p">],</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">})</span>
+ <span class="o">...</span><span class="p">:</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">5</span><span class="p">]:</span> <span class="n">df</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">5</span><span class="p">]:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+-----+---------+--------+--------+</span>
+<span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span> <span class="n">names</span> <span class="o">|</span> <span class="n">random</span> <span class="o">|</span> <span class="n">groups</span> <span class="o">|</span>
+<span class="o">+-----+---------+--------+--------+</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">python</span> <span class="o">|</span> <span class="mi">216</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mi">121</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">3</span> <span class="o">|</span> <span class="n">java</span> <span class="o">|</span> <span class="mi">567</span> <span class="o">|</span> <span class="n">B</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">4</span> <span class="o">|</span> <span class="n">haskell</span> <span class="o">|</span> <span class="mi">680</span> <span class="o">|</span> <span class="n">C</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span class="mi">186</span> <span class="o">|</span> <span class="n">B</span> <span class="o">|</span>
+<span class="o">+-----+---------+--------+--------+</span>
+</pre></div>
+</div>
+<p>Use <a class="reference internal" href="../../generated/datafusion.DataFrame.html#datafusion.DataFrame.limit" title="datafusion.DataFrame.limit"><code class="xref py py-meth docutils literal notranslate"><span class="pre">DataFrame.limit()</span></code></a> to view the top rows of the frame:</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">6</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">limit</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">6</span><span class="p">]:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+-----+--------+--------+--------+</span>
+<span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span> <span class="n">names</span> <span class="o">|</span> <span class="n">random</span> <span class="o">|</span> <span class="n">groups</span> <span class="o">|</span>
+<span class="o">+-----+--------+--------+--------+</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">python</span> <span class="o">|</span> <span class="mi">216</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mi">121</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">+-----+--------+--------+--------+</span>
+</pre></div>
+</div>
+<p>Display the columns of the DataFrame using <a class="reference internal" href="../../generated/datafusion.DataFrame.html#datafusion.DataFrame.schema" title="datafusion.DataFrame.schema"><code class="xref py py-meth docutils literal notranslate"><span class="pre">DataFrame.schema()</span></code></a>:</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">7</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">schema</span><span class="p">()</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">7</span><span class="p">]:</span>
+<span class="n">nrs</span><span class="p">:</span> <span class="n">int64</span>
+<span class="n">names</span><span class="p">:</span> <span class="n">string</span>
+<span class="n">random</span><span class="p">:</span> <span class="n">int64</span>
+<span class="n">groups</span><span class="p">:</span> <span class="n">string</span>
+</pre></div>
+</div>
+<p>The method <a class="reference internal" href="../../generated/datafusion.DataFrame.html#datafusion.DataFrame.to_pandas" title="datafusion.DataFrame.to_pandas"><code class="xref py py-meth docutils literal notranslate"><span class="pre">DataFrame.to_pandas()</span></code></a> uses pyarrow to convert to pandas DataFrame, by collecting the batches,
+passing them to an Arrow table, and then converting them to a pandas DataFrame.</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">to_pandas</span><span class="p">()</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">8</span><span class="p">]:</span>
+ <span class="n">nrs</span> <span class="n">names</span> <span class="n">random</span> <span class="n">groups</span>
+<span class="mi">0</span> <span class="mi">1</span> <span class="n">python</span> <span class="mi">216</span> <span class="n">A</span>
+<span class="mi">1</span> <span class="mi">2</span> <span class="n">ruby</span> <span class="mi">121</span> <span class="n">A</span>
+<span class="mi">2</span> <span class="mi">3</span> <span class="n">java</span> <span class="mi">567</span> <span class="n">B</span>
+<span class="mi">3</span> <span class="mi">4</span> <span class="n">haskell</span> <span class="mi">680</span> <span class="n">C</span>
+<span class="mi">4</span> <span class="mi">5</span> <span class="n">go</span> <span class="mi">186</span> <span class="n">B</span>
+</pre></div>
+</div>
+<p><a class="reference internal" href="../../generated/datafusion.DataFrame.html#datafusion.DataFrame.describe" title="datafusion.DataFrame.describe"><code class="xref py py-meth docutils literal notranslate"><span class="pre">DataFrame.describe()</span></code></a> shows a quick statistic summary of your data:</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">9</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">describe</span><span class="p">()</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">9</span><span class="p">]:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+------------+--------------------+-------+--------------------+--------+</span>
+<span class="o">|</span> <span class="n">describe</span> <span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span> <span class="n">names</span> <span class="o">|</span> <span class="n">random</span> <span class="o">|</span> <span class="n">groups</span> <span class="o">|</span>
+<span class="o">+------------+--------------------+-------+--------------------+--------+</span>
+<span class="o">|</span> <span class="n">count</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="mi">5</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">null_count</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="mi">5</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">mean</span> <span class="o">|</span> <span class="mf">3.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">354.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">std</span> <span class="o">|</span> <span class="mf">1.5811388300841898</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">251.59590616701217</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="nb">min</span> <span class="o">|</span> <span class="mf">1.0</span> <span class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span class="mf">121.0</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="nb">max</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mf">680.0</span> <span class="o">|</span> <span class="n">C</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">median</span> <span class="o">|</span> <span class="mf">3.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">216.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
+<span class="o">+------------+--------------------+-------+--------------------+--------+</span>
+</pre></div>
+</div>
+</section>
+
+
+ </div>
+
+
+ <!-- Previous / next buttons -->
+<div class='prev-next-area'>
+ <a class='left-prev' id="prev-link" href="index.html" title="previous page">
+ <i class="fas fa-angle-left"></i>
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">previous</p>
+ <p class="prev-next-title">Common Operations</p>
+ </div>
+ </a>
+ <a class='right-next' id="next-link" href="select-and-filter.html" title="next page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">Column Selections</p>
+ </div>
+ <i class="fas fa-angle-right"></i>
+ </a>
+</div>
+
+ </main>
+
+
+ </div>
+ </div>
+
+ <script src="../../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
+ <footer class="footer mt-5 mt-md-0">
+ <div class="container">
+
+ <div class="footer-item">
+ <p class="copyright">
+ © Copyright 2022, Apache Software Foundation.<br>
+</p>
+ </div>
+
+ <div class="footer-item">
+ <p class="sphinx-version">
+Created using <a href="http://sphinx-doc.org/">Sphinx</a> 5.3.0.<br>
+</p>
+ </div>
+
+ </div>
+</footer>
+ </body>
+</html>
\ No newline at end of file
diff --git a/user-guide/common-operations/functions.html b/user-guide/common-operations/functions.html
new file mode 100644
index 0000000..71baca3
--- /dev/null
+++ b/user-guide/common-operations/functions.html
@@ -0,0 +1,660 @@
+
+<!DOCTYPE html>
+
+<html lang="en">
+ <head>
+ <meta charset="utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
+
+ <title>Functions — Arrow DataFusion documentation</title>
+
+ <link href="../../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+<link href="../../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+
+
+ <link rel="stylesheet"
+ href="../../_static/vendor/fontawesome/5.13.0/css/all.min.css">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
+
+
+
+
+
+ <link rel="stylesheet" type="text/css" href="../../_static/pygments.css" />
+ <link rel="stylesheet" type="text/css" href="../../_static/styles/pydata-sphinx-theme.css" />
+ <link rel="stylesheet" type="text/css" href="../../_static/theme_overrides.css" />
+
+ <link rel="preload" as="script" href="../../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
+
+ <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script>
+ <script src="../../_static/jquery.js"></script>
+ <script src="../../_static/underscore.js"></script>
+ <script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script>
+ <script src="../../_static/doctools.js"></script>
+ <script src="../../_static/sphinx_highlight.js"></script>
+ <link rel="index" title="Index" href="../../genindex.html" />
+ <link rel="search" title="Search" href="../../search.html" />
+ <link rel="next" title="Aggregation" href="aggregations.html" />
+ <link rel="prev" title="Joins" href="joins.html" />
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
+ <meta name="docsearch:language" content="en">
+
+
+ <!-- Google Analytics -->
+
+ </head>
+ <body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
+
+ <div class="container-fluid" id="banner"></div>
+
+
+
+
+ <div class="container-xl">
+ <div class="row">
+
+
+ <!-- Only show if we have sidebars configured, else just a small margin -->
+ <div class="col-12 col-md-3 bd-sidebar">
+ <div class="sidebar-start-items">
+<a class="navbar-brand" href="../../index.html">
+ <img src="../../_static/images/DataFusion-Logo-Background-White.png" class="logo" alt="logo">
+</a>
+
+<form class="bd-search d-flex align-items-center" action="../../search.html" method="get">
+ <i class="icon fas fa-search"></i>
+ <input type="search" class="form-control" name="q" id="search-input" placeholder="Search the docs ..." aria-label="Search the docs ..." autocomplete="off" >
+</form>
+
+<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
+ <div class="bd-toc-item active">
+
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 current active has-children">
+ <a class="reference internal" href="index.html">
+ Common Operations
+ </a>
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul class="current">
+ <li class="toctree-l2">
+ <a class="reference internal" href="basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2 current active">
+ <a class="current reference internal" href="#">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../../api.html">
+ API Reference
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/dataframe.html">
+ DataFrame
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.DataFrame.html">
+ datafusion.DataFrame
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/execution_context.html">
+ SessionContext
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.SessionConfig.html">
+ datafusion.SessionConfig
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.RuntimeConfig.html">
+ datafusion.RuntimeConfig
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.SessionContext.html">
+ datafusion.SessionContext
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/expression.html">
+ Expr
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.Expr.html">
+ datafusion.Expr
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/functions.html">
+ Functions
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.functions.functions.html">
+ datafusion.functions.functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/object_store.html">
+ ObjectStore
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.object_store.object_store.html">
+ datafusion.object_store.object_store
+ </a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+</ul>
+
+
+ </div>
+</nav>
+ </div>
+ <div class="sidebar-end-items">
+ </div>
+ </div>
+
+
+
+
+ <div class="d-none d-xl-block col-xl-2 bd-toc">
+
+
+ <div class="toc-item">
+
+<div class="tocsection onthispage pt-5 pb-3">
+ <i class="fas fa-list"></i> On this page
+</div>
+
+<nav id="bd-toc-nav">
+ <ul class="visible nav section-nav flex-column">
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#mathematical">
+ Mathematical
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#conditional">
+ Conditional
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#temporal">
+ Temporal
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#string">
+ String
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#other">
+ Other
+ </a>
+ </li>
+</ul>
+
+</nav>
+ </div>
+
+ <div class="toc-item">
+
+
+<div class="tocsection editthispage">
+ <a href="https://github.com/apache/arrow-datafusion-python/edit/main/docs/source/user-guide/common-operations/functions.rst">
+ <i class="fas fa-pencil-alt"></i> Edit this page
+ </a>
+</div>
+
+ </div>
+
+
+ </div>
+
+
+
+
+
+
+ <main class="col-12 col-md-9 col-xl-7 py-md-5 pl-md-5 pr-md-4 bd-content" role="main">
+
+ <div>
+
+ <section id="functions">
+<h1>Functions<a class="headerlink" href="#functions" title="Permalink to this heading">¶</a></h1>
+<p>DataFusion provides a large number of built-in functions for performing complex queries without requiring user-defined functions.
+In here we will cover some of the more popular use cases. If you want to view all the functions go to the <span class="xref std std-ref">Functions</span> API Reference.</p>
+<p>We’ll use the pokemon dataset in the following examples.</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="kn">import</span> <span class="nn">urllib.request</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">SessionContext</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlretrieve</span><span class="p">(</span>
+ <span class="o">...</span><span class="p">:</span> <span class="s2">"https://gist.githubusercontent.com/ritchie46/cac6b337ea52281aa23c049250a4ff03/raw/89a957ff3919d90e6ef2d34235e6bf22304f3366/pokemon.csv"</span><span class="p">,</span>
+ <span class="o">...</span><span class="p">:</span> <span class="s2">"pokemon.csv"</span><span class="p">,</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">)</span>
+ <span class="o">...</span><span class="p">:</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="p">(</span><span class="s1">'pokemon.csv'</span><span class="p">,</span> <span class="o"><</span><span class="n">http</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">HTTPMessage</span> <span class="n">at</span> <span class="mh">0x7fb46f092500</span><span class="o">></span><span class="p">)</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">4</span><span class="p">]:</span> <span class="n">ctx</span> <span class="o">=</span> <span class="n">SessionContext</span><span class="p">()</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">5</span><span class="p">]:</span> <span class="n">ctx</span><span class="o">.</span><span class="n">register_csv</span><span class="p">(</span><span class="s2">"pokemon"</span><span class="p">,</span> <span class="s2">"pokemon.csv"</span><span class="p">)</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">6</span><span class="p">]:</span> <span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">table</span><span class="p">(</span><span class="s2">"pokemon"</span><span class="p">)</span>
+</pre></div>
+</div>
+<section id="mathematical">
+<h2>Mathematical<a class="headerlink" href="#mathematical" title="Permalink to this heading">¶</a></h2>
+<p>DataFusion offers mathematical functions such as <a class="reference internal" href="../../generated/datafusion.functions.functions.html#datafusion.functions.functions.pow" title="datafusion.functions.functions.pow"><code class="xref py py-func docutils literal notranslate"><span class="pre">pow()</span></code></a> or <a class="reference internal" href="../../generated/datafusion.functions.functions.html#datafusion.functions.functions.log" title="datafusion.functions.functions.log"><c [...]
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">7</span><span class="p">]:</span> <span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">col</span><span class="p">,</span> <span class="n">literal</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">functions</span> <span class="k">as</span> <span class="n">f</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">9</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+ <span class="o">...</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">pow</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s1">'"Attack"'</span><span class="p">),</span> <span class="n">literal</span><span class="p">(</span><span class="mi">2</span><span class="p">))</span> <span class="o">-</span> <span class="n">f</span><span class="o">.</span><span class="n">pow</span><spa [...]
+ <span class="o">...</span><span class="p">:</span> <span class="p">)</span><span class="o">.</span><span class="n">limit</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
+ <span class="o">...</span><span class="p">:</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">9</span><span class="p">]:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+------------------------------------------------------------------+</span>
+<span class="o">|</span> <span class="n">power</span><span class="p">(</span><span class="n">pokemon</span><span class="o">.</span><span class="n">Attack</span><span class="p">,</span><span class="n">Int64</span><span class="p">(</span><span class="mi">2</span><span class="p">))</span> <span class="o">-</span> <span class="n">power</span><span class="p">(</span><span class="n">pokemon</span><span class="o">.</span><span class="n">Defense</span><span class="p">,</span><span class="n">Int6 [...]
+<span class="o">+------------------------------------------------------------------+</span>
+<span class="o">|</span> <span class="mi">0</span> <span class="o">|</span>
+<span class="o">|</span> <span class="o">-</span><span class="mi">125</span> <span class="o">|</span>
+<span class="o">|</span> <span class="o">-</span><span class="mi">165</span> <span class="o">|</span>
+<span class="o">|</span> <span class="o">-</span><span class="mi">5129</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">855</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">732</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">972</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">4579</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">4732</span> <span class="o">|</span>
+<span class="o">|</span> <span class="o">-</span><span class="mi">1921</span> <span class="o">|</span>
+<span class="o">+------------------------------------------------------------------+</span>
+</pre></div>
+</div>
+</section>
+<section id="conditional">
+<h2>Conditional<a class="headerlink" href="#conditional" title="Permalink to this heading">¶</a></h2>
+<p>There 3 conditional functions in DataFusion <a class="reference internal" href="../../generated/datafusion.functions.functions.html#datafusion.functions.functions.coalesce" title="datafusion.functions.functions.coalesce"><code class="xref py py-func docutils literal notranslate"><span class="pre">coalesce()</span></code></a>, <a class="reference internal" href="../../generated/datafusion.functions.functions.html#datafusion.functions.functions.nullif" title="datafusion.functions.functi [...]
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">10</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+ <span class="o">....</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">coalesce</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s1">'"Type 1"'</span><span class="p">),</span> <span class="n">col</span><span class="p">(</span><span class="s1">'"Type 2"'</span><span class="p">))</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span [...]
+ <span class="o">....</span><span class="p">:</span> <span class="p">)</span><span class="o">.</span><span class="n">limit</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
+ <span class="o">....</span><span class="p">:</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">10</span><span class="p">]:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+---------------+</span>
+<span class="o">|</span> <span class="n">dominant_type</span> <span class="o">|</span>
+<span class="o">+---------------+</span>
+<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span>
+<span class="o">+---------------+</span>
+</pre></div>
+</div>
+</section>
+<section id="temporal">
+<h2>Temporal<a class="headerlink" href="#temporal" title="Permalink to this heading">¶</a></h2>
+<p>For selecting the current time use <a class="reference internal" href="../../generated/datafusion.functions.functions.html#datafusion.functions.functions.now" title="datafusion.functions.functions.now"><code class="xref py py-func docutils literal notranslate"><span class="pre">now()</span></code></a></p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">11</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">now</span><span class="p">())</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">11</span><span class="p">]:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+--------------------------------+</span>
+<span class="o">|</span> <span class="n">now</span><span class="p">()</span> <span class="o">|</span>
+<span class="o">+--------------------------------+</span>
+<span class="o">|</span> <span class="mi">2023</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">18</span><span class="n">T14</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span><span class="mf">46.289179725</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2023</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">18</span><span class="n">T14</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span><span class="mf">46.289179725</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2023</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">18</span><span class="n">T14</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span><span class="mf">46.289179725</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2023</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">18</span><span class="n">T14</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span><span class="mf">46.289179725</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2023</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">18</span><span class="n">T14</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span><span class="mf">46.289179725</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2023</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">18</span><span class="n">T14</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span><span class="mf">46.289179725</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2023</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">18</span><span class="n">T14</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span><span class="mf">46.289179725</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2023</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">18</span><span class="n">T14</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span><span class="mf">46.289179725</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2023</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">18</span><span class="n">T14</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span><span class="mf">46.289179725</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2023</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">18</span><span class="n">T14</span><span class="p">:</span><span class="mi">17</span><span class="p">:</span><span class="mf">46.289179725</span><span class="n">Z</span> <span class="o">|</span>
+<span class="o">+--------------------------------+</span>
+</pre></div>
+</div>
+<p>Convert to timestamps using <a class="reference internal" href="../../generated/datafusion.functions.functions.html#datafusion.functions.functions.to_timestamp" title="datafusion.functions.functions.to_timestamp"><code class="xref py py-func docutils literal notranslate"><span class="pre">to_timestamp()</span></code></a></p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">12</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">to_timestamp</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s1">'"Total"'</s [...]
+<span class="n">Out</span><span class="p">[</span><span class="mi">12</span><span class="p">]:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+-------------------------------+</span>
+<span class="o">|</span> <span class="n">timestamp</span> <span class="o">|</span>
+<span class="o">+-------------------------------+</span>
+<span class="o">|</span> <span class="mi">1970</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">01</span><span class="n">T00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mf">00.000000318</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">1970</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">01</span><span class="n">T00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mf">00.000000405</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">1970</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">01</span><span class="n">T00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mf">00.000000525</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">1970</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">01</span><span class="n">T00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mf">00.000000625</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">1970</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">01</span><span class="n">T00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mf">00.000000309</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">1970</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">01</span><span class="n">T00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mf">00.000000405</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">1970</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">01</span><span class="n">T00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mf">00.000000534</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">1970</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">01</span><span class="n">T00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mf">00.000000634</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">1970</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">01</span><span class="n">T00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mf">00.000000634</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">1970</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">01</span><span class="n">T00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mf">00.000000314</span> <span class="o">|</span>
+<span class="o">+-------------------------------+</span>
+</pre></div>
+</div>
+</section>
+<section id="string">
+<h2>String<a class="headerlink" href="#string" title="Permalink to this heading">¶</a></h2>
+<p>In the field of data science, working with textual data is a common task. To make string manipulation easier,
+DataFusion offers a range of helpful options.</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">13</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+ <span class="o">....</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">char_length</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s1">'"Name"'</span><span class="p">))</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"len"</span><span class="p">),</span>
+ <span class="o">....</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">lower</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s1">'"Name"'</span><span class="p">))</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"lower"</span><span class="p">),</span>
+ <span class="o">....</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">left</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s1">'"Name"'</span><span class="p">),</span> <span class="n">literal</span><span class="p">(</span><span class="mi">4</span><span class="p">))</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"code& [...]
+ <span class="o">....</span><span class="p">:</span> <span class="p">)</span>
+ <span class="o">....</span><span class="p">:</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">13</span><span class="p">]:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+-----+---------------------------+------+</span>
+<span class="o">|</span> <span class="nb">len</span> <span class="o">|</span> <span class="n">lower</span> <span class="o">|</span> <span class="n">code</span> <span class="o">|</span>
+<span class="o">+-----+---------------------------+------+</span>
+<span class="o">|</span> <span class="mi">9</span> <span class="o">|</span> <span class="n">bulbasaur</span> <span class="o">|</span> <span class="n">Bulb</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">7</span> <span class="o">|</span> <span class="n">ivysaur</span> <span class="o">|</span> <span class="n">Ivys</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">8</span> <span class="o">|</span> <span class="n">venusaur</span> <span class="o">|</span> <span class="n">Venu</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">21</span> <span class="o">|</span> <span class="n">venusaurmega</span> <span class="n">venusaur</span> <span class="o">|</span> <span class="n">Venu</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">10</span> <span class="o">|</span> <span class="n">charmander</span> <span class="o">|</span> <span class="n">Char</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">10</span> <span class="o">|</span> <span class="n">charmeleon</span> <span class="o">|</span> <span class="n">Char</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">9</span> <span class="o">|</span> <span class="n">charizard</span> <span class="o">|</span> <span class="n">Char</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">25</span> <span class="o">|</span> <span class="n">charizardmega</span> <span class="n">charizard</span> <span class="n">x</span> <span class="o">|</span> <span class="n">Char</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">25</span> <span class="o">|</span> <span class="n">charizardmega</span> <span class="n">charizard</span> <span class="n">y</span> <span class="o">|</span> <span class="n">Char</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">8</span> <span class="o">|</span> <span class="n">squirtle</span> <span class="o">|</span> <span class="n">Squi</span> <span class="o">|</span>
+<span class="o">+-----+---------------------------+------+</span>
+</pre></div>
+</div>
+<p>This also includes the functions for regular expressions <a class="reference internal" href="../../generated/datafusion.functions.functions.html#datafusion.functions.functions.regexp_replace" title="datafusion.functions.functions.regexp_replace"><code class="xref py py-func docutils literal notranslate"><span class="pre">regexp_replace()</span></code></a> and <a class="reference internal" href="../../generated/datafusion.functions.functions.html#datafusion.functions.functions.regexp_m [...]
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">14</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+ <span class="o">....</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">regexp_match</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s1">'"Name"'</span><span class="p">),</span> <span class="n">literal</span><span class="p">(</span><span class="s2">"Char"</span><span class="p">))</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span [...]
+ <span class="o">....</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">regexp_replace</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s1">'"Name"'</span><span class="p">),</span> <span class="n">literal</span><span class="p">(</span><span class="s2">"saur"</span><span class="p">),</span> <span class="n">literal</span><span class="p">(</span><span class="s2">"f [...]
+ <span class="o">....</span><span class="p">:</span> <span class="p">)</span>
+ <span class="o">....</span><span class="p">:</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">14</span><span class="p">]:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+---------+---------------------------+</span>
+<span class="o">|</span> <span class="n">dragons</span> <span class="o">|</span> <span class="n">flowers</span> <span class="o">|</span>
+<span class="o">+---------+---------------------------+</span>
+<span class="o">|</span> <span class="o">|</span> <span class="n">Bulbafleur</span> <span class="o">|</span>
+<span class="o">|</span> <span class="o">|</span> <span class="n">Ivyfleur</span> <span class="o">|</span>
+<span class="o">|</span> <span class="o">|</span> <span class="n">Venufleur</span> <span class="o">|</span>
+<span class="o">|</span> <span class="o">|</span> <span class="n">VenufleurMega</span> <span class="n">Venusaur</span> <span class="o">|</span>
+<span class="o">|</span> <span class="p">[</span><span class="n">Char</span><span class="p">]</span> <span class="o">|</span> <span class="n">Charmander</span> <span class="o">|</span>
+<span class="o">|</span> <span class="p">[</span><span class="n">Char</span><span class="p">]</span> <span class="o">|</span> <span class="n">Charmeleon</span> <span class="o">|</span>
+<span class="o">|</span> <span class="p">[</span><span class="n">Char</span><span class="p">]</span> <span class="o">|</span> <span class="n">Charizard</span> <span class="o">|</span>
+<span class="o">|</span> <span class="p">[</span><span class="n">Char</span><span class="p">]</span> <span class="o">|</span> <span class="n">CharizardMega</span> <span class="n">Charizard</span> <span class="n">X</span> <span class="o">|</span>
+<span class="o">|</span> <span class="p">[</span><span class="n">Char</span><span class="p">]</span> <span class="o">|</span> <span class="n">CharizardMega</span> <span class="n">Charizard</span> <span class="n">Y</span> <span class="o">|</span>
+<span class="o">|</span> <span class="o">|</span> <span class="n">Squirtle</span> <span class="o">|</span>
+<span class="o">+---------+---------------------------+</span>
+</pre></div>
+</div>
+</section>
+<section id="other">
+<h2>Other<a class="headerlink" href="#other" title="Permalink to this heading">¶</a></h2>
+<p>The function <a class="reference internal" href="../../generated/datafusion.functions.functions.html#datafusion.functions.functions.in_list" title="datafusion.functions.functions.in_list"><code class="xref py py-func docutils literal notranslate"><span class="pre">in_list()</span></code></a> allows to check a column for the presence of multiple values:</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">15</span><span class="p">]:</span> <span class="n">types</span> <span class="o">=</span> <span class="p">[</span><span class="n">literal</span><span class="p">(</span><span class="s2">"Grass"</span><span class="p">),</span> <span class="n">literal</span><span class="p">(</span><span class="s2">"Fire"</span><span class=" [...]
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">16</span><span class="p">]:</span> <span class="p">(</span>
+ <span class="o">....</span><span class="p">:</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">in_list</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s1">'"Type 1"'</span><span class="p">),</span> <span class="n">types</span><span class="p">,</span> <span class="n">negated</span><span class="o">=< [...]
+ <span class="o">....</span><span class="p">:</span> <span class="o">.</span><span class="n">limit</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span>
+ <span class="o">....</span><span class="p">:</span> <span class="o">.</span><span class="n">to_pandas</span><span class="p">()</span>
+ <span class="o">....</span><span class="p">:</span> <span class="p">)</span>
+ <span class="o">....</span><span class="p">:</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">16</span><span class="p">]:</span>
+ <span class="n">basic_types</span>
+<span class="mi">0</span> <span class="kc">True</span>
+<span class="mi">1</span> <span class="kc">True</span>
+<span class="mi">2</span> <span class="kc">True</span>
+<span class="mi">3</span> <span class="kc">True</span>
+<span class="mi">4</span> <span class="kc">True</span>
+<span class="mi">5</span> <span class="kc">True</span>
+<span class="mi">6</span> <span class="kc">True</span>
+<span class="mi">7</span> <span class="kc">True</span>
+<span class="mi">8</span> <span class="kc">True</span>
+<span class="mi">9</span> <span class="kc">True</span>
+<span class="mi">10</span> <span class="kc">True</span>
+<span class="mi">11</span> <span class="kc">True</span>
+<span class="mi">12</span> <span class="kc">True</span>
+<span class="mi">13</span> <span class="kc">False</span>
+<span class="mi">14</span> <span class="kc">False</span>
+<span class="mi">15</span> <span class="kc">False</span>
+<span class="mi">16</span> <span class="kc">False</span>
+<span class="mi">17</span> <span class="kc">False</span>
+<span class="mi">18</span> <span class="kc">False</span>
+<span class="mi">19</span> <span class="kc">False</span>
+</pre></div>
+</div>
+</section>
+</section>
+
+
+ </div>
+
+
+ <!-- Previous / next buttons -->
+<div class='prev-next-area'>
+ <a class='left-prev' id="prev-link" href="joins.html" title="previous page">
+ <i class="fas fa-angle-left"></i>
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">previous</p>
+ <p class="prev-next-title">Joins</p>
+ </div>
+ </a>
+ <a class='right-next' id="next-link" href="aggregations.html" title="next page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">Aggregation</p>
+ </div>
+ <i class="fas fa-angle-right"></i>
+ </a>
+</div>
+
+ </main>
+
+
+ </div>
+ </div>
+
+ <script src="../../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
+ <footer class="footer mt-5 mt-md-0">
+ <div class="container">
+
+ <div class="footer-item">
+ <p class="copyright">
+ © Copyright 2022, Apache Software Foundation.<br>
+</p>
+ </div>
+
+ <div class="footer-item">
+ <p class="sphinx-version">
+Created using <a href="http://sphinx-doc.org/">Sphinx</a> 5.3.0.<br>
+</p>
+ </div>
+
+ </div>
+</footer>
+ </body>
+</html>
\ No newline at end of file
diff --git a/user-guide/common-operations/index.html b/user-guide/common-operations/index.html
new file mode 100644
index 0000000..6715abf
--- /dev/null
+++ b/user-guide/common-operations/index.html
@@ -0,0 +1,446 @@
+
+<!DOCTYPE html>
+
+<html lang="en">
+ <head>
+ <meta charset="utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
+
+ <title>Common Operations — Arrow DataFusion documentation</title>
+
+ <link href="../../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+<link href="../../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+
+
+ <link rel="stylesheet"
+ href="../../_static/vendor/fontawesome/5.13.0/css/all.min.css">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
+
+
+
+
+
+ <link rel="stylesheet" type="text/css" href="../../_static/pygments.css" />
+ <link rel="stylesheet" type="text/css" href="../../_static/styles/pydata-sphinx-theme.css" />
+ <link rel="stylesheet" type="text/css" href="../../_static/theme_overrides.css" />
+
+ <link rel="preload" as="script" href="../../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
+
+ <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script>
+ <script src="../../_static/jquery.js"></script>
+ <script src="../../_static/underscore.js"></script>
+ <script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script>
+ <script src="../../_static/doctools.js"></script>
+ <script src="../../_static/sphinx_highlight.js"></script>
+ <link rel="index" title="Index" href="../../genindex.html" />
+ <link rel="search" title="Search" href="../../search.html" />
+ <link rel="next" title="Basic Operations" href="basic-info.html" />
+ <link rel="prev" title="Concepts" href="../basics.html" />
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
+ <meta name="docsearch:language" content="en">
+
+
+ <!-- Google Analytics -->
+
+ </head>
+ <body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
+
+ <div class="container-fluid" id="banner"></div>
+
+
+
+
+ <div class="container-xl">
+ <div class="row">
+
+
+ <!-- Only show if we have sidebars configured, else just a small margin -->
+ <div class="col-12 col-md-3 bd-sidebar">
+ <div class="sidebar-start-items">
+<a class="navbar-brand" href="../../index.html">
+ <img src="../../_static/images/DataFusion-Logo-Background-White.png" class="logo" alt="logo">
+</a>
+
+<form class="bd-search d-flex align-items-center" action="../../search.html" method="get">
+ <i class="icon fas fa-search"></i>
+ <input type="search" class="form-control" name="q" id="search-input" placeholder="Search the docs ..." aria-label="Search the docs ..." autocomplete="off" >
+</form>
+
+<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
+ <div class="bd-toc-item active">
+
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 current active has-children">
+ <a class="current reference internal" href="#">
+ Common Operations
+ </a>
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="joins.html">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../../api.html">
+ API Reference
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/dataframe.html">
+ DataFrame
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.DataFrame.html">
+ datafusion.DataFrame
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/execution_context.html">
+ SessionContext
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.SessionConfig.html">
+ datafusion.SessionConfig
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.RuntimeConfig.html">
+ datafusion.RuntimeConfig
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.SessionContext.html">
+ datafusion.SessionContext
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/expression.html">
+ Expr
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.Expr.html">
+ datafusion.Expr
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/functions.html">
+ Functions
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.functions.functions.html">
+ datafusion.functions.functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/object_store.html">
+ ObjectStore
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.object_store.object_store.html">
+ datafusion.object_store.object_store
+ </a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+</ul>
+
+
+ </div>
+</nav>
+ </div>
+ <div class="sidebar-end-items">
+ </div>
+ </div>
+
+
+
+
+ <div class="d-none d-xl-block col-xl-2 bd-toc">
+
+
+ <div class="toc-item">
+
+
+<nav id="bd-toc-nav">
+
+</nav>
+ </div>
+
+ <div class="toc-item">
+
+
+<div class="tocsection editthispage">
+ <a href="https://github.com/apache/arrow-datafusion-python/edit/main/docs/source/user-guide/common-operations/index.rst">
+ <i class="fas fa-pencil-alt"></i> Edit this page
+ </a>
+</div>
+
+ </div>
+
+
+ </div>
+
+
+
+
+
+
+ <main class="col-12 col-md-9 col-xl-7 py-md-5 pl-md-5 pr-md-4 bd-content" role="main">
+
+ <div>
+
+ <section id="common-operations">
+<h1>Common Operations<a class="headerlink" href="#common-operations" title="Permalink to this heading">¶</a></h1>
+<div class="toctree-wrapper compound">
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="basic-info.html">Basic Operations</a></li>
+<li class="toctree-l1"><a class="reference internal" href="select-and-filter.html">Column Selections</a></li>
+<li class="toctree-l1"><a class="reference internal" href="joins.html">Joins</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="joins.html#inner-join">Inner Join</a></li>
+<li class="toctree-l2"><a class="reference internal" href="joins.html#left-join">Left Join</a></li>
+<li class="toctree-l2"><a class="reference internal" href="joins.html#full-join">Full Join</a></li>
+<li class="toctree-l2"><a class="reference internal" href="joins.html#left-semi-join">Left Semi Join</a></li>
+<li class="toctree-l2"><a class="reference internal" href="joins.html#left-anti-join">Left Anti Join</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="functions.html">Functions</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="functions.html#mathematical">Mathematical</a></li>
+<li class="toctree-l2"><a class="reference internal" href="functions.html#conditional">Conditional</a></li>
+<li class="toctree-l2"><a class="reference internal" href="functions.html#temporal">Temporal</a></li>
+<li class="toctree-l2"><a class="reference internal" href="functions.html#string">String</a></li>
+<li class="toctree-l2"><a class="reference internal" href="functions.html#other">Other</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="aggregations.html">Aggregation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="windows.html">Window Functions</a></li>
+<li class="toctree-l1"><a class="reference internal" href="udf-and-udfa.html">User Defined Functions</a></li>
+</ul>
+</div>
+</section>
+
+
+ </div>
+
+
+ <!-- Previous / next buttons -->
+<div class='prev-next-area'>
+ <a class='left-prev' id="prev-link" href="../basics.html" title="previous page">
+ <i class="fas fa-angle-left"></i>
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">previous</p>
+ <p class="prev-next-title">Concepts</p>
+ </div>
+ </a>
+ <a class='right-next' id="next-link" href="basic-info.html" title="next page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">Basic Operations</p>
+ </div>
+ <i class="fas fa-angle-right"></i>
+ </a>
+</div>
+
+ </main>
+
+
+ </div>
+ </div>
+
+ <script src="../../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
+ <footer class="footer mt-5 mt-md-0">
+ <div class="container">
+
+ <div class="footer-item">
+ <p class="copyright">
+ © Copyright 2022, Apache Software Foundation.<br>
+</p>
+ </div>
+
+ <div class="footer-item">
+ <p class="sphinx-version">
+Created using <a href="http://sphinx-doc.org/">Sphinx</a> 5.3.0.<br>
+</p>
+ </div>
+
+ </div>
+</footer>
+ </body>
+</html>
\ No newline at end of file
diff --git a/user-guide/common-operations/joins.html b/user-guide/common-operations/joins.html
new file mode 100644
index 0000000..81ebeca
--- /dev/null
+++ b/user-guide/common-operations/joins.html
@@ -0,0 +1,567 @@
+
+<!DOCTYPE html>
+
+<html lang="en">
+ <head>
+ <meta charset="utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
+
+ <title>Joins — Arrow DataFusion documentation</title>
+
+ <link href="../../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+<link href="../../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+
+
+ <link rel="stylesheet"
+ href="../../_static/vendor/fontawesome/5.13.0/css/all.min.css">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
+
+
+
+
+
+ <link rel="stylesheet" type="text/css" href="../../_static/pygments.css" />
+ <link rel="stylesheet" type="text/css" href="../../_static/styles/pydata-sphinx-theme.css" />
+ <link rel="stylesheet" type="text/css" href="../../_static/theme_overrides.css" />
+
+ <link rel="preload" as="script" href="../../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
+
+ <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script>
+ <script src="../../_static/jquery.js"></script>
+ <script src="../../_static/underscore.js"></script>
+ <script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script>
+ <script src="../../_static/doctools.js"></script>
+ <script src="../../_static/sphinx_highlight.js"></script>
+ <link rel="index" title="Index" href="../../genindex.html" />
+ <link rel="search" title="Search" href="../../search.html" />
+ <link rel="next" title="Functions" href="functions.html" />
+ <link rel="prev" title="Column Selections" href="select-and-filter.html" />
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
+ <meta name="docsearch:language" content="en">
+
+
+ <!-- Google Analytics -->
+
+ </head>
+ <body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
+
+ <div class="container-fluid" id="banner"></div>
+
+
+
+
+ <div class="container-xl">
+ <div class="row">
+
+
+ <!-- Only show if we have sidebars configured, else just a small margin -->
+ <div class="col-12 col-md-3 bd-sidebar">
+ <div class="sidebar-start-items">
+<a class="navbar-brand" href="../../index.html">
+ <img src="../../_static/images/DataFusion-Logo-Background-White.png" class="logo" alt="logo">
+</a>
+
+<form class="bd-search d-flex align-items-center" action="../../search.html" method="get">
+ <i class="icon fas fa-search"></i>
+ <input type="search" class="form-control" name="q" id="search-input" placeholder="Search the docs ..." aria-label="Search the docs ..." autocomplete="off" >
+</form>
+
+<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
+ <div class="bd-toc-item active">
+
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ LINKS
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion-python">
+ Github and Issue Tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/">
+ Rust's API Docs
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-datafusion/blob/main/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ USER GUIDE
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../introduction.html">
+ Introduction
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../basics.html">
+ Concepts
+ </a>
+ </li>
+ <li class="toctree-l1 current active has-children">
+ <a class="reference internal" href="index.html">
+ Common Operations
+ </a>
+ <input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul class="current">
+ <li class="toctree-l2">
+ <a class="reference internal" href="basic-info.html">
+ Basic Operations
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="select-and-filter.html">
+ Column Selections
+ </a>
+ </li>
+ <li class="toctree-l2 current active">
+ <a class="current reference internal" href="#">
+ Joins
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="functions.html">
+ Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="aggregations.html">
+ Aggregation
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="windows.html">
+ Window Functions
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="udf-and-udfa.html">
+ User Defined Functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../io/index.html">
+ IO
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-2" name="toctree-checkbox-2" type="checkbox"/>
+ <label for="toctree-checkbox-2">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/csv.html">
+ CSV
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/parquet.html">
+ Parquet
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/json.html">
+ JSON
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="../io/avro.html">
+ Avro
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../sql.html">
+ SQL
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ CONTRIBUTOR GUIDE
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../../contributor-guide/introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ API
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="../../api.html">
+ API Reference
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-3" name="toctree-checkbox-3" type="checkbox"/>
+ <label for="toctree-checkbox-3">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/dataframe.html">
+ DataFrame
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-4" name="toctree-checkbox-4" type="checkbox"/>
+ <label for="toctree-checkbox-4">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.DataFrame.html">
+ datafusion.DataFrame
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/execution_context.html">
+ SessionContext
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-5" name="toctree-checkbox-5" type="checkbox"/>
+ <label for="toctree-checkbox-5">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.SessionConfig.html">
+ datafusion.SessionConfig
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.RuntimeConfig.html">
+ datafusion.RuntimeConfig
+ </a>
+ </li>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.SessionContext.html">
+ datafusion.SessionContext
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/expression.html">
+ Expr
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-6" name="toctree-checkbox-6" type="checkbox"/>
+ <label for="toctree-checkbox-6">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.Expr.html">
+ datafusion.Expr
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/functions.html">
+ Functions
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/>
+ <label for="toctree-checkbox-7">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.functions.functions.html">
+ datafusion.functions.functions
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l2 has-children">
+ <a class="reference internal" href="../../api/object_store.html">
+ ObjectStore
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-8" name="toctree-checkbox-8" type="checkbox"/>
+ <label for="toctree-checkbox-8">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l3">
+ <a class="reference internal" href="../../generated/datafusion.object_store.object_store.html">
+ datafusion.object_store.object_store
+ </a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+</ul>
+
+
+ </div>
+</nav>
+ </div>
+ <div class="sidebar-end-items">
+ </div>
+ </div>
+
+
+
+
+ <div class="d-none d-xl-block col-xl-2 bd-toc">
+
+
+ <div class="toc-item">
+
+<div class="tocsection onthispage pt-5 pb-3">
+ <i class="fas fa-list"></i> On this page
+</div>
+
+<nav id="bd-toc-nav">
+ <ul class="visible nav section-nav flex-column">
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#inner-join">
+ Inner Join
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#left-join">
+ Left Join
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#full-join">
+ Full Join
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#left-semi-join">
+ Left Semi Join
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#left-anti-join">
+ Left Anti Join
+ </a>
+ </li>
+</ul>
+
+</nav>
+ </div>
+
+ <div class="toc-item">
+
+
+<div class="tocsection editthispage">
+ <a href="https://github.com/apache/arrow-datafusion-python/edit/main/docs/source/user-guide/common-operations/joins.rst">
+ <i class="fas fa-pencil-alt"></i> Edit this page
+ </a>
+</div>
+
+ </div>
+
+
+ </div>
+
+
+
+
+
+
+ <main class="col-12 col-md-9 col-xl-7 py-md-5 pl-md-5 pr-md-4 bd-content" role="main">
+
+ <div>
+
+ <section id="joins">
+<h1>Joins<a class="headerlink" href="#joins" title="Permalink to this heading">¶</a></h1>
+<p>DataFusion supports the following join variants via the method <a class="reference internal" href="../../generated/datafusion.DataFrame.html#datafusion.DataFrame.join" title="datafusion.DataFrame.join"><code class="xref py py-meth docutils literal notranslate"><span class="pre">DataFrame.join()</span></code></a></p>
+<ul class="simple">
+<li><p>Inner Join</p></li>
+<li><p>Left Join</p></li>
+<li><p>Right Join</p></li>
+<li><p>Full Join</p></li>
+<li><p>Left Semi Join</p></li>
+<li><p>Left Anti Join</p></li>
+</ul>
+<p>For the examples in this section we’ll use the following two DataFrames</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="kn">from</span> <span class="nn">datafusion</span> <span class="kn">import</span> <span class="n">SessionContext</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="n">ctx</span> <span class="o">=</span> <span class="n">SessionContext</span><span class="p">()</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="n">left</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">(</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">{</span>
+ <span class="o">...</span><span class="p">:</span> <span class="s2">"customer_id"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span>
+ <span class="o">...</span><span class="p">:</span> <span class="s2">"customer"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"Alice"</span><span class="p">,</span> <span class="s2">"Bob"</span><span class="p">,</span> <span class="s2">"Charlie"</span><span class="p">],</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">}</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">)</span>
+ <span class="o">...</span><span class="p">:</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">4</span><span class="p">]:</span> <span class="n">right</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pylist</span><span class="p">([</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">{</span><span class="s2">"id"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s2">"name"</span><span class="p">:</span> <span class="s2">"CityCabs"</span><span class="p">},</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">{</span><span class="s2">"id"</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span> <span class="s2">"name"</span><span class="p">:</span> <span class="s2">"MetroRide"</span><span class="p">},</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">{</span><span class="s2">"id"</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span> <span class="s2">"name"</span><span class="p">:</span> <span class="s2">"UrbanGo"</span><span class="p">},</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">])</span>
+ <span class="o">...</span><span class="p">:</span>
+</pre></div>
+</div>
+<section id="inner-join">
+<h2>Inner Join<a class="headerlink" href="#inner-join" title="Permalink to this heading">¶</a></h2>
... 4715 lines suppressed ...