You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/03/26 15:28:40 UTC
[GitHub] [arrow] lidavidm opened a new pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
lidavidm opened a new pull request #9810:
URL: https://github.com/apache/arrow/pull/9810
This is mostly based on the Python documentation. This also adds a new C++ example to accompany the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r610317754
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
Review comment:
Actually, looking at the below section, maybe lift the first bullet point so that the datasets library isn't defined as a library for datasets? This suggestion includes @westonpace 's next few comments and the following bullet points:
```
The Arrow Datasets library provides a unified interface for operations on tabular data from various sources, in various formats. The data may be from buffers, local filesystems, and cloud filesystems; and may be larger than memory or span multiple units (e.g. multiple files). Some supported data formats include parquet, feather, and IPC. Some supported operations include discovery, predicate pushdowns, and read parallelism.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r610336087
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
Review comment:
> Drop `in combination with expressions.`
Maybe just combine this sentence with the following one:
```
In addition to selecting columns, Project can take a vector of
expressions (for more complex projections) and a vector of
column names. This usage evaluates each expression to a
column value, associated with the co-indexed column name.
```
(newlines added for comment formatting)
I'm not sure the "co-indexing" is clear (or necessary) so feel free to remove/improve that part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#issuecomment-819636374
> I'm going to merge, but @drin thanks for all your comments! If you have further feedback or run into other things, please feel free to follow up here or create a JIRA issue (https://issues.apache.org/jira/, feel free to tag @lidavidm there) and we can iterate further.
sounds good, thanks and great job!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r610327864
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
Review comment:
maybe end with `...from the file.` since I think the filesystem is a slightly higher abstraction than the file? At least, as far as the implementations here. Generally speaking, the underlying file system defines whether parts of a file can be read into memory, but I don't think that's what's being referred to here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r612851171
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,389 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory, and multi-file datasets. This includes:
+
+* A unified interface that supports different sources and file formats (currently,
+ Parquet, Feather / Arrow IPC, and CSV files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handling partitioned datasets with
+ various partitioning schemes, basic schema normalization, ..)
Review comment:
`..` -> `...`
##########
File path: cpp/src/arrow/dataset/scanner.h
##########
@@ -43,33 +43,39 @@ using RecordBatchGenerator = std::function<Future<std::shared_ptr<RecordBatch>>(
namespace dataset {
+/// \defgroup dataset-scanning Scanning API
+///
+/// @{
+
constexpr int64_t kDefaultBatchSize = 1 << 20;
constexpr int32_t kDefaultBatchReadahead = 32;
constexpr int32_t kDefaultFragmentReadahead = 8;
+/// Scan-specific options, which can be changed between scans of the same dataset.
struct ARROW_DS_EXPORT ScanOptions {
- // Filter and projection
+ /// A row filter (which can be pushed down to partitioning/reading if supported).
Review comment:
`can` -> `will` or get rid of `if supported`.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,389 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory, and multi-file datasets. This includes:
+
+* A unified interface that supports different sources and file formats (currently,
+ Parquet, Feather / Arrow IPC, and CSV files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handling partitioned datasets with
+ various partitioning schemes, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting and deriving columns), and optionally parallel reading.
+
+The goal is to expand support to other file formats and data sources
+(e.g. database connections) in the future.
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given a base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-165
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+We're also passing the filesystem to use and the file format to use for reading.
+This lets us choose between (for example) reading local files or files in Amazon
+S3, or between Parquet and CSV.
+
+In addition to searching a base directory, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` does not begin reading the data
+itself. If needed, it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ // Print out the files crawled (only for FileSystemDataset)
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk but the Dataset API
+provides a consistent interface across multiple file formats and filesystems.
+(See :ref:`cpp-dataset-cloud-storage` for more information on the latter.)
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as Feather files instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+…then we can read the Feather file by passing a :class:`arrow::dataset::IpcFileFormat`:
Review comment:
`a` -> `an`
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,389 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory, and multi-file datasets. This includes:
+
+* A unified interface that supports different sources and file formats (currently,
+ Parquet, Feather / Arrow IPC, and CSV files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handling partitioned datasets with
+ various partitioning schemes, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting and deriving columns), and optionally parallel reading.
+
+The goal is to expand support to other file formats and data sources
+(e.g. database connections) in the future.
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given a base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-165
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+We're also passing the filesystem to use and the file format to use for reading.
+This lets us choose between (for example) reading local files or files in Amazon
+S3, or between Parquet and CSV.
+
+In addition to searching a base directory, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` does not begin reading the data
+itself. If needed, it only crawls the directory to find all the files
Review comment:
`If needed, It only crawls...all the files` -> `It only crawls...all the files (if needed).`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r604173489
##########
File path: docs/source/cpp/api/dataset.rst
##########
@@ -0,0 +1,165 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+=======
+Dataset
+=======
+
+.. sidebar:: Contents
+
+ .. contents:: :local:
+
+Interface
+=========
+
+.. doxygenclass:: arrow::dataset::Fragment
Review comment:
Note that you may use `doxygengroup`s to reduce the burden of listing and maintaining individual APIs here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r610324994
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
Review comment:
I also think that since the member variable is `reader_options`, I'd prefer to see:
`FileFormat objects have options that control read behavior`
...but only ParquetFileFormat has `reader_options`? Is this called something different in each `FileFormat` class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#issuecomment-819134244
> I also think that since the member variable is reader_options, I'd prefer to see:
>
> FileFormat objects have options that control read behavior
>
> ...but only ParquetFileFormat has reader_options? Is this called something different in each FileFormat class?
>
@drin
I'm not sure if it's called out here but the preferred way of setting options would probably be to supply them to the `ScannerBuilder` using the `FragmentScanOptions` method. There are `IpcFragmentScanOptions`, `CsvFragmentScanOptions`, and `ParquetFragmentScanOptions`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#issuecomment-810329678
I'll let @bkietz comment on the correctness and idiomaticity of the example. I find the code length a bit intimidating, but I guess C++ doesn't allow for a very concise example :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#issuecomment-810330783
The example is a bit long because it does the same thing several times with minor variations, with most things duplicated each time so snippets can be used as standalone examples in the docs. However the example could certainly be better structured/documented so it's readable on its own as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#issuecomment-808311064
Rendered (though Github is acting up):
- https://lidavidm.github.io/arrow-docs-next/cpp/dataset.html#tabular-datasets
- https://lidavidm.github.io/arrow-docs-next/cpp/api/dataset.html#dataset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#issuecomment-818914539
@westonpace (and/or @drin!) would you like to give a final read-through? I think this may cause minor conflicts with the rest of our PRs due to the new doc comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r610307942
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
Review comment:
I don't think this is necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#issuecomment-819518702
I'm going to merge, but @drin thanks for all your comments! If you have further feedback or run into other things, please feel free to follow up here or create a JIRA issue (https://issues.apache.org/jira/, feel free to tag @lidavidm there) and we can iterate further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r610336087
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
Review comment:
> Drop `in combination with expressions.`
Maybe combine this sentence with the following one with some clarification:
```
In addition to selecting columns, Project can take a vector of
expressions (for more complex projections) and a vector of
column names. This usage evaluates each expression to a
column value, associated with the co-indexed column name.
```
(newlines added for comment formatting)
I'm not sure the "co-indexing" is clear (or necessary) so feel free to remove/improve that part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r610322157
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
Review comment:
I think it is actually correct here (https://www.grammarly.com/blog/comma-before-but/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#issuecomment-810608422
@drin thanks for the feedback. I can add a snippet about that as well. If there are other things that were confusing or difficult to figure out, I'd like to tackle them too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r610318529
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
Review comment:
Change `supported` -> `implemented`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#issuecomment-808311116
https://issues.apache.org/jira/browse/ARROW-11677
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r610317754
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
Review comment:
Actually, looking at the below section, maybe lift the first bullet point so that the datasets library isn't defined as a library for datasets? This suggestion includes @westonpace 's next few comments and the following bullet points:
```
The Arrow Datasets library provides a unified interface for
operations on tabular data from various sources, in various
formats. The data may be from buffers, local filesystems,
and cloud filesystems; and may be larger than memory or span
multiple units (e.g. multiple files). Some supported data formats
include parquet, feather, and IPC. Some supported operations
include discovery, predicate pushdowns, and read parallelism.
```
(newlines added based on formatting for this comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r610326551
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
Review comment:
I mostly agree with this, but my assumption is that `Project` selects which columns are produced? I don't know the implementation details, but I hope that something else actually determines which columns are read from the source (since a filter may be applied to a column not in the projection)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#issuecomment-810615985
> @drin thanks for the feedback. I can add a snippet about that as well. If there are other things that were confusing or difficult to figure out, I'd like to tackle them too.
Sounds great! I appreciate your work. I have only recently gathered enough experience to be able to express what is/was difficult and why, so I'll try to keep adding my thoughts here (and directly in JIRA).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r610177257
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
Review comment:
Maybe get rid of `fine-grained managing of tasks`? You could change `parallel reading` to `potentially parallel reading`.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
Review comment:
`and the file format to read` -> `and the file format to use for reading`
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
Review comment:
It seems a bit odd that here we iterate through and print `dataset->files()` when in an earlier example we were iterating `dataset->GetFragments()`. Although I suppose I understand it is because we are working with the more specialized `FileSystemDataset` and not a plain old `Dataset`.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
Review comment:
Maybe change:
> A unified interface for different sources: supporting different sources and
to...
`A unified interface for supporting different sources and`
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
+
+In this case, we pass a vector of expressions used to construct column values,
+and a vector of names for the columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 193-223
+ :emphasize-lines: 18-28
+ :linenos:
+ :lineno-match:
+
+This also determines the column selection; only the given columns will be
+present in the resulting table. If you want to include a derived column in
+*addition* to the existing columns, you can build up the expressions from the
+dataset schema:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 225-254
+ :emphasize-lines: 17-27
+ :linenos:
+ :lineno-match:
+
+Reading and writing partitioned data
+------------------------------------
+
+So far, we've been working with datasets consisting of flat directories with
+files. However, a dataset can exploit nested directory structures defining a
+partitioned dataset, where sub-directory names hold information about which
+subset of the data is stored in that directory.
+
+For example, a dataset partitioned by year and month may look like on disk:
Review comment:
`may look like on disk` -> `may have the following layout`.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
Review comment:
```
(e.g. database connections)
```
to
```
(e.g. database connections)"
```
[extra space]
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
Review comment:
Drop `in combination with expressions.`
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
Review comment:
`Here, we're` -> `We're`
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
Review comment:
"given the base directory path" -> "given a base directory path"
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
Review comment:
Take all these grammar / style nits with a grain of salt.
> tabular, potentially larger than memory and multi-file datasets:
to...
`tabular, potentially larger than memory and multi-file datasets. For example:`
or...
`tabular, potentially larger than memory and multi-file datasets. This includes:`
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
Review comment:
Maybe stop at line 165 in this snippet?
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
+
+In this case, we pass a vector of expressions used to construct column values,
+and a vector of names for the columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 193-223
+ :emphasize-lines: 18-28
+ :linenos:
+ :lineno-match:
+
+This also determines the column selection; only the given columns will be
+present in the resulting table. If you want to include a derived column in
+*addition* to the existing columns, you can build up the expressions from the
+dataset schema:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 225-254
+ :emphasize-lines: 17-27
+ :linenos:
+ :lineno-match:
+
+Reading and writing partitioned data
+------------------------------------
+
+So far, we've been working with datasets consisting of flat directories with
+files. However, a dataset can exploit nested directory structures defining a
+partitioned dataset, where sub-directory names hold information about which
+subset of the data is stored in that directory.
+
+For example, a dataset partitioned by year and month may look like on disk:
+
+.. code-block:: text
+
+ dataset_name/
+ year=2007/
+ month=01/
+ data0.parquet
+ data1.parquet
+ ...
+ month=02/
+ data0.parquet
+ data1.parquet
+ ...
+ month=03/
+ ...
+ year=2008/
+ month=01/
+ ...
+ ...
+
+The above partitioning scheme is using "/key=value/" directory names, as found
+in Apache Hive.
+
+Let's create a small partitioned dataset. For this, we'll use Dataset's writing
+functionality.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 106-149
+ :emphasize-lines: 25-42
+ :linenos:
+ :lineno-match:
+
+The above created a directory with two subdirectories ("part=a" and "part=b"),
+and the Parquet files written in those directories no longer include the "part"
+column.
+
+Reading this dataset, we now specify that the dataset uses a Hive-like
+partitioning scheme:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 256-279
+ :emphasize-lines: 7,9-11
+ :linenos:
+ :lineno-match:
+
+Although the partition fields are not included in the actual Parquet files,
+they will be added back to the resulting table when scanning this dataset:
+
+.. code-block:: text
+
+ $ ./debug/dataset_documentation_example file:///tmp parquet_hive partitioned
+ Found fragment: /tmp/parquet_dataset/part=a/part0.parquet
+ Partition expression: (part == "a")
+ Found fragment: /tmp/parquet_dataset/part=b/part1.parquet
+ Partition expression: (part == "b")
+ Read 20 rows
+ a: int64
+ -- field metadata --
+ PARQUET:field_id: '1'
+ b: double
+ -- field metadata --
+ PARQUET:field_id: '2'
+ c: int64
+ -- field metadata --
+ PARQUET:field_id: '3'
+ part: string
+ ----
+ # snip...
+
+We can now filter on the partition keys, which avoids loading files
+altogether if they do not match the predicate:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 281-301
+ :emphasize-lines: 15-18
+ :linenos:
+ :lineno-match:
+
+Different partitioning schemes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above example uses a hive-like directory scheme, such as "/year=2009/month=11/day=15".
+We specified this by passing the partition *factory*. In this case, the types of
Review comment:
`the partition *factory*` -> `the hive partitioning factory`
##########
File path: docs/source/python/dataset.rst
##########
@@ -38,9 +38,9 @@ tabular, potentially larger than memory and multi-file datasets:
* Optimized reading with predicate pushdown (filtering rows), projection
(selecting columns), parallel reading or fine-grained managing of tasks.
-Currently, only Parquet and Feather / Arrow IPC files are supported. The goal
-is to expand this in the future to other file formats and data sources (e.g.
-database connections).
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
Review comment:
```
(e.g. database
```
```
(e.g. database
```
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
Review comment:
`a base directory path` -> `searching a base directory`
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
Review comment:
`choose between things like reading` -> `choose between reading`
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
+
+In this case, we pass a vector of expressions used to construct column values,
+and a vector of names for the columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 193-223
+ :emphasize-lines: 18-28
+ :linenos:
+ :lineno-match:
+
+This also determines the column selection; only the given columns will be
+present in the resulting table. If you want to include a derived column in
+*addition* to the existing columns, you can build up the expressions from the
+dataset schema:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 225-254
+ :emphasize-lines: 17-27
+ :linenos:
+ :lineno-match:
+
+Reading and writing partitioned data
+------------------------------------
+
+So far, we've been working with datasets consisting of flat directories with
+files. However, a dataset can exploit nested directory structures defining a
+partitioned dataset, where sub-directory names hold information about which
+subset of the data is stored in that directory.
+
+For example, a dataset partitioned by year and month may look like on disk:
+
+.. code-block:: text
+
+ dataset_name/
+ year=2007/
+ month=01/
+ data0.parquet
+ data1.parquet
+ ...
+ month=02/
+ data0.parquet
+ data1.parquet
+ ...
+ month=03/
+ ...
+ year=2008/
+ month=01/
+ ...
+ ...
+
+The above partitioning scheme is using "/key=value/" directory names, as found
+in Apache Hive.
+
+Let's create a small partitioned dataset. For this, we'll use Dataset's writing
+functionality.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 106-149
+ :emphasize-lines: 25-42
+ :linenos:
+ :lineno-match:
+
+The above created a directory with two subdirectories ("part=a" and "part=b"),
+and the Parquet files written in those directories no longer include the "part"
+column.
+
+Reading this dataset, we now specify that the dataset uses a Hive-like
+partitioning scheme:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 256-279
+ :emphasize-lines: 7,9-11
+ :linenos:
+ :lineno-match:
+
+Although the partition fields are not included in the actual Parquet files,
+they will be added back to the resulting table when scanning this dataset:
+
+.. code-block:: text
+
+ $ ./debug/dataset_documentation_example file:///tmp parquet_hive partitioned
+ Found fragment: /tmp/parquet_dataset/part=a/part0.parquet
+ Partition expression: (part == "a")
+ Found fragment: /tmp/parquet_dataset/part=b/part1.parquet
+ Partition expression: (part == "b")
+ Read 20 rows
+ a: int64
+ -- field metadata --
+ PARQUET:field_id: '1'
+ b: double
+ -- field metadata --
+ PARQUET:field_id: '2'
+ c: int64
+ -- field metadata --
+ PARQUET:field_id: '3'
+ part: string
+ ----
+ # snip...
+
+We can now filter on the partition keys, which avoids loading files
+altogether if they do not match the predicate:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 281-301
+ :emphasize-lines: 15-18
+ :linenos:
+ :lineno-match:
+
+Different partitioning schemes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above example uses a hive-like directory scheme, such as "/year=2009/month=11/day=15".
+We specified this by passing the partition *factory*. In this case, the types of
+the partition keys are inferred from the file paths.
+
+It is also possible directly construct the partitioning and explicitly define
+the schema of the partition keys. For example:
+
+.. code-block:: cpp
+
+ auto part = std::make_shared<ds::HivePartitioning>(arrow::schema({
+ arrow::field("year", arrow::int16()),
+ arrow::field("month", arrow::int8()),
+ arrow::field("day", arrow::int32())
+ }));
+
+"Directory partitioning" is also supported, where the segments in the file path
+represent the values of the partition keys without including the name (the
+field name are implicit in the segment's index). For example, given field names
+"year", "month", and "day", one path might be "/2019/11/15".
+
+Since the names are not included in the file paths, these must be specified
+when constructing a directory partitioning:
+
+.. code-block:: cpp
+
+ auto part = ds::DirectoryPartitioning::MakeFactory({"year", "month", "day"});
+
+Directory partitioning also supports providing a full schema rather than inferring
+types from file paths.
+
+Reading from other data sources
+-------------------------------
+
+Reading in-memory data
+~~~~~~~~~~~~~~~~~~~~~~
+
+If you already have data in memory that you'd like to use with the Datasets
+interface (e.g. to filter/project data, or to write it out to a filesystem),
+you can wrap it in a :class:`arrow::dataset::InMemoryDataset`:
+
+.. code-block:: cpp
+
+ auto table = arrow::Table::FromRecordBatches(...);
+ auto dataset = std::make_shared<arrow::dataset::InMemoryDataset>(std::move(table));
+ // Scan the dataset, filter, it, etc.
+ auto scanner_builder = dataset->NewScan();
+
+In the example, we used the InMemoryDataset to write our example data to local
+disk to be used in the rest of the example:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 106-149
+ :emphasize-lines: 24-28
+ :linenos:
+ :lineno-match:
+
+Reading from cloud storage
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In addition to local files, Arrow Datasets also supports reading from cloud
Review comment:
`supports` -> `support`
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
Review comment:
"loads nothing into memory" seems a little misleading. Wouldn't it be more accurate to say "does not begin reading the data"? Also what is `in this way` referring to? Isn't this statement true whether you are creating a dataset from a directory or from a list of paths? In fact, you might want to explain why it is crawling the directory to find all the files given we just presumably listed the paths manually.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
Review comment:
I don't believe the `,` belongs here.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
Review comment:
Also, you use a `::` here but elsewhere you use a `:`. But this may just be an `rst` thing I don't understand.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
Review comment:
`multiple file formats and sources` -> `multiple file formats and filesystems`? Not sure on this one. Either way, you don't really expand on this point. There are no examples of using a "different source" whatever that might be.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
Review comment:
`as a Feather file` -> `as Feather files`
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
Review comment:
You're inconsistent about whether you start a continuing sentence with `...` or not. I'm not sure which is more correct but you do it differently on line 93.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
Review comment:
`we've read the entire dataset` -> `we've been reading the entire dataset`
Instead of `inefficient` can you be more specific? Maybe `use too much memory or waste time reading data we don't need.`?
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
Review comment:
Technically `Project` does more than just that but I think it's fine to omit that complexity at the moment. Maybe change to `In this snippet we use Project to select which columns to read`.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
Review comment:
`how they are read` -> `how files are read`.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
Review comment:
`Feather file using the same functions, but passing a` -> `Feather file by passing a`.
Also, I'm pretty sure this is the wrong snippet to be referencing here.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
Review comment:
Maybe add a sentence...
` Some formats (such as Parquet) can use this filter to reduce the amount of I/O needed.`
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
Review comment:
"pushdown" is maybe not a term known by the audience here. Also, I don't think column selection is technically "pushdown". Maybe `Some formats (such as Parquet) can reduce I/O costs by reading specific columns from the filesystem.`
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
+
+In this case, we pass a vector of expressions used to construct column values,
Review comment:
Second comma on this line not needed.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
+
+In this case, we pass a vector of expressions used to construct column values,
+and a vector of names for the columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 193-223
+ :emphasize-lines: 18-28
+ :linenos:
+ :lineno-match:
+
+This also determines the column selection; only the given columns will be
+present in the resulting table. If you want to include a derived column in
+*addition* to the existing columns, you can build up the expressions from the
+dataset schema:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 225-254
+ :emphasize-lines: 17-27
+ :linenos:
+ :lineno-match:
+
+Reading and writing partitioned data
+------------------------------------
+
+So far, we've been working with datasets consisting of flat directories with
+files. However, a dataset can exploit nested directory structures defining a
+partitioned dataset, where sub-directory names hold information about which
+subset of the data is stored in that directory.
+
+For example, a dataset partitioned by year and month may look like on disk:
+
+.. code-block:: text
+
+ dataset_name/
+ year=2007/
+ month=01/
+ data0.parquet
+ data1.parquet
+ ...
+ month=02/
+ data0.parquet
+ data1.parquet
+ ...
+ month=03/
+ ...
+ year=2008/
+ month=01/
+ ...
+ ...
+
+The above partitioning scheme is using "/key=value/" directory names, as found
+in Apache Hive.
+
+Let's create a small partitioned dataset. For this, we'll use Dataset's writing
+functionality.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 106-149
+ :emphasize-lines: 25-42
+ :linenos:
+ :lineno-match:
+
+The above created a directory with two subdirectories ("part=a" and "part=b"),
+and the Parquet files written in those directories no longer include the "part"
+column.
+
+Reading this dataset, we now specify that the dataset uses a Hive-like
Review comment:
`uses` -> `should use`
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
+
+In this case, we pass a vector of expressions used to construct column values,
+and a vector of names for the columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 193-223
+ :emphasize-lines: 18-28
+ :linenos:
+ :lineno-match:
+
+This also determines the column selection; only the given columns will be
+present in the resulting table. If you want to include a derived column in
+*addition* to the existing columns, you can build up the expressions from the
+dataset schema:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 225-254
+ :emphasize-lines: 17-27
+ :linenos:
+ :lineno-match:
+
+Reading and writing partitioned data
+------------------------------------
+
+So far, we've been working with datasets consisting of flat directories with
+files. However, a dataset can exploit nested directory structures defining a
Review comment:
The whole sentence starting with `However` is a little awkward to me. Maybe...
```
However, a "partitioned" dataset relies on nested directory structures where sub-directory names hold describe the data stored in that directory
```
...but I'm not really sold on that either.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
+
+In this case, we pass a vector of expressions used to construct column values,
+and a vector of names for the columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 193-223
+ :emphasize-lines: 18-28
+ :linenos:
+ :lineno-match:
+
+This also determines the column selection; only the given columns will be
+present in the resulting table. If you want to include a derived column in
+*addition* to the existing columns, you can build up the expressions from the
+dataset schema:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 225-254
+ :emphasize-lines: 17-27
+ :linenos:
+ :lineno-match:
+
+Reading and writing partitioned data
+------------------------------------
+
+So far, we've been working with datasets consisting of flat directories with
+files. However, a dataset can exploit nested directory structures defining a
+partitioned dataset, where sub-directory names hold information about which
+subset of the data is stored in that directory.
+
+For example, a dataset partitioned by year and month may look like on disk:
+
+.. code-block:: text
+
+ dataset_name/
+ year=2007/
+ month=01/
+ data0.parquet
+ data1.parquet
+ ...
+ month=02/
+ data0.parquet
+ data1.parquet
+ ...
+ month=03/
+ ...
+ year=2008/
+ month=01/
+ ...
+ ...
+
+The above partitioning scheme is using "/key=value/" directory names, as found
+in Apache Hive.
+
+Let's create a small partitioned dataset. For this, we'll use Dataset's writing
+functionality.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 106-149
+ :emphasize-lines: 25-42
+ :linenos:
+ :lineno-match:
+
+The above created a directory with two subdirectories ("part=a" and "part=b"),
+and the Parquet files written in those directories no longer include the "part"
+column.
+
+Reading this dataset, we now specify that the dataset uses a Hive-like
+partitioning scheme:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 256-279
+ :emphasize-lines: 7,9-11
+ :linenos:
+ :lineno-match:
+
+Although the partition fields are not included in the actual Parquet files,
+they will be added back to the resulting table when scanning this dataset:
+
+.. code-block:: text
+
+ $ ./debug/dataset_documentation_example file:///tmp parquet_hive partitioned
+ Found fragment: /tmp/parquet_dataset/part=a/part0.parquet
+ Partition expression: (part == "a")
+ Found fragment: /tmp/parquet_dataset/part=b/part1.parquet
+ Partition expression: (part == "b")
+ Read 20 rows
+ a: int64
+ -- field metadata --
+ PARQUET:field_id: '1'
+ b: double
+ -- field metadata --
+ PARQUET:field_id: '2'
+ c: int64
+ -- field metadata --
+ PARQUET:field_id: '3'
+ part: string
+ ----
+ # snip...
+
+We can now filter on the partition keys, which avoids loading files
+altogether if they do not match the predicate:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 281-301
+ :emphasize-lines: 15-18
+ :linenos:
+ :lineno-match:
+
+Different partitioning schemes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above example uses a hive-like directory scheme, such as "/year=2009/month=11/day=15".
+We specified this by passing the partition *factory*. In this case, the types of
+the partition keys are inferred from the file paths.
+
+It is also possible directly construct the partitioning and explicitly define
+the schema of the partition keys. For example:
+
+.. code-block:: cpp
+
+ auto part = std::make_shared<ds::HivePartitioning>(arrow::schema({
+ arrow::field("year", arrow::int16()),
+ arrow::field("month", arrow::int8()),
+ arrow::field("day", arrow::int32())
+ }));
+
+"Directory partitioning" is also supported, where the segments in the file path
Review comment:
"Directory partitioning" is kind of a confusing name when "partitioning" means "using directories". Maybe to help with that we could rephrase this to...
```
Arrow supports another form of partitioning, "Directory partitioning", where the segments...
```
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
+
+In this case, we pass a vector of expressions used to construct column values,
+and a vector of names for the columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 193-223
+ :emphasize-lines: 18-28
+ :linenos:
+ :lineno-match:
+
+This also determines the column selection; only the given columns will be
+present in the resulting table. If you want to include a derived column in
+*addition* to the existing columns, you can build up the expressions from the
+dataset schema:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 225-254
+ :emphasize-lines: 17-27
+ :linenos:
+ :lineno-match:
+
+Reading and writing partitioned data
+------------------------------------
+
+So far, we've been working with datasets consisting of flat directories with
+files. However, a dataset can exploit nested directory structures defining a
+partitioned dataset, where sub-directory names hold information about which
+subset of the data is stored in that directory.
+
+For example, a dataset partitioned by year and month may look like on disk:
+
+.. code-block:: text
+
+ dataset_name/
+ year=2007/
+ month=01/
+ data0.parquet
+ data1.parquet
+ ...
+ month=02/
+ data0.parquet
+ data1.parquet
+ ...
+ month=03/
+ ...
+ year=2008/
+ month=01/
+ ...
+ ...
+
+The above partitioning scheme is using "/key=value/" directory names, as found
+in Apache Hive.
+
+Let's create a small partitioned dataset. For this, we'll use Dataset's writing
Review comment:
It feels a bit weird to have "Dataset" with a capital D and no backticks or link. Could maybe phrase it as "Arrow's dataset writing functionality."
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
+
+In this case, we pass a vector of expressions used to construct column values,
+and a vector of names for the columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 193-223
+ :emphasize-lines: 18-28
+ :linenos:
+ :lineno-match:
+
+This also determines the column selection; only the given columns will be
+present in the resulting table. If you want to include a derived column in
+*addition* to the existing columns, you can build up the expressions from the
+dataset schema:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 225-254
+ :emphasize-lines: 17-27
+ :linenos:
+ :lineno-match:
+
+Reading and writing partitioned data
+------------------------------------
+
+So far, we've been working with datasets consisting of flat directories with
+files. However, a dataset can exploit nested directory structures defining a
+partitioned dataset, where sub-directory names hold information about which
+subset of the data is stored in that directory.
+
+For example, a dataset partitioned by year and month may look like on disk:
+
+.. code-block:: text
+
+ dataset_name/
+ year=2007/
+ month=01/
+ data0.parquet
+ data1.parquet
+ ...
+ month=02/
+ data0.parquet
+ data1.parquet
+ ...
+ month=03/
+ ...
+ year=2008/
+ month=01/
+ ...
+ ...
+
+The above partitioning scheme is using "/key=value/" directory names, as found
+in Apache Hive.
+
+Let's create a small partitioned dataset. For this, we'll use Dataset's writing
+functionality.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 106-149
+ :emphasize-lines: 25-42
+ :linenos:
+ :lineno-match:
+
+The above created a directory with two subdirectories ("part=a" and "part=b"),
+and the Parquet files written in those directories no longer include the "part"
+column.
+
+Reading this dataset, we now specify that the dataset uses a Hive-like
+partitioning scheme:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 256-279
+ :emphasize-lines: 7,9-11
+ :linenos:
+ :lineno-match:
+
+Although the partition fields are not included in the actual Parquet files,
+they will be added back to the resulting table when scanning this dataset:
+
+.. code-block:: text
+
+ $ ./debug/dataset_documentation_example file:///tmp parquet_hive partitioned
+ Found fragment: /tmp/parquet_dataset/part=a/part0.parquet
+ Partition expression: (part == "a")
+ Found fragment: /tmp/parquet_dataset/part=b/part1.parquet
+ Partition expression: (part == "b")
+ Read 20 rows
+ a: int64
+ -- field metadata --
+ PARQUET:field_id: '1'
+ b: double
+ -- field metadata --
+ PARQUET:field_id: '2'
+ c: int64
+ -- field metadata --
+ PARQUET:field_id: '3'
+ part: string
+ ----
+ # snip...
+
+We can now filter on the partition keys, which avoids loading files
+altogether if they do not match the predicate:
Review comment:
`predicate` -> `filter`.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
+
+In this case, we pass a vector of expressions used to construct column values,
+and a vector of names for the columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 193-223
+ :emphasize-lines: 18-28
+ :linenos:
+ :lineno-match:
+
+This also determines the column selection; only the given columns will be
+present in the resulting table. If you want to include a derived column in
+*addition* to the existing columns, you can build up the expressions from the
+dataset schema:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 225-254
+ :emphasize-lines: 17-27
+ :linenos:
+ :lineno-match:
+
+Reading and writing partitioned data
+------------------------------------
+
+So far, we've been working with datasets consisting of flat directories with
+files. However, a dataset can exploit nested directory structures defining a
+partitioned dataset, where sub-directory names hold information about which
+subset of the data is stored in that directory.
+
+For example, a dataset partitioned by year and month may look like on disk:
+
+.. code-block:: text
+
+ dataset_name/
+ year=2007/
+ month=01/
+ data0.parquet
+ data1.parquet
+ ...
+ month=02/
+ data0.parquet
+ data1.parquet
+ ...
+ month=03/
+ ...
+ year=2008/
+ month=01/
+ ...
+ ...
+
+The above partitioning scheme is using "/key=value/" directory names, as found
+in Apache Hive.
+
+Let's create a small partitioned dataset. For this, we'll use Dataset's writing
+functionality.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 106-149
+ :emphasize-lines: 25-42
+ :linenos:
+ :lineno-match:
+
+The above created a directory with two subdirectories ("part=a" and "part=b"),
+and the Parquet files written in those directories no longer include the "part"
+column.
+
+Reading this dataset, we now specify that the dataset uses a Hive-like
+partitioning scheme:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 256-279
+ :emphasize-lines: 7,9-11
+ :linenos:
+ :lineno-match:
+
+Although the partition fields are not included in the actual Parquet files,
+they will be added back to the resulting table when scanning this dataset:
+
+.. code-block:: text
+
+ $ ./debug/dataset_documentation_example file:///tmp parquet_hive partitioned
+ Found fragment: /tmp/parquet_dataset/part=a/part0.parquet
+ Partition expression: (part == "a")
+ Found fragment: /tmp/parquet_dataset/part=b/part1.parquet
+ Partition expression: (part == "b")
+ Read 20 rows
+ a: int64
+ -- field metadata --
+ PARQUET:field_id: '1'
+ b: double
+ -- field metadata --
+ PARQUET:field_id: '2'
+ c: int64
+ -- field metadata --
+ PARQUET:field_id: '3'
+ part: string
+ ----
+ # snip...
+
+We can now filter on the partition keys, which avoids loading files
+altogether if they do not match the predicate:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 281-301
+ :emphasize-lines: 15-18
+ :linenos:
+ :lineno-match:
+
+Different partitioning schemes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above example uses a hive-like directory scheme, such as "/year=2009/month=11/day=15".
+We specified this by passing the partition *factory*. In this case, the types of
+the partition keys are inferred from the file paths.
+
+It is also possible directly construct the partitioning and explicitly define
+the schema of the partition keys. For example:
+
+.. code-block:: cpp
+
+ auto part = std::make_shared<ds::HivePartitioning>(arrow::schema({
+ arrow::field("year", arrow::int16()),
+ arrow::field("month", arrow::int8()),
+ arrow::field("day", arrow::int32())
+ }));
+
+"Directory partitioning" is also supported, where the segments in the file path
+represent the values of the partition keys without including the name (the
+field name are implicit in the segment's index). For example, given field names
+"year", "month", and "day", one path might be "/2019/11/15".
+
+Since the names are not included in the file paths, these must be specified
+when constructing a directory partitioning:
+
+.. code-block:: cpp
+
+ auto part = ds::DirectoryPartitioning::MakeFactory({"year", "month", "day"});
+
+Directory partitioning also supports providing a full schema rather than inferring
+types from file paths.
+
+Reading from other data sources
+-------------------------------
+
+Reading in-memory data
+~~~~~~~~~~~~~~~~~~~~~~
+
+If you already have data in memory that you'd like to use with the Datasets
+interface (e.g. to filter/project data, or to write it out to a filesystem),
+you can wrap it in a :class:`arrow::dataset::InMemoryDataset`:
Review comment:
`a` -> `an`
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
+
+In this case, we pass a vector of expressions used to construct column values,
+and a vector of names for the columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 193-223
+ :emphasize-lines: 18-28
+ :linenos:
+ :lineno-match:
+
+This also determines the column selection; only the given columns will be
+present in the resulting table. If you want to include a derived column in
+*addition* to the existing columns, you can build up the expressions from the
+dataset schema:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 225-254
+ :emphasize-lines: 17-27
+ :linenos:
+ :lineno-match:
+
+Reading and writing partitioned data
+------------------------------------
+
+So far, we've been working with datasets consisting of flat directories with
+files. However, a dataset can exploit nested directory structures defining a
+partitioned dataset, where sub-directory names hold information about which
+subset of the data is stored in that directory.
+
+For example, a dataset partitioned by year and month may look like on disk:
+
+.. code-block:: text
+
+ dataset_name/
+ year=2007/
+ month=01/
+ data0.parquet
+ data1.parquet
+ ...
+ month=02/
+ data0.parquet
+ data1.parquet
+ ...
+ month=03/
+ ...
+ year=2008/
+ month=01/
+ ...
+ ...
+
+The above partitioning scheme is using "/key=value/" directory names, as found
+in Apache Hive.
+
+Let's create a small partitioned dataset. For this, we'll use Dataset's writing
+functionality.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 106-149
+ :emphasize-lines: 25-42
+ :linenos:
+ :lineno-match:
+
+The above created a directory with two subdirectories ("part=a" and "part=b"),
+and the Parquet files written in those directories no longer include the "part"
+column.
+
+Reading this dataset, we now specify that the dataset uses a Hive-like
+partitioning scheme:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 256-279
+ :emphasize-lines: 7,9-11
+ :linenos:
+ :lineno-match:
+
+Although the partition fields are not included in the actual Parquet files,
+they will be added back to the resulting table when scanning this dataset:
+
+.. code-block:: text
+
+ $ ./debug/dataset_documentation_example file:///tmp parquet_hive partitioned
+ Found fragment: /tmp/parquet_dataset/part=a/part0.parquet
+ Partition expression: (part == "a")
+ Found fragment: /tmp/parquet_dataset/part=b/part1.parquet
+ Partition expression: (part == "b")
+ Read 20 rows
+ a: int64
+ -- field metadata --
+ PARQUET:field_id: '1'
+ b: double
+ -- field metadata --
+ PARQUET:field_id: '2'
+ c: int64
+ -- field metadata --
+ PARQUET:field_id: '3'
+ part: string
+ ----
+ # snip...
+
+We can now filter on the partition keys, which avoids loading files
+altogether if they do not match the predicate:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 281-301
+ :emphasize-lines: 15-18
+ :linenos:
+ :lineno-match:
+
+Different partitioning schemes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above example uses a hive-like directory scheme, such as "/year=2009/month=11/day=15".
+We specified this by passing the partition *factory*. In this case, the types of
+the partition keys are inferred from the file paths.
+
+It is also possible directly construct the partitioning and explicitly define
+the schema of the partition keys. For example:
+
+.. code-block:: cpp
+
+ auto part = std::make_shared<ds::HivePartitioning>(arrow::schema({
+ arrow::field("year", arrow::int16()),
+ arrow::field("month", arrow::int8()),
+ arrow::field("day", arrow::int32())
+ }));
+
+"Directory partitioning" is also supported, where the segments in the file path
+represent the values of the partition keys without including the name (the
+field name are implicit in the segment's index). For example, given field names
+"year", "month", and "day", one path might be "/2019/11/15".
+
+Since the names are not included in the file paths, these must be specified
+when constructing a directory partitioning:
+
+.. code-block:: cpp
+
+ auto part = ds::DirectoryPartitioning::MakeFactory({"year", "month", "day"});
+
+Directory partitioning also supports providing a full schema rather than inferring
+types from file paths.
+
+Reading from other data sources
+-------------------------------
+
+Reading in-memory data
+~~~~~~~~~~~~~~~~~~~~~~
+
+If you already have data in memory that you'd like to use with the Datasets
+interface (e.g. to filter/project data, or to write it out to a filesystem),
Review comment:
Not sure but `interface` -> `API`?
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
+
+In this case, we pass a vector of expressions used to construct column values,
+and a vector of names for the columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 193-223
+ :emphasize-lines: 18-28
+ :linenos:
+ :lineno-match:
+
+This also determines the column selection; only the given columns will be
+present in the resulting table. If you want to include a derived column in
+*addition* to the existing columns, you can build up the expressions from the
+dataset schema:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 225-254
+ :emphasize-lines: 17-27
+ :linenos:
+ :lineno-match:
+
+Reading and writing partitioned data
+------------------------------------
+
+So far, we've been working with datasets consisting of flat directories with
+files. However, a dataset can exploit nested directory structures defining a
+partitioned dataset, where sub-directory names hold information about which
+subset of the data is stored in that directory.
+
+For example, a dataset partitioned by year and month may look like on disk:
+
+.. code-block:: text
+
+ dataset_name/
+ year=2007/
+ month=01/
+ data0.parquet
+ data1.parquet
+ ...
+ month=02/
+ data0.parquet
+ data1.parquet
+ ...
+ month=03/
+ ...
+ year=2008/
+ month=01/
+ ...
+ ...
+
+The above partitioning scheme is using "/key=value/" directory names, as found
+in Apache Hive.
+
+Let's create a small partitioned dataset. For this, we'll use Dataset's writing
+functionality.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 106-149
+ :emphasize-lines: 25-42
+ :linenos:
+ :lineno-match:
+
+The above created a directory with two subdirectories ("part=a" and "part=b"),
+and the Parquet files written in those directories no longer include the "part"
+column.
+
+Reading this dataset, we now specify that the dataset uses a Hive-like
+partitioning scheme:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 256-279
+ :emphasize-lines: 7,9-11
+ :linenos:
+ :lineno-match:
+
+Although the partition fields are not included in the actual Parquet files,
+they will be added back to the resulting table when scanning this dataset:
+
+.. code-block:: text
+
+ $ ./debug/dataset_documentation_example file:///tmp parquet_hive partitioned
+ Found fragment: /tmp/parquet_dataset/part=a/part0.parquet
+ Partition expression: (part == "a")
+ Found fragment: /tmp/parquet_dataset/part=b/part1.parquet
+ Partition expression: (part == "b")
+ Read 20 rows
+ a: int64
+ -- field metadata --
+ PARQUET:field_id: '1'
+ b: double
+ -- field metadata --
+ PARQUET:field_id: '2'
+ c: int64
+ -- field metadata --
+ PARQUET:field_id: '3'
+ part: string
+ ----
+ # snip...
+
+We can now filter on the partition keys, which avoids loading files
+altogether if they do not match the predicate:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 281-301
+ :emphasize-lines: 15-18
+ :linenos:
+ :lineno-match:
+
+Different partitioning schemes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above example uses a hive-like directory scheme, such as "/year=2009/month=11/day=15".
+We specified this by passing the partition *factory*. In this case, the types of
+the partition keys are inferred from the file paths.
+
+It is also possible directly construct the partitioning and explicitly define
+the schema of the partition keys. For example:
+
+.. code-block:: cpp
+
+ auto part = std::make_shared<ds::HivePartitioning>(arrow::schema({
+ arrow::field("year", arrow::int16()),
+ arrow::field("month", arrow::int8()),
+ arrow::field("day", arrow::int32())
+ }));
+
+"Directory partitioning" is also supported, where the segments in the file path
+represent the values of the partition keys without including the name (the
+field name are implicit in the segment's index). For example, given field names
Review comment:
`name` -> `names` or `name are` -> `name is`. I think both work.
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 17
+ :linenos:
+ :lineno-match:
+
+.. TODO Expressions not documented pending renamespacing
+
+Projecting columns
+------------------
+
+In addition to selecting columns,
+:func:`arrow::dataset::ScannerBuilder::Project` can also be used for more
+complex projections in combination with expressions.
+
+In this case, we pass a vector of expressions used to construct column values,
+and a vector of names for the columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 193-223
+ :emphasize-lines: 18-28
+ :linenos:
+ :lineno-match:
+
+This also determines the column selection; only the given columns will be
+present in the resulting table. If you want to include a derived column in
+*addition* to the existing columns, you can build up the expressions from the
+dataset schema:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 225-254
+ :emphasize-lines: 17-27
+ :linenos:
+ :lineno-match:
+
+Reading and writing partitioned data
+------------------------------------
+
+So far, we've been working with datasets consisting of flat directories with
+files. However, a dataset can exploit nested directory structures defining a
+partitioned dataset, where sub-directory names hold information about which
+subset of the data is stored in that directory.
+
+For example, a dataset partitioned by year and month may look like on disk:
+
+.. code-block:: text
+
+ dataset_name/
+ year=2007/
+ month=01/
+ data0.parquet
+ data1.parquet
+ ...
+ month=02/
+ data0.parquet
+ data1.parquet
+ ...
+ month=03/
+ ...
+ year=2008/
+ month=01/
+ ...
+ ...
+
+The above partitioning scheme is using "/key=value/" directory names, as found
+in Apache Hive.
+
+Let's create a small partitioned dataset. For this, we'll use Dataset's writing
+functionality.
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 106-149
+ :emphasize-lines: 25-42
+ :linenos:
+ :lineno-match:
+
+The above created a directory with two subdirectories ("part=a" and "part=b"),
+and the Parquet files written in those directories no longer include the "part"
+column.
+
+Reading this dataset, we now specify that the dataset uses a Hive-like
+partitioning scheme:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 256-279
+ :emphasize-lines: 7,9-11
+ :linenos:
+ :lineno-match:
+
+Although the partition fields are not included in the actual Parquet files,
+they will be added back to the resulting table when scanning this dataset:
+
+.. code-block:: text
+
+ $ ./debug/dataset_documentation_example file:///tmp parquet_hive partitioned
+ Found fragment: /tmp/parquet_dataset/part=a/part0.parquet
+ Partition expression: (part == "a")
+ Found fragment: /tmp/parquet_dataset/part=b/part1.parquet
+ Partition expression: (part == "b")
+ Read 20 rows
+ a: int64
+ -- field metadata --
+ PARQUET:field_id: '1'
+ b: double
+ -- field metadata --
+ PARQUET:field_id: '2'
+ c: int64
+ -- field metadata --
+ PARQUET:field_id: '3'
+ part: string
+ ----
+ # snip...
+
+We can now filter on the partition keys, which avoids loading files
+altogether if they do not match the predicate:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 281-301
+ :emphasize-lines: 15-18
+ :linenos:
+ :lineno-match:
+
+Different partitioning schemes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above example uses a hive-like directory scheme, such as "/year=2009/month=11/day=15".
+We specified this by passing the partition *factory*. In this case, the types of
+the partition keys are inferred from the file paths.
+
+It is also possible directly construct the partitioning and explicitly define
+the schema of the partition keys. For example:
+
+.. code-block:: cpp
+
+ auto part = std::make_shared<ds::HivePartitioning>(arrow::schema({
+ arrow::field("year", arrow::int16()),
+ arrow::field("month", arrow::int8()),
+ arrow::field("day", arrow::int32())
+ }));
+
+"Directory partitioning" is also supported, where the segments in the file path
+represent the values of the partition keys without including the name (the
+field name are implicit in the segment's index). For example, given field names
+"year", "month", and "day", one path might be "/2019/11/15".
+
+Since the names are not included in the file paths, these must be specified
+when constructing a directory partitioning:
+
+.. code-block:: cpp
+
+ auto part = ds::DirectoryPartitioning::MakeFactory({"year", "month", "day"});
+
+Directory partitioning also supports providing a full schema rather than inferring
+types from file paths.
+
+Reading from other data sources
+-------------------------------
+
+Reading in-memory data
+~~~~~~~~~~~~~~~~~~~~~~
+
+If you already have data in memory that you'd like to use with the Datasets
+interface (e.g. to filter/project data, or to write it out to a filesystem),
+you can wrap it in a :class:`arrow::dataset::InMemoryDataset`:
+
+.. code-block:: cpp
+
+ auto table = arrow::Table::FromRecordBatches(...);
+ auto dataset = std::make_shared<arrow::dataset::InMemoryDataset>(std::move(table));
+ // Scan the dataset, filter, it, etc.
+ auto scanner_builder = dataset->NewScan();
+
+In the example, we used the InMemoryDataset to write our example data to local
+disk to be used in the rest of the example:
Review comment:
`to be used` -> `which was used`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r610330458
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
+goal is to expand this in the future to other file formats and data sources
+(e.g. database connections).
+
+Reading Datasets
+----------------
+
+For the examples below, let's create a small dataset consisting
+of a directory with two parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 50-85
+ :linenos:
+ :lineno-match:
+
+(See the full example at bottom: :ref:`cpp-dataset-full-example`.)
+
+Dataset discovery
+~~~~~~~~~~~~~~~~~
+
+A :class:`arrow::dataset::Dataset` object can be created using the various
+:class:`arrow::dataset::DatasetFactory` objects. Here, we'll use the
+:class:`arrow::dataset::FileSystemDatasetFactory`, which can create a dataset
+given the base directory path:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 6-11
+ :linenos:
+ :lineno-match:
+
+Here, we're also passing the filesystem to use and the file format to
+read. This lets us choose between things like reading local files or files in
+Amazon S3, or between Parquet and CSV.
+
+In addition to a base directory path, we can list file paths manually.
+
+Creating a :class:`arrow::dataset::Dataset` in this way loads nothing into
+memory; it only crawls the directory to find all the files
+(:func:`arrow::dataset::FileSystemDataset::files`):
+
+.. code-block:: cpp
+
+ for (const auto& filename : dataset->files()) {
+ std::cout << filename << std::endl;
+ }
+
+…and infers the dataset's schema (by default from the first file):
+
+.. code-block:: cpp
+
+ std::cout << dataset->schema()->ToString() << std::endl;
+
+Using the :func:`arrow::dataset::Dataset::NewScan` method, we can build a
+:class:`arrow::dataset::Scanner` and read the dataset (or a portion of it) into
+a :class:`arrow::Table` with the :func:`arrow::dataset::Scanner::ToTable`
+method:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 151-170
+ :emphasize-lines: 16-19
+ :linenos:
+ :lineno-match:
+
+.. TODO: iterative loading not documented pending API changes
+.. note:: Depending on the size of your dataset, this can require a lot of
+ memory; see :ref:`cpp-dataset-filtering-data` below on
+ filtering/projecting.
+
+Reading different file formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above examples use Parquet files on local disk, but the Dataset API
+provides a consistent interface across multiple file formats and sources.
+Currently, Parquet, Feather / Arrow IPC, and CSV file formats are supported;
+more formats are planned in the future.
+
+If we save the table as a Feather file instead of Parquet files:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 87-104
+ :linenos:
+ :lineno-match:
+
+then we can read the Feather file using the same functions, but passing a
+:class:`arrow::dataset::IpcFileFormat`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 310,323
+ :linenos:
+
+Customizing file formats
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+:class:`arrow::dataset::FileFormat` objects have properties that control how
+they are read. For example::
+
+ auto format = std::make_shared<ds::ParquetFileFormat>();
+ format->reader_options.dict_columns.insert("a");
+
+Will configure column ``"a"`` to be dictionary-encoded when read.
+
+.. _cpp-dataset-filtering-data:
+
+Filtering data
+--------------
+
+So far, we've read the entire dataset, but this can be inefficient or use too
+much memory. The :class:`arrow::dataset::Scanner` offers control over what data
+to read.
+
+:func:`arrow::dataset::ScannerBuilder::Project` lets us read only specified
+columns:
+
+.. literalinclude:: ../../../cpp/examples/arrow/dataset_documentation_example.cc
+ :language: cpp
+ :lines: 172-191
+ :emphasize-lines: 16
+ :linenos:
+ :lineno-match:
+
+For formats which support pushdown (such as Parquet), this means that only the
+specified columns will be read from the filesystem, saving I/O costs.
+
+A filter can be provided with :func:`arrow::dataset::ScannerBuilder::Filter`,
+so that rows which do not match the filter predicate will not be included in
+the returned table.
Review comment:
> Maybe add a sentence...
>
> ` Some formats (such as Parquet) can use this filter to reduce the amount of I/O needed.`
I think this type of note is only useful if there's a description of (or reference to) how the I/O is reduced. So, I think it would be better mentioned in format-specific documentation somewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#issuecomment-819139691
I've updated https://lidavidm.github.io/arrow-docs-next/cpp/dataset.html#tabular-datasets. Thanks @westonpace for pointing out those comments I've missed. I've updated the text to address them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#issuecomment-810607005
@lidavidm I'm curious on your thoughts about other things that `ScannerBuilder`s can be constructed from and whether this is something we would want to add to these commits, or if we should have a separate commit?
As an example, one of the things that was difficult to figure out for our research group from existing documentation was how to apply a scan to something already in memory:
```cpp
using InMemDatasetPtr = std::shared_ptr<arrow::dataset::InMemoryDataset>;
using ScanBuilderPtr = std::shared_ptr<arrow::dataset::ScannerBuilder>;
...
// Create an in-memory dataset from the parsed record batch
InMemDatasetPtr batch_as_dataset = std::make_shared<arrow::dataset::InMemoryDataset>(
table_batch->schema()
,RecordBatchVec({ table_batch })
);
// Create a scanner to pass the expression
ARROW_ASSIGN_OR_RAISE(
ScanBuilderPtr scanbuilder
,batch_as_dataset->NewScan()
);
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r604880136
##########
File path: cpp/examples/arrow/dataset-documentation-example.cc
##########
@@ -217,24 +229,29 @@ std::shared_ptr<arrow::Table> SelectAndProjectDataset(
auto scan_builder = dataset->NewScan().ValueOrDie();
std::vector<std::string> names;
std::vector<ds::Expression> exprs;
+ // Read all the original columns.
for (const auto& field : dataset->schema()->fields()) {
names.push_back(field->name());
exprs.push_back(ds::field_ref(field->name()));
}
+ // Also derive a new column.
names.push_back("b_large");
exprs.push_back(ds::greater(ds::field_ref("b"), ds::literal(1)));
ABORT_ON_FAILURE(scan_builder->Project(exprs, names));
Review comment:
Ah, the example here is using Project to define new virtual columns from the physical columns, while your snippet is simply selecting a subset of the physical columns to read. The former API is a superset of the latter. There's an example of what you currently do a little bit prior to this in the file. I've added explanatory comments for these examples, though in the prose I think it should also be clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r604466376
##########
File path: cpp/examples/arrow/dataset-documentation-example.cc
##########
@@ -217,24 +229,29 @@ std::shared_ptr<arrow::Table> SelectAndProjectDataset(
auto scan_builder = dataset->NewScan().ValueOrDie();
std::vector<std::string> names;
std::vector<ds::Expression> exprs;
+ // Read all the original columns.
for (const auto& field : dataset->schema()->fields()) {
names.push_back(field->name());
exprs.push_back(ds::field_ref(field->name()));
}
+ // Also derive a new column.
names.push_back("b_large");
exprs.push_back(ds::greater(ds::field_ref("b"), ds::literal(1)));
ABORT_ON_FAILURE(scan_builder->Project(exprs, names));
Review comment:
From what I've seen, we've tried to do this directly on the ScannerBuilder:
```cpp
std::vector<std::string> projection_attrs;
arrow::dataset::Expression selection_expr;
...
ARROW_RETURN_NOT_OK(scanbuilder->Project(projection_attrs));
ARROW_RETURN_NOT_OK(scanbuilder->Filter(selection_expr));
...
```
I am looking at the code first, so I may just need to read the actual documentation included in this PR, but it may be nice to include a comment that references the documentation of why to prefer one approach to another or if they're equivalent mechanisms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] drin commented on a change in pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
drin commented on a change in pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#discussion_r610318957
##########
File path: docs/source/cpp/dataset.rst
##########
@@ -0,0 +1,381 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+================
+Tabular Datasets
+================
+
+.. seealso::
+ :doc:`Dataset API reference <api/dataset>`
+
+.. warning::
+
+ The ``arrow::dataset`` namespace is experimental, and a stable API
+ is not yet guaranteed.
+
+The Arrow Datasets library provides functionality to efficiently work with
+tabular, potentially larger than memory and multi-file datasets:
+
+* A unified interface for different sources: supporting different sources and
+ file formats (Parquet, Feather files) and different file systems (local,
+ cloud).
+* Discovery of sources (crawling directories, handle directory-based partitioned
+ datasets, basic schema normalization, ..)
+* Optimized reading with predicate pushdown (filtering rows), projection
+ (selecting columns), parallel reading, or fine-grained managing of tasks.
+
+Currently, only Parquet, Feather / Arrow IPC, and CSV files are supported. The
Review comment:
also, if this is kept, then maybe the reference to data formats above should just be removed to reduce unnecessary repetition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm closed pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
lidavidm closed pull request #9810:
URL: https://github.com/apache/arrow/pull/9810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#issuecomment-819134918
> I mostly agree with this, but my assumption is that Project selects which columns are produced? I don't know the implementation details, but I hope that something else actually determines which columns are read from the source (since a filter may be applied to a column not in the projection)?
@drin
Yes, the columns read from the disk will be the based on the columns needed for the projection expression and the columns needed for the filter expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #9810: ARROW-11677: [C++][Docs] Add basic C++ datasets documentation
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #9810:
URL: https://github.com/apache/arrow/pull/9810#issuecomment-816312147
Thanks for the comments. I'll go through and make sure changes are consistent between Python/C++ (the C++ side of the docs was based on the Python one).
FWIW, the double-colon is an reST thing: https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#literal-blocks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org