You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/31 22:36:48 UTC
[GitHub] [arrow] wjones127 opened a new pull request, #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
wjones127 opened a new pull request, #14018:
URL: https://github.com/apache/arrow/pull/14018
Also improving a few APIs along the way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ksuarez1423 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
ksuarez1423 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r981398668
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+The Parquet :class:`arrow::FileReader` requires a
+:class:`::arrow::io::RandomAccessFile` instance representing the input
+file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, and the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`.
+It will use the batch size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto reader_properties = parquet::ReaderProperties(pool);
+ reader_properties.enable_buffered_stream();
+ reader_properties.set_buffer_size(4096 * 4); // This is default value
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do not turn on read coalescing (pre-buffering).
+#. Read data in batches.
+#. Turn off ``use_buffered_stream``.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as dictionary encoded columns. This is enabled with the ``set_read_dictionary``
+setting on :class:`ArrowReaderProperties`. If the files were written with Arrow
+C++ and the ``store_schema`` was activated, then the original Arrow schema will
+be automatically read and will override this setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 8-9
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See below for how to choose a
Review Comment:
It is unclear where below is without reviewing the page's table of contents -- could include an internal reference link?
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
Review Comment:
This happens a few times throughout the article. Why `::arrow::Table` instead of `arrow::Table`, or even including the Arrow namespace in the first place and just using `Table`?
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+The Parquet :class:`arrow::FileReader` requires a
+:class:`::arrow::io::RandomAccessFile` instance representing the input
+file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
Review Comment:
I'd suggest putting a sub-header here -- I was expecting linearity, and had to double-take when I realized the code example following this prose does not follow from the one above, but is instead another path to file reading.
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+The Parquet :class:`arrow::FileReader` requires a
+:class:`::arrow::io::RandomAccessFile` instance representing the input
+file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, and the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`.
+It will use the batch size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto reader_properties = parquet::ReaderProperties(pool);
+ reader_properties.enable_buffered_stream();
+ reader_properties.set_buffer_size(4096 * 4); // This is default value
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do not turn on read coalescing (pre-buffering).
+#. Read data in batches.
+#. Turn off ``use_buffered_stream``.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as dictionary encoded columns. This is enabled with the ``set_read_dictionary``
+setting on :class:`ArrowReaderProperties`. If the files were written with Arrow
+C++ and the ``store_schema`` was activated, then the original Arrow schema will
+be automatically read and will override this setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 8-9
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See below for how to choose a
+ compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 12-13,20,24
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+Writer properties
Review Comment:
It is not clear in this article where the `WriterProperties` could be used once it is built -- could this include a block that shows the use of the properties, like in the reading example?
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+The Parquet :class:`arrow::FileReader` requires a
+:class:`::arrow::io::RandomAccessFile` instance representing the input
+file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, and the :class:`ReaderProperties`
Review Comment:
```suggestion
:class:`arrow::FileReaderBuilder` helper class, when paired with the :class:`ReaderProperties`
```
It appears to me that you use the property classes in tandem with the `FileReaderBuilder`, so it seems worth being explicit about that.
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+The Parquet :class:`arrow::FileReader` requires a
+:class:`::arrow::io::RandomAccessFile` instance representing the input
+file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, and the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`.
Review Comment:
```suggestion
For reading as a stream of batches, use the :class:`arrow::RecordBatchReader`, which you can get via :func:`arrow::FileReader::GetRecordBatchReader`.
```
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+The Parquet :class:`arrow::FileReader` requires a
+:class:`::arrow::io::RandomAccessFile` instance representing the input
+file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, and the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`.
+It will use the batch size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto reader_properties = parquet::ReaderProperties(pool);
+ reader_properties.enable_buffered_stream();
+ reader_properties.set_buffer_size(4096 * 4); // This is default value
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do not turn on read coalescing (pre-buffering).
Review Comment:
Could drop this pre-buffering if you add the one above.
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
Review Comment:
Is this value by value, or a choice between reading full rows or full columns?
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+The Parquet :class:`arrow::FileReader` requires a
+:class:`::arrow::io::RandomAccessFile` instance representing the input
+file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, and the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`.
+It will use the batch size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto reader_properties = parquet::ReaderProperties(pool);
+ reader_properties.enable_buffered_stream();
+ reader_properties.set_buffer_size(4096 * 4); // This is default value
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do not turn on read coalescing (pre-buffering).
+#. Read data in batches.
+#. Turn off ``use_buffered_stream``.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as dictionary encoded columns. This is enabled with the ``set_read_dictionary``
Review Comment:
Do dictionary-encoded columns come up before here in the Arrow documentation? I don't remember them off the top of my head.
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+The Parquet :class:`arrow::FileReader` requires a
+:class:`::arrow::io::RandomAccessFile` instance representing the input
+file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, and the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`.
+It will use the batch size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing to reduce number of API calls:
Review Comment:
```suggestion
For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ksuarez1423 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
ksuarez1423 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r982681341
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+The Parquet :class:`arrow::FileReader` requires a
+:class:`::arrow::io::RandomAccessFile` instance representing the input
+file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
Review Comment:
The linearity expectation isn't as strong, but I wasn't quite able to realize each snippet was standalone without a fresh header.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r960079278
##########
cpp/src/parquet/properties.h:
##########
@@ -607,7 +622,7 @@ static constexpr bool kArrowDefaultUseThreads = false;
// Default number of rows to read when using ::arrow::RecordBatchReader
static constexpr int64_t kArrowDefaultBatchSize = 64 * 1024;
-/// EXPERIMENTAL: Properties for configuring FileReader behavior.
+/// Properties for configuring FileReader behavior.
Review Comment:
TODO: discuss whether we want to remove this experimental.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on PR #14018:
URL: https://github.com/apache/arrow/pull/14018#issuecomment-1258403006
@ksuarez1423 would you be interested in reviewing this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r981742612
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+The Parquet :class:`arrow::FileReader` requires a
+:class:`::arrow::io::RandomAccessFile` instance representing the input
+file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, and the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`.
+It will use the batch size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto reader_properties = parquet::ReaderProperties(pool);
+ reader_properties.enable_buffered_stream();
+ reader_properties.set_buffer_size(4096 * 4); // This is default value
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do not turn on read coalescing (pre-buffering).
Review Comment:
Well these are two distinct pieces of advice:
If you are using a remote filesystem: turn **on** prebuffering
If you care about reducing memory usage: turn **off** prebuffering
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ksuarez1423 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
ksuarez1423 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r1009842839
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound that takes precedent over the
+``chunk_size`` passed in the write methods.
+
+You can set the version of Parquet to write with ``version``, which determines
+which logical types are available. In addition, you can set the data page version
+with ``data_page_version``. It's V1 by default; setting to V2 will allow more
+optimal compression (skipping compressing pages where there isn't a space
+benefit), but not all readers support this data page version.
Review Comment:
Sorry for delay here, thought I responded -- I don't mean a full support matrix, I just mean saying whether or not "readers," as in classes in Arrow, all support V2. That is, is Arrow fully V2-accepting? If so, it may be worth making it more clear that external ones are the ones that're not fully compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #14018:
URL: https://github.com/apache/arrow/pull/14018#issuecomment-1338237244
['Python', 'R'] benchmarks have high level of regressions.
[ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/ee6e6e88aacf4da78d5bd6575d556943...470638939ae74af6a6d6979203133cbe/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r998738331
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound that takes precedent over the
+``chunk_size`` passed in the write methods.
+
+You can set the version of Parquet to write with ``version``, which determines
+which logical types are available. In addition, you can set the data page version
+with ``data_page_version``. It's V1 by default; setting to V2 will allow more
+optimal compression (skipping compressing pages where there isn't a space
+benefit), but not all readers support this data page version.
+
+Compression is off by default, but to get the most out of Parquet, you should
+also choose a compression codec. You can choose one for the whole file or
+choose one for individual columns. If you choose a mix, the file-level option
+will apply to columns that don't have a specific compression codec. See
+:class:`::arrow::Compression` for options.
+
+Column data encodings can likewise be applied at the file-level or at the
+column level. By default, the writer will attempt to dictionary encode all
Review Comment:
Since it's serving as a verb I don't think it needs to be hyphenated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r1015788915
##########
cpp/src/parquet/arrow/reader.h:
##########
@@ -301,6 +321,11 @@ class PARQUET_EXPORT FileReaderBuilder {
const ReaderProperties& properties = default_reader_properties(),
std::shared_ptr<FileMetaData> metadata = NULLPTR);
+ /// Create FileReaderBuilder from Arrow file and optional properties / metadata
Review Comment:
```suggestion
/// Create FileReaderBuilder from file path and optional properties / metadata
```
##########
cpp/src/parquet/arrow/writer.h:
##########
@@ -54,36 +57,69 @@ class PARQUET_EXPORT FileWriter {
std::shared_ptr<ArrowWriterProperties> arrow_properties,
std::unique_ptr<FileWriter>* out);
+ /// \brief Try to create an Arrow to Parquet file writer.
+ ///
+ /// \param schema schema of data that will be passed.
+ /// \param pool memory pool to use.
+ /// \param sink output stream to write Parquet data.
+ /// \param properties general Parquet writer properties.
+ /// \param arrow_properties Arrow-specific writer properties.
+ ///
+ /// \since 11.0.0
+ static ::arrow::Result<std::unique_ptr<FileWriter>> Open(
+ const ::arrow::Schema& schema, MemoryPool* pool,
+ std::shared_ptr<::arrow::io::OutputStream> sink,
+ std::shared_ptr<WriterProperties> properties = default_writer_properties(),
+ std::shared_ptr<ArrowWriterProperties> arrow_properties =
+ default_arrow_writer_properties());
+
+ ARROW_DEPRECATED("Deprecated in 11.0.0. Use result variants instead.")
static ::arrow::Status Open(const ::arrow::Schema& schema, MemoryPool* pool,
std::shared_ptr<::arrow::io::OutputStream> sink,
std::shared_ptr<WriterProperties> properties,
std::unique_ptr<FileWriter>* writer);
-
+ ARROW_DEPRECATED("Deprecated in 11.0.0. Use result variants instead.")
Review Comment:
Same suggestions here.
##########
cpp/src/parquet/arrow/writer.h:
##########
@@ -98,9 +134,20 @@ ::arrow::Status WriteMetaDataFile(const FileMetaData& file_metadata,
::arrow::io::OutputStream* sink);
/// \brief Write a Table to Parquet.
+///
+/// This writes one table in a single shot. To write a Parquet file with
+/// multiple tables iteratively, see parquet::arrow::FileWriter.
+///
+/// \param table Table to write.
+/// \param pool memory pool to use.
+/// \param sink output stream to write Parquet data.
+/// \param chunk_size maximum size of row groups to write.
Review Comment:
Same question here.
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,310 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+``uint16_t`` type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader stream{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !stream.eof() )
+ {
+ stream >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. note::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound on the number of rows per row
+group that takes precedent over the ``chunk_size`` passed in the write methods.
+
+You can set the version of Parquet to write with ``version``, which determines
+which logical types are available. In addition, you can set the data page version
+with ``data_page_version``. It's V1 by default; setting to V2 will allow more
+optimal compression (skipping compressing pages where there isn't a space
+benefit), but not all readers support this data page version.
+
+Compression is off by default, but to get the most out of Parquet, you should
+also choose a compression codec. You can choose one for the whole file or
+choose one for individual columns. If you choose a mix, the file-level option
+will apply to columns that don't have a specific compression codec. See
+:class:`::arrow::Compression` for options.
+
+Column data encodings can likewise be applied at the file-level or at the
+column level. By default, the writer will attempt to dictionary encode all
+supported columns, unless the dictionary grows too large. This behavior can
+be changed at file-level or at the column level with ``disable_dictionary()``.
+When not using dictionary encoding, it will fallback to the encoding set for
+the column or the overall file; by default ``Encoding::PLAIN``, but this can
+be changed with ``encoding()``.
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using arrow::Compression;
+ using parquet::Encoding;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .compression(Compression::SNAPPY) // Fallback
+ ->compression("colA", Compression::ZSTD) // Only applies to column "colA"
+ ->encoding(Encoding::BIT_PACKED) // Fallback
+ ->encoding("colB", Encoding::RLE) // Only applies to column "colB"
+ ->disable_dictionary("colB") // Never dictionary-encode column "colB"
+ ->build();
+
+Statistics are enabled by default for all columns. You can disable statistics for
+all columns or specific columns using ``disable_statistics`` on the builder.
+There is a ``max_statistics_size`` which limits the maximum number of bytes that
+may be used for min and max values, useful for types like strings or binary blobs.
+
+There are also Arrow-specific settings that can be configured with
+:class:`parquet::ArrowWriterProperties`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+
+ using parquet::ArrowWriterProperties;
+
+ std::shared_ptr<ArrowWriterProperties> arrow_props = ArrowWriterProperties::Builder()
+ .enable_deprecated_int96_timestamps() // default False
+ ->store_schema() // default False
+ ->enable_compliant_nested_types() // default False
+ ->build();
+
+These options mostly dictate how Arrow types are converted to Parquet types.
+Turning on ``store_schema`` will cause the writer to store the serialized Arrow
+schema within the file metadata. Since there is no bijection between Parquet
+schemas and Arrow schemas, storing the Arrow schema allows the Arrow reader
+to more faithfully recreate the original data. This mapping from Parquet types
+back to original Arrow types includes:
+
+* Reading timestamps with original timezone information (Parquet does not
+ support time zones);
+* Reading Arrow types from their storage types (such as Duration from int64
+ columns);
+* Reading string and binary columns back into large variants with 64-bit offsets;
+* Reading back columns as dictionary encoded (whether an Arrow column and a
+ the serialized Parquet version are dictionary encoded are independent).
Review Comment:
```suggestion
* Reading back columns as dictionary encoded (whether an Arrow column and
the serialized Parquet version are dictionary encoded are independent).
```
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,310 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
Review Comment:
I second this comment :-)
##########
cpp/src/parquet/arrow/writer.h:
##########
@@ -54,36 +57,69 @@ class PARQUET_EXPORT FileWriter {
std::shared_ptr<ArrowWriterProperties> arrow_properties,
std::unique_ptr<FileWriter>* out);
+ /// \brief Try to create an Arrow to Parquet file writer.
+ ///
+ /// \param schema schema of data that will be passed.
+ /// \param pool memory pool to use.
+ /// \param sink output stream to write Parquet data.
+ /// \param properties general Parquet writer properties.
+ /// \param arrow_properties Arrow-specific writer properties.
+ ///
+ /// \since 11.0.0
+ static ::arrow::Result<std::unique_ptr<FileWriter>> Open(
+ const ::arrow::Schema& schema, MemoryPool* pool,
+ std::shared_ptr<::arrow::io::OutputStream> sink,
+ std::shared_ptr<WriterProperties> properties = default_writer_properties(),
+ std::shared_ptr<ArrowWriterProperties> arrow_properties =
+ default_arrow_writer_properties());
+
+ ARROW_DEPRECATED("Deprecated in 11.0.0. Use result variants instead.")
Review Comment:
```suggestion
ARROW_DEPRECATED("Deprecated in 11.0.0. Use Result-returning variants instead.")
```
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,310 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+``uint16_t`` type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader stream{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !stream.eof() )
+ {
+ stream >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. note::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
Review Comment:
Too few as well, or are the remaining fields treated as omitted?
##########
cpp/examples/arrow/parquet_read_write.cc:
##########
@@ -0,0 +1,189 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/api.h"
+#include "arrow/io/api.h"
+#include "arrow/result.h"
+#include "arrow/util/type_fwd.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/writer.h"
+
+#include <iostream>
+
+arrow::Status ReadFullFile(std::string path_to_file) {
+ // #include "arrow/io/api.h"
+ // #include "arrow/parquet/arrow/reader.h"
+
+ arrow::MemoryPool* pool = arrow::default_memory_pool();
+ std::shared_ptr<arrow::io::RandomAccessFile> input;
+ ARROW_ASSIGN_OR_RAISE(input, arrow::io::ReadableFile::Open(path_to_file));
+
+ // Open Parquet file reader
+ std::unique_ptr<parquet::arrow::FileReader> arrow_reader;
+ ARROW_RETURN_NOT_OK(parquet::arrow::OpenFile(input, pool, &arrow_reader));
+
+ // Read entire file as a single Arrow table
+ std::shared_ptr<arrow::Table> table;
+ ARROW_RETURN_NOT_OK(arrow_reader->ReadTable(&table));
+ return arrow::Status::OK();
+}
+
+arrow::Status ReadInBatches(std::string path_to_file) {
+ // #include "arrow/io/api.h"
+ // #include "arrow/parquet/arrow/reader.h"
+
+ arrow::MemoryPool* pool = arrow::default_memory_pool();
+
+ // Configure general Parquet reader settings
+ auto reader_properties = parquet::ReaderProperties(pool);
+ reader_properties.set_buffer_size(4096 * 4);
+ reader_properties.enable_buffered_stream();
+
+ // Configure Arrow-specific Parquet reader settings
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ arrow_reader_props.set_batch_size(128 * 1024); // default 64 * 1024
+
+ parquet::arrow::FileReaderBuilder reader_builder;
+ ARROW_RETURN_NOT_OK(
+ reader_builder.OpenFile(path_to_file, /*memory_map=*/false, reader_properties));
+ reader_builder.memory_pool(pool);
+ reader_builder.properties(arrow_reader_props);
+
+ std::unique_ptr<parquet::arrow::FileReader> arrow_reader;
+ ARROW_ASSIGN_OR_RAISE(arrow_reader, reader_builder.Build());
+
+ std::shared_ptr<::arrow::RecordBatchReader> rb_reader;
+ ARROW_RETURN_NOT_OK(arrow_reader->GetRecordBatchReader(&rb_reader));
+
+ for (arrow::Result<std::shared_ptr<arrow::RecordBatch>> maybe_batch : *rb_reader) {
+ // Operate on each batch...
+ }
+ return arrow::Status::OK();
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+ auto builder = arrow::Int32Builder();
+
+ std::shared_ptr<arrow::Array> arr_x;
+ ARROW_RETURN_NOT_OK(builder.AppendValues({1, 3, 5, 7, 1}));
+ ARROW_RETURN_NOT_OK(builder.Finish(&arr_x));
+
+ std::shared_ptr<arrow::Array> arr_y;
+ ARROW_RETURN_NOT_OK(builder.AppendValues({2, 4, 6, 8, 10}));
+ ARROW_RETURN_NOT_OK(builder.Finish(&arr_y));
+
+ auto schema = arrow::schema(
+ {arrow::field("x", arrow::int32()), arrow::field("y", arrow::int32())});
+
+ return arrow::Table::Make(schema, {arr_x, arr_y});
+}
+
+arrow::Result<std::shared_ptr<arrow::TableBatchReader>> GetRBR() {
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Table> table, GetTable());
+ auto reader = std::make_shared<arrow::TableBatchReader>(table);
+ reader->set_chunksize(10);
+ return reader;
+}
+
+arrow::Status WriteFullFile(std::string path_to_file) {
+ // #include "parquet/arrow/writer.h"
+ // #include "arrow/util/type_fwd.h"
+ using parquet::ArrowWriterProperties;
+ using parquet::WriterProperties;
+
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Table> table, GetTable());
+
+ // Choose compression
+ std::shared_ptr<WriterProperties> props =
+ WriterProperties::Builder().compression(arrow::Compression::SNAPPY)->build();
+
+ // Opt to store Arrow schema for easier reads back into Arrow
+ std::shared_ptr<ArrowWriterProperties> arrow_props =
+ ArrowWriterProperties::Builder().store_schema()->build();
+
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+ ARROW_ASSIGN_OR_RAISE(outfile, arrow::io::FileOutputStream::Open(path_to_file));
+
+ ARROW_RETURN_NOT_OK(parquet::arrow::WriteTable(*table.get(),
+ arrow::default_memory_pool(), outfile,
+ /*chunk_size=*/3, props, arrow_props));
+ return arrow::Status::OK();
+}
+
+arrow::Status WriteInBatches(std::string path_to_file) {
+ // #include "parquet/arrow/writer.h"
+ // #include "arrow/util/type_fwd.h"
+ using parquet::ArrowWriterProperties;
+ using parquet::WriterProperties;
+
+ // Data is in RBR
+ std::shared_ptr<arrow::RecordBatchReader> batch_stream;
+ ARROW_ASSIGN_OR_RAISE(batch_stream, GetRBR());
+
+ // Choose compression
+ std::shared_ptr<WriterProperties> props =
+ WriterProperties::Builder().compression(arrow::Compression::SNAPPY)->build();
+
+ // Opt to store Arrow schema for easier reads back into Arrow
+ std::shared_ptr<ArrowWriterProperties> arrow_props =
+ ArrowWriterProperties::Builder().store_schema()->build();
+
+ // Create a writer
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+ ARROW_ASSIGN_OR_RAISE(outfile, arrow::io::FileOutputStream::Open(path_to_file));
+ std::unique_ptr<parquet::arrow::FileWriter> writer;
+ ARROW_ASSIGN_OR_RAISE(
+ writer, parquet::arrow::FileWriter::Open(*batch_stream->schema().get(),
+ arrow::default_memory_pool(), outfile,
+ props, arrow_props));
+
+ // Write each batch as a row_group
+ for (arrow::Result<std::shared_ptr<arrow::RecordBatch>> maybe_batch : *batch_stream) {
+ ARROW_ASSIGN_OR_RAISE(auto batch, maybe_batch);
+ ARROW_ASSIGN_OR_RAISE(auto table,
+ arrow::Table::FromRecordBatches(batch->schema(), {batch}));
+ ARROW_RETURN_NOT_OK(writer->WriteTable(*table.get(), batch->num_rows()));
+ }
+
+ // Write file footer and close
+ ARROW_RETURN_NOT_OK(writer->Close());
+
+ return arrow::Status::OK();
+}
+
+arrow::Status RunExamples(std::string path_to_file) {
+ ARROW_RETURN_NOT_OK(WriteFullFile(path_to_file));
+ ARROW_RETURN_NOT_OK(ReadFullFile(path_to_file));
+ ARROW_RETURN_NOT_OK(ReadInBatches(path_to_file));
+ return arrow::Status::OK();
Review Comment:
`WriteInBatches` isn't exercised here, is that deliberate?
##########
cpp/src/parquet/arrow/writer.h:
##########
@@ -54,36 +57,69 @@ class PARQUET_EXPORT FileWriter {
std::shared_ptr<ArrowWriterProperties> arrow_properties,
std::unique_ptr<FileWriter>* out);
+ /// \brief Try to create an Arrow to Parquet file writer.
+ ///
+ /// \param schema schema of data that will be passed.
+ /// \param pool memory pool to use.
+ /// \param sink output stream to write Parquet data.
+ /// \param properties general Parquet writer properties.
+ /// \param arrow_properties Arrow-specific writer properties.
+ ///
+ /// \since 11.0.0
+ static ::arrow::Result<std::unique_ptr<FileWriter>> Open(
+ const ::arrow::Schema& schema, MemoryPool* pool,
+ std::shared_ptr<::arrow::io::OutputStream> sink,
+ std::shared_ptr<WriterProperties> properties = default_writer_properties(),
+ std::shared_ptr<ArrowWriterProperties> arrow_properties =
+ default_arrow_writer_properties());
+
+ ARROW_DEPRECATED("Deprecated in 11.0.0. Use result variants instead.")
static ::arrow::Status Open(const ::arrow::Schema& schema, MemoryPool* pool,
std::shared_ptr<::arrow::io::OutputStream> sink,
std::shared_ptr<WriterProperties> properties,
std::unique_ptr<FileWriter>* writer);
-
+ ARROW_DEPRECATED("Deprecated in 11.0.0. Use result variants instead.")
static ::arrow::Status Open(const ::arrow::Schema& schema, MemoryPool* pool,
std::shared_ptr<::arrow::io::OutputStream> sink,
std::shared_ptr<WriterProperties> properties,
std::shared_ptr<ArrowWriterProperties> arrow_properties,
std::unique_ptr<FileWriter>* writer);
+ /// Return the Arrow schema to be written to.
virtual std::shared_ptr<::arrow::Schema> schema() const = 0;
/// \brief Write a Table to Parquet.
- virtual ::arrow::Status WriteTable(const ::arrow::Table& table, int64_t chunk_size) = 0;
-
+ ///
+ /// \param table Arrow table to write.
+ /// \param chunk_size maximum size of row groups to write.
Review Comment:
In rows or bytes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14018:
URL: https://github.com/apache/arrow/pull/14018#issuecomment-1233493508
https://issues.apache.org/jira/browse/ARROW-14161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r998734879
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound that takes precedent over the
+``chunk_size`` passed in the write methods.
+
+You can set the version of Parquet to write with ``version``, which determines
+which logical types are available. In addition, you can set the data page version
+with ``data_page_version``. It's V1 by default; setting to V2 will allow more
+optimal compression (skipping compressing pages where there isn't a space
+benefit), but not all readers support this data page version.
+
+Compression is off by default, but to get the most out of Parquet, you should
+also choose a compression codec. You can choose one for the whole file or
+choose one for individual columns. If you choose a mix, the file-level option
+will apply to columns that don't have a specific compression codec. See
+:class:`::arrow::Compression` for options.
+
+Column data encodings can likewise be applied at the file-level or at the
+column level. By default, the writer will attempt to dictionary encode all
+supported columns, unless the dictionary grows too large. This behavior can
+be changed at file-level or at the column level with ``disable_dictionary()``.
+When not using dictionary encoding, it will fallback to the encoding set for
+the column or the overall file; by default ``Encoding::PLAIN``, but this can
+be changed with ``encoding()``.
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using arrow::Compression;
+ using parquet::Encoding;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .compression(Compression::SNAPPY) // Fallback
+ ->compression("colA", Compression::ZSTD) // Only applies to colA
+ ->encoding(Encoding::BIT_PACKED) // Fallback
+ ->encoding("colB", Encoding::RLE) // Only applies to colB
+ ->disable_dictionary("colB") // Always use RLE, never dictionary
+ ->disable_statistics("colB") // Stats won't be written for colB
+ ->build();
+
+Statistics are enabled by default for all columns. You can disable statistics for
+all columns or specific columns using ``disable_statistics`` on the builder.
+There is a ``max_statistics_size`` which limits the maximum number of bytes that
+may be used for min and max values, useful for types like strings or binary blobs.
+
+There are also Arrow-specific settings that can be configured with
+:class:`parquet::ArrowWriterProperties`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+
+ using parquet::ArrowWriterProperties;
+
+ std::shared_ptr<ArrowWriterProperties> arrow_props = ArrowWriterProperties::Builder()
+ .enable_deprecated_int96_timestamps() // default False
+ ->store_schema() // default False
+ ->enable_compliant_nested_types() // default False
+ ->build();
+
+These options mostly dictate how Arrow types are converted to Parquet types.
+Turning on ``store_schema`` will cause the writer to place the serialized Arrow
+schema within the file metadata. This allows the Arrow reader to automatically
+determine which columns should be read back as dictionary-encoded columns,
+potentially saving memory.
Review Comment:
Thanks! That's a really good point. I'll add that change and provide a few helpful examples of what this mapping does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r981739792
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
Review Comment:
value by value, IIUC. I didn't touch this part of this docs since this part of the interface isn't that important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r998634636
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound that takes precedent over the
+``chunk_size`` passed in the write methods.
+
+You can set the version of Parquet to write with ``version``, which determines
+which logical types are available. In addition, you can set the data page version
+with ``data_page_version``. It's V1 by default; setting to V2 will allow more
+optimal compression (skipping compressing pages where there isn't a space
+benefit), but not all readers support this data page version.
Review Comment:
Right, that should certainly happen at the level of the Parquet project, not Arrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r1035837824
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,310 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
Review Comment:
@wjones127 I think you forgot about this comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r981745107
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+The Parquet :class:`arrow::FileReader` requires a
+:class:`::arrow::io::RandomAccessFile` instance representing the input
+file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, and the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`.
+It will use the batch size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto reader_properties = parquet::ReaderProperties(pool);
+ reader_properties.enable_buffered_stream();
+ reader_properties.set_buffer_size(4096 * 4); // This is default value
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do not turn on read coalescing (pre-buffering).
+#. Read data in batches.
+#. Turn off ``use_buffered_stream``.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as dictionary encoded columns. This is enabled with the ``set_read_dictionary``
Review Comment:
Good question. I'll add a link to the glossary reference for now.
But I don't see any mention of dictionary encoded columns in the user guide yet. I'll create a Jira to revamp the Arrays section, and make sure to include encoded columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ksuarez1423 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
ksuarez1423 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r1009844147
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,310 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
Review Comment:
Isn't `StreamReader` all that needs to be mentioned here, since this is the discussion of input, rather than output? I see they both have their own sections later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r981737422
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
Review Comment:
I'll delete the leading `::`. We are in the `parquet` namespace here, not `arrow`, and I think it would be confusing to include both, so I'll keep the `arrow::` prefixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r1002198768
##########
cpp/src/parquet/arrow/reader.h:
##########
@@ -180,6 +184,33 @@ class PARQUET_EXPORT FileReader {
virtual ::arrow::Status GetRecordBatchReader(
const std::vector<int>& row_group_indices, const std::vector<int>& column_indices,
std::unique_ptr<::arrow::RecordBatchReader>* out) = 0;
+ ::arrow::Status GetRecordBatchReader(const std::vector<int>& row_group_indices,
+ const std::vector<int>& column_indices,
+ std::shared_ptr<::arrow::RecordBatchReader>* out);
+
+ /// \brief Return a RecordBatchReader of row groups selected from
+ /// row_group_indices, whose columns are selected by column_indices.
+ ///
+ /// \param row_group_indices indices of which row groups to include.
+ /// \param column_indices indices of columns to include.
+ ///
+ /// \since 10.0.0
+ ::arrow::Result<std::shared_ptr<::arrow::RecordBatchReader>> GetRecordBatchReader(
Review Comment:
I decided to revert my changes creating result-returning APIs. Because we have both `unique_ptr` and `shared_ptr` variants, it seemed easiest to keep these APIs as they are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on PR #14018:
URL: https://github.com/apache/arrow/pull/14018#issuecomment-1247263436
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r1017205101
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,310 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+``uint16_t`` type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader stream{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !stream.eof() )
+ {
+ stream >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. note::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
Review Comment:
Yes, it appears you can "skip" fields. Though this is part of the pre-existing docs that I moved around, so I didn't write this particular section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r1002196934
##########
cpp/src/parquet/arrow/writer.h:
##########
@@ -54,36 +57,69 @@ class PARQUET_EXPORT FileWriter {
std::shared_ptr<ArrowWriterProperties> arrow_properties,
std::unique_ptr<FileWriter>* out);
+ /// \brief Try to create an Arrow to Parquet file writer.
+ ///
+ /// \param schema schema of data that will be passed.
+ /// \param pool memory pool to use.
+ /// \param sink output stream to write Parquet data.
+ /// \param properties general Parquet writer properties.
+ /// \param arrow_properties Arrow-specific writer properties.
+ ///
+ /// \since 11.0.0
+ static ::arrow::Result<std::unique_ptr<FileWriter>> Open(
+ const ::arrow::Schema& schema, MemoryPool* pool,
+ std::shared_ptr<::arrow::io::OutputStream> sink,
+ std::shared_ptr<WriterProperties> properties = default_writer_properties(),
+ std::shared_ptr<ArrowWriterProperties> arrow_properties =
+ default_arrow_writer_properties());
+
+ ARROW_DEPRECATED("Deprecated in 11.0.0. Use result variants instead.")
static ::arrow::Status Open(const ::arrow::Schema& schema, MemoryPool* pool,
std::shared_ptr<::arrow::io::OutputStream> sink,
std::shared_ptr<WriterProperties> properties,
std::unique_ptr<FileWriter>* writer);
-
+ ARROW_DEPRECATED("Deprecated in 11.0.0. Use result variants instead.")
static ::arrow::Status Open(const ::arrow::Schema& schema, MemoryPool* pool,
std::shared_ptr<::arrow::io::OutputStream> sink,
std::shared_ptr<WriterProperties> properties,
std::shared_ptr<ArrowWriterProperties> arrow_properties,
std::unique_ptr<FileWriter>* writer);
+ /// Return the Arrow schema to be written to.
virtual std::shared_ptr<::arrow::Schema> schema() const = 0;
/// \brief Write a Table to Parquet.
- virtual ::arrow::Status WriteTable(const ::arrow::Table& table, int64_t chunk_size) = 0;
-
+ ///
+ /// \param table Arrow table to write.
+ /// \param chunk_size maximum size of row groups to write.
+ virtual ::arrow::Status WriteTable(
+ const ::arrow::Table& table, int64_t chunk_size = DEFAULT_MAX_ROW_GROUP_LENGTH) = 0;
Review Comment:
I provided a reasonable default, which aligns with the file-level default max.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r998641828
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound that takes precedent over the
+``chunk_size`` passed in the write methods.
+
+You can set the version of Parquet to write with ``version``, which determines
+which logical types are available. In addition, you can set the data page version
+with ``data_page_version``. It's V1 by default; setting to V2 will allow more
+optimal compression (skipping compressing pages where there isn't a space
+benefit), but not all readers support this data page version.
+
+Compression is off by default, but to get the most out of Parquet, you should
+also choose a compression codec. You can choose one for the whole file or
+choose one for individual columns. If you choose a mix, the file-level option
+will apply to columns that don't have a specific compression codec. See
+:class:`::arrow::Compression` for options.
+
+Column data encodings can likewise be applied at the file-level or at the
+column level. By default, the writer will attempt to dictionary encode all
+supported columns, unless the dictionary grows too large. This behavior can
+be changed at file-level or at the column level with ``disable_dictionary()``.
+When not using dictionary encoding, it will fallback to the encoding set for
+the column or the overall file; by default ``Encoding::PLAIN``, but this can
+be changed with ``encoding()``.
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using arrow::Compression;
+ using parquet::Encoding;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .compression(Compression::SNAPPY) // Fallback
+ ->compression("colA", Compression::ZSTD) // Only applies to colA
+ ->encoding(Encoding::BIT_PACKED) // Fallback
+ ->encoding("colB", Encoding::RLE) // Only applies to colB
+ ->disable_dictionary("colB") // Always use RLE, never dictionary
+ ->disable_statistics("colB") // Stats won't be written for colB
Review Comment:
Perhaps not, except in edge cases. I'll remove the example but leave the mention in the prose, since I think it is worth knowing that they are enabled for all columns by default and that there is a size limit on them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #14018:
URL: https://github.com/apache/arrow/pull/14018#issuecomment-1338236789
Benchmark runs are scheduled for baseline = db004443e631fd72c0fd9a16a02294cd14b456e5 and contender = 80295b066f92481140106ec1fd165c242fad016d. 80295b066f92481140106ec1fd165c242fad016d is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/bbb2eb2636ef4fe0b4052b8b5b62ff20...1844b2872e2c4276a929e2982977cf88/)
[Finished :arrow_down:0.13% :arrow_up:0.13%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/aa082f2e70884988978d2eb00ccf73b2...db07ae7184644a7aa5b3523346913f3c/)
[Finished :arrow_down:1.36% :arrow_up:0.54%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/ee6e6e88aacf4da78d5bd6575d556943...470638939ae74af6a6d6979203133cbe/)
[Finished :arrow_down:0.38% :arrow_up:0.07%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/8f6d1fc32db94bdd8b690e533b565b3b...8d3b49db50fa45aab32a230f3d198356/)
Buildkite builds:
[Finished] [`80295b06` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1972)
[Finished] [`80295b06` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1994)
[Finished] [`80295b06` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1964)
[Finished] [`80295b06` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1986)
[Finished] [`db004443` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1971)
[Finished] [`db004443` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1993)
[Finished] [`db004443` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1963)
[Finished] [`db004443` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1985)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r981797920
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+The Parquet :class:`arrow::FileReader` requires a
+:class:`::arrow::io::RandomAccessFile` instance representing the input
+file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, and the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`.
+It will use the batch size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto reader_properties = parquet::ReaderProperties(pool);
+ reader_properties.enable_buffered_stream();
+ reader_properties.set_buffer_size(4096 * 4); // This is default value
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do not turn on read coalescing (pre-buffering).
+#. Read data in batches.
+#. Turn off ``use_buffered_stream``.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as dictionary encoded columns. This is enabled with the ``set_read_dictionary``
+setting on :class:`ArrowReaderProperties`. If the files were written with Arrow
+C++ and the ``store_schema`` was activated, then the original Arrow schema will
+be automatically read and will override this setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 8-9
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See below for how to choose a
+ compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 12-13,20,24
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+Writer properties
Review Comment:
Yeah I added the reader properties into the existing writer examples. I showcased setting compression to snappy and turning on saving the Arrow schema, which we recommend pretty much everyone does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r981739196
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+The Parquet :class:`arrow::FileReader` requires a
+:class:`::arrow::io::RandomAccessFile` instance representing the input
+file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
Review Comment:
Well this is a user guide, not a tutorial, so I think the linearity expectation isn't as strong. Each snippet is meant to be a (mostly) standalone example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r1039767014
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,310 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
Review Comment:
Yes, sorry. Just pushed an update to fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou merged pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
pitrou merged PR #14018:
URL: https://github.com/apache/arrow/pull/14018
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14018:
URL: https://github.com/apache/arrow/pull/14018#issuecomment-1247289338
Revision: 4fb82ec494dc4d8ee21a03ce2be8b5fdee945dd1
Submitted crossbow builds: [ursacomputing/crossbow @ actions-560c6c80dc](https://github.com/ursacomputing/crossbow/branches/all?query=actions-560c6c80dc)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-560c6c80dc-github-preview-docs)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ksuarez1423 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
ksuarez1423 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r982683335
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,298 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data for an entire
+file or row group into an :class:`::arrow::Table`.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+The Parquet :class:`arrow::FileReader` requires a
+:class:`::arrow::io::RandomAccessFile` instance representing the input
+file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, and the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`.
+It will use the batch size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto reader_properties = parquet::ReaderProperties(pool);
+ reader_properties.enable_buffered_stream();
+ reader_properties.set_buffer_size(4096 * 4); // This is default value
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do not turn on read coalescing (pre-buffering).
Review Comment:
No, I meant the "(pre-buffering)", the literal text; that you could drop the bit in parentheses if you wanted, if you added it to the previous "read coalescing" introduction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r993399368
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound that takes precedent over the
+``chunk_size`` passed in the write methods.
+
+You can set the version of Parquet to write with ``version``, which determines
+which logical types are available. In addition, you can set the data page version
+with ``data_page_version``. It's V1 by default; setting to V2 will allow more
+optimal compression (skipping compressing pages where there isn't a space
+benefit), but not all readers support this data page version.
+
+Compression is off by default, but to get the most out of Parquet, you should
+also choose a compression codec. You can choose one for the whole file or
+choose one for individual columns. If you choose a mix, the file-level option
+will apply to columns that don't have a specific compression codec. See
+:class:`::arrow::Compression` for options.
+
+Column data encodings can likewise be applied at the file-level or at the
+column level. By default, the writer will attempt to dictionary encode all
Review Comment:
Perhaps
```suggestion
column level. By default, the writer will attempt to dictionary-encode all
```
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound that takes precedent over the
Review Comment:
An upper bound of what? The number of rows, or the number of bytes?
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
Review Comment:
```suggestion
``uint16_t`` type.
```
##########
cpp/src/arrow/dataset/file_parquet_test.cc:
##########
@@ -112,8 +112,9 @@ class ParquetFormatHelper {
const std::shared_ptr<ArrowWriterProperties>& arrow_properties =
default_arrow_writer_properties()) {
std::unique_ptr<parquet::arrow::FileWriter> writer;
- RETURN_NOT_OK(parquet::arrow::FileWriter::Open(
- *reader->schema(), pool, sink, properties, arrow_properties, &writer));
+ EXPECT_OK_AND_ASSIGN(writer,
Review Comment:
Since this is returning a Status, should instead use `ARROW_ASSIGN_OR_RAISE` here.
##########
cpp/src/parquet/arrow/reader.h:
##########
@@ -180,6 +184,33 @@ class PARQUET_EXPORT FileReader {
virtual ::arrow::Status GetRecordBatchReader(
const std::vector<int>& row_group_indices, const std::vector<int>& column_indices,
std::unique_ptr<::arrow::RecordBatchReader>* out) = 0;
+ ::arrow::Status GetRecordBatchReader(const std::vector<int>& row_group_indices,
+ const std::vector<int>& column_indices,
+ std::shared_ptr<::arrow::RecordBatchReader>* out);
+
+ /// \brief Return a RecordBatchReader of row groups selected from
+ /// row_group_indices, whose columns are selected by column_indices.
+ ///
+ /// \param row_group_indices indices of which row groups to include.
+ /// \param column_indices indices of columns to include.
+ ///
+ /// \since 10.0.0
+ ::arrow::Result<std::shared_ptr<::arrow::RecordBatchReader>> GetRecordBatchReader(
Review Comment:
Can we also deprecate the variants taking a pointer-out parameter above?
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound that takes precedent over the
+``chunk_size`` passed in the write methods.
+
+You can set the version of Parquet to write with ``version``, which determines
+which logical types are available. In addition, you can set the data page version
+with ``data_page_version``. It's V1 by default; setting to V2 will allow more
+optimal compression (skipping compressing pages where there isn't a space
+benefit), but not all readers support this data page version.
+
+Compression is off by default, but to get the most out of Parquet, you should
+also choose a compression codec. You can choose one for the whole file or
+choose one for individual columns. If you choose a mix, the file-level option
+will apply to columns that don't have a specific compression codec. See
+:class:`::arrow::Compression` for options.
+
+Column data encodings can likewise be applied at the file-level or at the
+column level. By default, the writer will attempt to dictionary encode all
+supported columns, unless the dictionary grows too large. This behavior can
+be changed at file-level or at the column level with ``disable_dictionary()``.
+When not using dictionary encoding, it will fallback to the encoding set for
+the column or the overall file; by default ``Encoding::PLAIN``, but this can
+be changed with ``encoding()``.
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using arrow::Compression;
+ using parquet::Encoding;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .compression(Compression::SNAPPY) // Fallback
+ ->compression("colA", Compression::ZSTD) // Only applies to colA
+ ->encoding(Encoding::BIT_PACKED) // Fallback
+ ->encoding("colB", Encoding::RLE) // Only applies to colB
+ ->disable_dictionary("colB") // Always use RLE, never dictionary
+ ->disable_statistics("colB") // Stats won't be written for colB
Review Comment:
Is it useful to show how not to write statistics?
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound that takes precedent over the
+``chunk_size`` passed in the write methods.
+
+You can set the version of Parquet to write with ``version``, which determines
+which logical types are available. In addition, you can set the data page version
+with ``data_page_version``. It's V1 by default; setting to V2 will allow more
+optimal compression (skipping compressing pages where there isn't a space
+benefit), but not all readers support this data page version.
+
+Compression is off by default, but to get the most out of Parquet, you should
+also choose a compression codec. You can choose one for the whole file or
+choose one for individual columns. If you choose a mix, the file-level option
+will apply to columns that don't have a specific compression codec. See
+:class:`::arrow::Compression` for options.
+
+Column data encodings can likewise be applied at the file-level or at the
+column level. By default, the writer will attempt to dictionary encode all
+supported columns, unless the dictionary grows too large. This behavior can
+be changed at file-level or at the column level with ``disable_dictionary()``.
+When not using dictionary encoding, it will fallback to the encoding set for
+the column or the overall file; by default ``Encoding::PLAIN``, but this can
+be changed with ``encoding()``.
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using arrow::Compression;
+ using parquet::Encoding;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .compression(Compression::SNAPPY) // Fallback
+ ->compression("colA", Compression::ZSTD) // Only applies to colA
+ ->encoding(Encoding::BIT_PACKED) // Fallback
+ ->encoding("colB", Encoding::RLE) // Only applies to colB
Review Comment:
```suggestion
->encoding("colB", Encoding::RLE) // Only applies to column "colB"
```
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound that takes precedent over the
+``chunk_size`` passed in the write methods.
+
+You can set the version of Parquet to write with ``version``, which determines
+which logical types are available. In addition, you can set the data page version
+with ``data_page_version``. It's V1 by default; setting to V2 will allow more
+optimal compression (skipping compressing pages where there isn't a space
+benefit), but not all readers support this data page version.
Review Comment:
Hmm, I'm not sure I understand your question @ksuarez1423 . Would you like to rephrase?
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound that takes precedent over the
+``chunk_size`` passed in the write methods.
+
+You can set the version of Parquet to write with ``version``, which determines
+which logical types are available. In addition, you can set the data page version
+with ``data_page_version``. It's V1 by default; setting to V2 will allow more
+optimal compression (skipping compressing pages where there isn't a space
+benefit), but not all readers support this data page version.
+
+Compression is off by default, but to get the most out of Parquet, you should
+also choose a compression codec. You can choose one for the whole file or
+choose one for individual columns. If you choose a mix, the file-level option
+will apply to columns that don't have a specific compression codec. See
+:class:`::arrow::Compression` for options.
+
+Column data encodings can likewise be applied at the file-level or at the
+column level. By default, the writer will attempt to dictionary encode all
+supported columns, unless the dictionary grows too large. This behavior can
+be changed at file-level or at the column level with ``disable_dictionary()``.
+When not using dictionary encoding, it will fallback to the encoding set for
+the column or the overall file; by default ``Encoding::PLAIN``, but this can
+be changed with ``encoding()``.
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using arrow::Compression;
+ using parquet::Encoding;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .compression(Compression::SNAPPY) // Fallback
+ ->compression("colA", Compression::ZSTD) // Only applies to colA
+ ->encoding(Encoding::BIT_PACKED) // Fallback
+ ->encoding("colB", Encoding::RLE) // Only applies to colB
+ ->disable_dictionary("colB") // Always use RLE, never dictionary
+ ->disable_statistics("colB") // Stats won't be written for colB
+ ->build();
+
+Statistics are enabled by default for all columns. You can disable statistics for
+all columns or specific columns using ``disable_statistics`` on the builder.
+There is a ``max_statistics_size`` which limits the maximum number of bytes that
+may be used for min and max values, useful for types like strings or binary blobs.
+
+There are also Arrow-specific settings that can be configured with
+:class:`parquet::ArrowWriterProperties`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+
+ using parquet::ArrowWriterProperties;
+
+ std::shared_ptr<ArrowWriterProperties> arrow_props = ArrowWriterProperties::Builder()
+ .enable_deprecated_int96_timestamps() // default False
+ ->store_schema() // default False
+ ->enable_compliant_nested_types() // default False
+ ->build();
+
+These options mostly dictate how Arrow types are converted to Parquet types.
+Turning on ``store_schema`` will cause the writer to place the serialized Arrow
Review Comment:
```suggestion
Turning on ``store_schema`` will cause the writer to store the serialized Arrow
```
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound that takes precedent over the
+``chunk_size`` passed in the write methods.
+
+You can set the version of Parquet to write with ``version``, which determines
+which logical types are available. In addition, you can set the data page version
+with ``data_page_version``. It's V1 by default; setting to V2 will allow more
+optimal compression (skipping compressing pages where there isn't a space
+benefit), but not all readers support this data page version.
+
+Compression is off by default, but to get the most out of Parquet, you should
+also choose a compression codec. You can choose one for the whole file or
+choose one for individual columns. If you choose a mix, the file-level option
+will apply to columns that don't have a specific compression codec. See
+:class:`::arrow::Compression` for options.
+
+Column data encodings can likewise be applied at the file-level or at the
+column level. By default, the writer will attempt to dictionary encode all
+supported columns, unless the dictionary grows too large. This behavior can
+be changed at file-level or at the column level with ``disable_dictionary()``.
+When not using dictionary encoding, it will fallback to the encoding set for
+the column or the overall file; by default ``Encoding::PLAIN``, but this can
+be changed with ``encoding()``.
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using arrow::Compression;
+ using parquet::Encoding;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .compression(Compression::SNAPPY) // Fallback
+ ->compression("colA", Compression::ZSTD) // Only applies to colA
+ ->encoding(Encoding::BIT_PACKED) // Fallback
+ ->encoding("colB", Encoding::RLE) // Only applies to colB
+ ->disable_dictionary("colB") // Always use RLE, never dictionary
Review Comment:
```suggestion
->disable_dictionary("colB") // Never dictionary-encode column "colB"
```
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound that takes precedent over the
+``chunk_size`` passed in the write methods.
+
+You can set the version of Parquet to write with ``version``, which determines
+which logical types are available. In addition, you can set the data page version
+with ``data_page_version``. It's V1 by default; setting to V2 will allow more
+optimal compression (skipping compressing pages where there isn't a space
+benefit), but not all readers support this data page version.
+
+Compression is off by default, but to get the most out of Parquet, you should
+also choose a compression codec. You can choose one for the whole file or
+choose one for individual columns. If you choose a mix, the file-level option
+will apply to columns that don't have a specific compression codec. See
+:class:`::arrow::Compression` for options.
+
+Column data encodings can likewise be applied at the file-level or at the
+column level. By default, the writer will attempt to dictionary encode all
+supported columns, unless the dictionary grows too large. This behavior can
+be changed at file-level or at the column level with ``disable_dictionary()``.
+When not using dictionary encoding, it will fallback to the encoding set for
+the column or the overall file; by default ``Encoding::PLAIN``, but this can
+be changed with ``encoding()``.
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using arrow::Compression;
+ using parquet::Encoding;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .compression(Compression::SNAPPY) // Fallback
+ ->compression("colA", Compression::ZSTD) // Only applies to colA
Review Comment:
```suggestion
->compression("colA", Compression::ZSTD) // Only applies to column "colA"
```
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
Review Comment:
I tend to implicitly interpret ``os`` as "output stream", which is misleading here. Just call it ``stream`` perhaps?
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound that takes precedent over the
+``chunk_size`` passed in the write methods.
+
+You can set the version of Parquet to write with ``version``, which determines
+which logical types are available. In addition, you can set the data page version
+with ``data_page_version``. It's V1 by default; setting to V2 will allow more
+optimal compression (skipping compressing pages where there isn't a space
+benefit), but not all readers support this data page version.
+
+Compression is off by default, but to get the most out of Parquet, you should
+also choose a compression codec. You can choose one for the whole file or
+choose one for individual columns. If you choose a mix, the file-level option
+will apply to columns that don't have a specific compression codec. See
+:class:`::arrow::Compression` for options.
+
+Column data encodings can likewise be applied at the file-level or at the
+column level. By default, the writer will attempt to dictionary encode all
+supported columns, unless the dictionary grows too large. This behavior can
+be changed at file-level or at the column level with ``disable_dictionary()``.
+When not using dictionary encoding, it will fallback to the encoding set for
+the column or the overall file; by default ``Encoding::PLAIN``, but this can
+be changed with ``encoding()``.
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using arrow::Compression;
+ using parquet::Encoding;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .compression(Compression::SNAPPY) // Fallback
+ ->compression("colA", Compression::ZSTD) // Only applies to colA
+ ->encoding(Encoding::BIT_PACKED) // Fallback
+ ->encoding("colB", Encoding::RLE) // Only applies to colB
+ ->disable_dictionary("colB") // Always use RLE, never dictionary
+ ->disable_statistics("colB") // Stats won't be written for colB
+ ->build();
+
+Statistics are enabled by default for all columns. You can disable statistics for
+all columns or specific columns using ``disable_statistics`` on the builder.
+There is a ``max_statistics_size`` which limits the maximum number of bytes that
+may be used for min and max values, useful for types like strings or binary blobs.
+
+There are also Arrow-specific settings that can be configured with
+:class:`parquet::ArrowWriterProperties`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+
+ using parquet::ArrowWriterProperties;
+
+ std::shared_ptr<ArrowWriterProperties> arrow_props = ArrowWriterProperties::Builder()
+ .enable_deprecated_int96_timestamps() // default False
+ ->store_schema() // default False
+ ->enable_compliant_nested_types() // default False
+ ->build();
+
+These options mostly dictate how Arrow types are converted to Parquet types.
+Turning on ``store_schema`` will cause the writer to place the serialized Arrow
+schema within the file metadata. This allows the Arrow reader to automatically
+determine which columns should be read back as dictionary-encoded columns,
+potentially saving memory.
Review Comment:
Not only dictionary encoding is concerned. You can take a look at the existing rules [here](https://github.com/apache/arrow/blob/a9d2504b02f7c40a6c2dbed2a69ab6c447c1fa5b/cpp/src/parquet/arrow/schema.cc#L827-L996).
```suggestion
schema within the file metadata. Since there is no bijection between Parquet
schemas and Arrow schemas, storing the Arrow schema allows the Arrow reader
to more faithfully recreate the original data.
```
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
Review Comment:
We should keep warnings for dangerous things, experimental API markers, etc.
```suggestion
.. note::
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r998632424
##########
docs/source/cpp/parquet.rst:
##########
@@ -32,6 +32,302 @@ is a space-efficient columnar storage format for complex data. The Parquet
C++ implementation is part of the Apache Arrow project and benefits
from tight integration with the Arrow C++ classes and facilities.
+Reading Parquet files
+=====================
+
+The :class:`arrow::FileReader` class reads data into Arrow Tables and Record
+Batches.
+
+The :class:`StreamReader` and :class:`StreamWriter` classes allow for
+data to be written using a C++ input/output streams approach to
+read/write fields column by column and row by row. This approach is
+offered for ease of use and type-safety. It is of course also useful
+when data must be streamed as files are read and written
+incrementally.
+
+Please note that the performance of the :class:`StreamReader` and
+:class:`StreamWriter` classes will not be as good due to the type
+checking and the fact that column values are processed one at a time.
+
+FileReader
+----------
+
+To read Parquet data into Arrow structures, use :class:`arrow::FileReader`.
+To construct, it requires a :class:`::arrow::io::RandomAccessFile` instance
+representing the input file. To read the whole file at once,
+use :func:`arrow::FileReader::ReadTable`:
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 9-10,14
+ :dedent: 2
+
+Finer-grained options are available through the
+:class:`arrow::FileReaderBuilder` helper class, which accepts the :class:`ReaderProperties`
+and :class:`ArrowReaderProperties` classes.
+
+For reading as a stream of batches, use the :func:`arrow::FileReader::GetRecordBatchReader`
+method to retrieve a :class:`arrow::RecordBatchReader`. It will use the batch
+size set in :class:`ArrowReaderProperties`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status ReadInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 25
+ :dedent: 2
+
+.. seealso::
+
+ For reading multi-file datasets or pushing down filters to prune row groups,
+ see :ref:`Tabular Datasets<cpp-dataset>`.
+
+Performance and Memory Efficiency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For remote filesystems, use read coalescing (pre-buffering) to reduce number of API calls:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ reader_properties.set_prebuffer(true);
+
+The defaults are generally tuned towards good performance, but parallel column
+decoding is off by default. Enable it in the constructor of :class:`ArrowReaderProperties`:
+
+.. code-block:: cpp
+
+ auto arrow_reader_props = parquet::ArrowReaderProperties(/*use_threads=*/true);
+
+If memory efficiency is more important than performance, then:
+
+#. Do *not* turn on read coalescing (pre-buffering) in :class:`parquet::ArrowReaderProperties`.
+#. Read data in batches using :func:`arrow::FileReader::GetRecordBatchReader`.
+#. Turn on ``enable_buffered_stream`` in :class:`parquet::ReaderProperties`.
+
+In addition, if you know certain columns contain many repeated values, you can
+read them as :term:`dictionary encoded<dictionary-encoding>` columns. This is
+enabled with the ``set_read_dictionary`` setting on :class:`ArrowReaderProperties`.
+If the files were written with Arrow C++ and the ``store_schema`` was activated,
+then the original Arrow schema will be automatically read and will override this
+setting.
+
+StreamReader
+------------
+
+The :class:`StreamReader` allows for Parquet files to be read using
+standard C++ input operators which ensures type-safety.
+
+Please note that types must match the schema exactly i.e. if the
+schema field is an unsigned 16-bit integer then you must supply a
+uint16_t type.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to read field by supplying the incorrect type.
+
+* Attempt to read beyond end of row.
+
+* Attempt to read beyond end of file.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_reader.h"
+
+ {
+ std::shared_ptr<arrow::io::ReadableFile> infile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ infile,
+ arrow::io::ReadableFile::Open("test.parquet"));
+
+ parquet::StreamReader os{parquet::ParquetFileReader::Open(infile)};
+
+ std::string article;
+ float price;
+ uint32_t quantity;
+
+ while ( !os.eof() )
+ {
+ os >> article >> price >> quantity >> parquet::EndRow;
+ // ...
+ }
+ }
+
+Writing Parquet files
+=====================
+
+WriteTable
+----------
+
+The :func:`arrow::WriteTable` function writes an entire
+:class:`::arrow::Table` to an output file.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteFullFile(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 19-21
+ :dedent: 2
+
+.. warning::
+
+ Column compression is off by default in C++. See :ref:`below <parquet-writer-properties>`
+ for how to choose a compression codec in the writer properties.
+
+To write out data batch-by-batch, use :class:`arrow::FileWriter`.
+
+.. literalinclude:: ../../../cpp/examples/arrow/parquet_read_write.cc
+ :language: cpp
+ :start-after: arrow::Status WriteInBatches(
+ :end-before: return arrow::Status::OK();
+ :emphasize-lines: 23-25,32,36
+ :dedent: 2
+
+StreamWriter
+------------
+
+The :class:`StreamWriter` allows for Parquet files to be written using
+standard C++ output operators. This type-safe approach also ensures
+that rows are written without omitting fields and allows for new row
+groups to be created automatically (after certain volume of data) or
+explicitly by using the :type:`EndRowGroup` stream modifier.
+
+Exceptions are used to signal errors. A :class:`ParquetException` is
+thrown in the following circumstances:
+
+* Attempt to write a field using an incorrect type.
+
+* Attempt to write too many fields in a row.
+
+* Attempt to skip a required field.
+
+.. code-block:: cpp
+
+ #include "arrow/io/file.h"
+ #include "parquet/stream_writer.h"
+
+ {
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+
+ PARQUET_ASSIGN_OR_THROW(
+ outfile,
+ arrow::io::FileOutputStream::Open("test.parquet"));
+
+ parquet::WriterProperties::Builder builder;
+ std::shared_ptr<parquet::schema::GroupNode> schema;
+
+ // Set up builder with required compression type etc.
+ // Define schema.
+ // ...
+
+ parquet::StreamWriter os{
+ parquet::ParquetFileWriter::Open(outfile, schema, builder.build())};
+
+ // Loop over some data structure which provides the required
+ // fields to be written and write each row.
+ for (const auto& a : getArticles())
+ {
+ os << a.name() << a.price() << a.quantity() << parquet::EndRow;
+ }
+ }
+
+.. _parquet-writer-properties:
+
+Writer properties
+-----------------
+
+To configure how Parquet files are written, use the :class:`WriterProperties::Builder`:
+
+.. code-block:: cpp
+
+ #include "parquet/arrow/writer.h"
+ #include "arrow/util/type_fwd.h"
+
+ using parquet::WriterProperties;
+ using parquet::ParquetVersion;
+ using parquet::ParquetDataPageVersion;
+ using arrow::Compression;
+
+ std::shared_ptr<WriterProperties> props = WriterProperties::Builder()
+ .max_row_group_length(64 * 1024)
+ .created_by("My Application")
+ .version(ParquetVersion::PARQUET_2_6)
+ .data_page_version(ParquetDataPageVersion::V2)
+ .compression(Compression::SNAPPY)
+ .build();
+
+The ``max_row_group_length`` sets an upper bound that takes precedent over the
+``chunk_size`` passed in the write methods.
+
+You can set the version of Parquet to write with ``version``, which determines
+which logical types are available. In addition, you can set the data page version
+with ``data_page_version``. It's V1 by default; setting to V2 will allow more
+optimal compression (skipping compressing pages where there isn't a space
+benefit), but not all readers support this data page version.
Review Comment:
I think Kae is asking which readers don't support V2 pages; I think listing that support matrix is out-of-scope of this PR, but it would be nice to see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #14018: ARROW-14161: [C++][Docs] Improve Parquet C++ docs
Posted by GitBox <gi...@apache.org>.
wjones127 commented on code in PR #14018:
URL: https://github.com/apache/arrow/pull/14018#discussion_r1017201704
##########
cpp/examples/arrow/parquet_read_write.cc:
##########
@@ -0,0 +1,189 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/api.h"
+#include "arrow/io/api.h"
+#include "arrow/result.h"
+#include "arrow/util/type_fwd.h"
+#include "parquet/arrow/reader.h"
+#include "parquet/arrow/writer.h"
+
+#include <iostream>
+
+arrow::Status ReadFullFile(std::string path_to_file) {
+ // #include "arrow/io/api.h"
+ // #include "arrow/parquet/arrow/reader.h"
+
+ arrow::MemoryPool* pool = arrow::default_memory_pool();
+ std::shared_ptr<arrow::io::RandomAccessFile> input;
+ ARROW_ASSIGN_OR_RAISE(input, arrow::io::ReadableFile::Open(path_to_file));
+
+ // Open Parquet file reader
+ std::unique_ptr<parquet::arrow::FileReader> arrow_reader;
+ ARROW_RETURN_NOT_OK(parquet::arrow::OpenFile(input, pool, &arrow_reader));
+
+ // Read entire file as a single Arrow table
+ std::shared_ptr<arrow::Table> table;
+ ARROW_RETURN_NOT_OK(arrow_reader->ReadTable(&table));
+ return arrow::Status::OK();
+}
+
+arrow::Status ReadInBatches(std::string path_to_file) {
+ // #include "arrow/io/api.h"
+ // #include "arrow/parquet/arrow/reader.h"
+
+ arrow::MemoryPool* pool = arrow::default_memory_pool();
+
+ // Configure general Parquet reader settings
+ auto reader_properties = parquet::ReaderProperties(pool);
+ reader_properties.set_buffer_size(4096 * 4);
+ reader_properties.enable_buffered_stream();
+
+ // Configure Arrow-specific Parquet reader settings
+ auto arrow_reader_props = parquet::ArrowReaderProperties();
+ arrow_reader_props.set_batch_size(128 * 1024); // default 64 * 1024
+
+ parquet::arrow::FileReaderBuilder reader_builder;
+ ARROW_RETURN_NOT_OK(
+ reader_builder.OpenFile(path_to_file, /*memory_map=*/false, reader_properties));
+ reader_builder.memory_pool(pool);
+ reader_builder.properties(arrow_reader_props);
+
+ std::unique_ptr<parquet::arrow::FileReader> arrow_reader;
+ ARROW_ASSIGN_OR_RAISE(arrow_reader, reader_builder.Build());
+
+ std::shared_ptr<::arrow::RecordBatchReader> rb_reader;
+ ARROW_RETURN_NOT_OK(arrow_reader->GetRecordBatchReader(&rb_reader));
+
+ for (arrow::Result<std::shared_ptr<arrow::RecordBatch>> maybe_batch : *rb_reader) {
+ // Operate on each batch...
+ }
+ return arrow::Status::OK();
+}
+
+arrow::Result<std::shared_ptr<arrow::Table>> GetTable() {
+ auto builder = arrow::Int32Builder();
+
+ std::shared_ptr<arrow::Array> arr_x;
+ ARROW_RETURN_NOT_OK(builder.AppendValues({1, 3, 5, 7, 1}));
+ ARROW_RETURN_NOT_OK(builder.Finish(&arr_x));
+
+ std::shared_ptr<arrow::Array> arr_y;
+ ARROW_RETURN_NOT_OK(builder.AppendValues({2, 4, 6, 8, 10}));
+ ARROW_RETURN_NOT_OK(builder.Finish(&arr_y));
+
+ auto schema = arrow::schema(
+ {arrow::field("x", arrow::int32()), arrow::field("y", arrow::int32())});
+
+ return arrow::Table::Make(schema, {arr_x, arr_y});
+}
+
+arrow::Result<std::shared_ptr<arrow::TableBatchReader>> GetRBR() {
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Table> table, GetTable());
+ auto reader = std::make_shared<arrow::TableBatchReader>(table);
+ reader->set_chunksize(10);
+ return reader;
+}
+
+arrow::Status WriteFullFile(std::string path_to_file) {
+ // #include "parquet/arrow/writer.h"
+ // #include "arrow/util/type_fwd.h"
+ using parquet::ArrowWriterProperties;
+ using parquet::WriterProperties;
+
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<arrow::Table> table, GetTable());
+
+ // Choose compression
+ std::shared_ptr<WriterProperties> props =
+ WriterProperties::Builder().compression(arrow::Compression::SNAPPY)->build();
+
+ // Opt to store Arrow schema for easier reads back into Arrow
+ std::shared_ptr<ArrowWriterProperties> arrow_props =
+ ArrowWriterProperties::Builder().store_schema()->build();
+
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+ ARROW_ASSIGN_OR_RAISE(outfile, arrow::io::FileOutputStream::Open(path_to_file));
+
+ ARROW_RETURN_NOT_OK(parquet::arrow::WriteTable(*table.get(),
+ arrow::default_memory_pool(), outfile,
+ /*chunk_size=*/3, props, arrow_props));
+ return arrow::Status::OK();
+}
+
+arrow::Status WriteInBatches(std::string path_to_file) {
+ // #include "parquet/arrow/writer.h"
+ // #include "arrow/util/type_fwd.h"
+ using parquet::ArrowWriterProperties;
+ using parquet::WriterProperties;
+
+ // Data is in RBR
+ std::shared_ptr<arrow::RecordBatchReader> batch_stream;
+ ARROW_ASSIGN_OR_RAISE(batch_stream, GetRBR());
+
+ // Choose compression
+ std::shared_ptr<WriterProperties> props =
+ WriterProperties::Builder().compression(arrow::Compression::SNAPPY)->build();
+
+ // Opt to store Arrow schema for easier reads back into Arrow
+ std::shared_ptr<ArrowWriterProperties> arrow_props =
+ ArrowWriterProperties::Builder().store_schema()->build();
+
+ // Create a writer
+ std::shared_ptr<arrow::io::FileOutputStream> outfile;
+ ARROW_ASSIGN_OR_RAISE(outfile, arrow::io::FileOutputStream::Open(path_to_file));
+ std::unique_ptr<parquet::arrow::FileWriter> writer;
+ ARROW_ASSIGN_OR_RAISE(
+ writer, parquet::arrow::FileWriter::Open(*batch_stream->schema().get(),
+ arrow::default_memory_pool(), outfile,
+ props, arrow_props));
+
+ // Write each batch as a row_group
+ for (arrow::Result<std::shared_ptr<arrow::RecordBatch>> maybe_batch : *batch_stream) {
+ ARROW_ASSIGN_OR_RAISE(auto batch, maybe_batch);
+ ARROW_ASSIGN_OR_RAISE(auto table,
+ arrow::Table::FromRecordBatches(batch->schema(), {batch}));
+ ARROW_RETURN_NOT_OK(writer->WriteTable(*table.get(), batch->num_rows()));
+ }
+
+ // Write file footer and close
+ ARROW_RETURN_NOT_OK(writer->Close());
+
+ return arrow::Status::OK();
+}
+
+arrow::Status RunExamples(std::string path_to_file) {
+ ARROW_RETURN_NOT_OK(WriteFullFile(path_to_file));
+ ARROW_RETURN_NOT_OK(ReadFullFile(path_to_file));
+ ARROW_RETURN_NOT_OK(ReadInBatches(path_to_file));
+ return arrow::Status::OK();
Review Comment:
Nope 🤦
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org