You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by jo...@apache.org on 2022/04/04 08:54:35 UTC
[arrow] branch master updated: ARROW-13231: [Doc] Add ORC documentation
This is an automated email from the ASF dual-hosted git repository.
jorisvandenbossche pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 4c3edd27a9 ARROW-13231: [Doc] Add ORC documentation
4c3edd27a9 is described below
commit 4c3edd27a9a2cec90a186250eebdda37b30238fd
Author: Ian Alexander Joiner <ia...@gmail.com>
AuthorDate: Mon Apr 4 10:54:28 2022 +0200
ARROW-13231: [Doc] Add ORC documentation
Closes #11779 from iajoiner/ARROW-13231-docs
Lead-authored-by: Ian Alexander Joiner <ia...@gmail.com>
Co-authored-by: Joris Van den Bossche <jo...@gmail.com>
Signed-off-by: Joris Van den Bossche <jo...@gmail.com>
---
docs/source/cpp/getting_started.rst | 1 +
docs/source/cpp/orc.rst | 183 +++++++++++++++++++++++++++++++++++
docs/source/python/index.rst | 1 +
docs/source/python/orc.rst | 184 ++++++++++++++++++++++++++++++++++++
4 files changed, 369 insertions(+)
diff --git a/docs/source/cpp/getting_started.rst b/docs/source/cpp/getting_started.rst
index f19b7bcee5..31095cbd21 100644
--- a/docs/source/cpp/getting_started.rst
+++ b/docs/source/cpp/getting_started.rst
@@ -34,6 +34,7 @@ User Guide
streaming_execution
io
ipc
+ orc
parquet
csv
json
diff --git a/docs/source/cpp/orc.rst b/docs/source/cpp/orc.rst
new file mode 100644
index 0000000000..00b5394a24
--- /dev/null
+++ b/docs/source/cpp/orc.rst
@@ -0,0 +1,183 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. default-domain:: cpp
+.. highlight:: cpp
+
+.. cpp:namespace:: arrow::adapters::orc
+
+=============================
+Reading and Writing ORC files
+=============================
+
+The `Apache ORC <http://orc.apache.org/>`_ project provides a
+standardized open-source columnar storage format for use in data analysis
+systems. It was created originally for use in `Apache Hadoop
+<http://hadoop.apache.org/>`_ with systems like `Apache Drill
+<http://drill.apache.org>`_, `Apache Hive <http://hive.apache.org>`_, `Apache
+Impala (incubating) <http://impala.apache.org>`_, and `Apache Spark
+<http://spark.apache.org>`_ adopting it as a shared standard for high
+performance data IO.
+
+Apache Arrow is an ideal in-memory representation layer for data that is being read
+or written with ORC files.
+
+.. seealso::
+ :ref:`ORC reader/writer API reference <cpp-api-orc>`.
+
+Supported ORC features
+==========================
+
+The ORC format has many features, and we support a subset of them.
+
+Data types
+----------
+Here are a list of ORC types and mapped Arrow types.
+
++--------------+-----------------------------------+-----------+
+| Logical type | Mapped Arrow type | Notes |
++==============+===================================+===========+
+| BOOLEAN | Boolean | |
++--------------+-----------------------------------+-----------+
+| BYTE | Int8 | |
++--------------+-----------------------------------+-----------+
+| SHORT | Int16 | |
++--------------+-----------------------------------+-----------+
+| INT | Int32 | |
++--------------+-----------------------------------+-----------+
+| LONG | Int64 | |
++--------------+-----------------------------------+-----------+
+| FLOAT | Float32 | |
++--------------+-----------------------------------+-----------+
+| DOUBLE | Float64 | |
++--------------+-----------------------------------+-----------+
+| STRING | String/LargeString | \(2) |
++--------------+-----------------------------------+-----------+
+| BINARY | Binary/LargeBinary/FixedSizeBinary| \(2) |
++--------------+-----------------------------------+-----------+
+| TIMESTAMP | Timestamp/Date64 | \(2) \(3) |
++--------------+-----------------------------------+-----------+
+| LIST | List/LargeList/FixedSizeList | \(2) |
++--------------+-----------------------------------+-----------+
+| MAP | Map | |
++--------------+-----------------------------------+-----------+
+| STRUCT | Struct | |
++--------------+-----------------------------------+-----------+
+| UNION | DenseUnion/SparseUnion | \(1) |
++--------------+-----------------------------------+-----------+
+| DECIMAL | Decimal128/Decimal64 | \(2) |
++--------------+-----------------------------------+-----------+
+| DATE | Date32 | |
++--------------+-----------------------------------+-----------+
+| VARCHAR | String | |
++--------------+-----------------------------------+-----------+
+
+*Unsupported ORC types:* CHAR, TIMESTAMP_INSTANT.
+
+* \(1) We do not support writing UNION types.
+
+* \(2) On the read side the ORC type is read as the first corresponding Arrow type in the table.
+
+* \(3) On the read side the ORC TIMESTAMP type is read as the Arrow Timestamp type with
+ :type:`arrow::TimeUnit::NANO`. Also we currently don't support timezones.
+
+
+Compression
+-----------
+
++-------------------+
+| Compression codec |
++===================+
+| SNAPPY |
++-------------------+
+| GZIP/ZLIB |
++-------------------+
+| LZ4 |
++-------------------+
+| ZSTD |
++-------------------+
+
+*Unsupported compression codec:* LZO.
+
+
+Reading ORC Files
+=================
+
+The :class:`ORCFileReader` class reads data for an entire
+file or stripe into an :class:`::arrow::Table`.
+
+ORCFileReader
+-------------
+
+The :class:`ORCFileReader` class requires a
+:class:`::arrow::io::RandomAccessFile` instance representing the input
+file.
+
+.. code-block:: cpp
+
+ #include <arrow/adapters/orc/adapter.h>
+
+ {
+ // ...
+ arrow::Status st;
+ arrow::MemoryPool* pool = default_memory_pool();
+ std::shared_ptr<arrow::io::RandomAccessFile> input = ...;
+
+ // Open ORC file reader
+ auto maybe_reader = arrow::adapters::orc::ORCFileReader::Open(input, pool);
+ if (!maybe_reader.ok()) {
+ // Handle error instantiating file reader...
+ }
+ std::unique_ptr<arrow::adapters::orc::ORCFileReader> reader = maybe_reader.ValueOrDie();
+
+ // Read entire file as a single Arrow table
+ auto maybe_table = reader->Read();
+ if (!maybe_table.ok()) {
+ // Handle error reading ORC data...
+ }
+ std::shared_ptr<arrow::Table> table = maybe_table.ValueOrDie();
+ }
+
+
+Writing ORC Files
+=================
+
+ORCFileWriter
+-------------
+
+An ORC file is written to a :class:`~arrow::io::OutputStream`.
+
+.. code-block:: cpp
+
+ #include <arrow/adapters/orc/adapter.h>
+ {
+ // Oneshot write
+ // ...
+ std::shared_ptr<arrow::io::OutputStream> output = ...;
+ auto writer_options = WriterOptions();
+ auto maybe_writer = arrow::adapters::orc::ORCFileWriter::Open(output.get(), writer_options);
+ if (!maybe_writer.ok()) {
+ // Handle error instantiating file writer...
+ }
+ std::unique_ptr<arrow::adapters::orc::ORCFileWriter> writer = maybe_writer.ValueOrDie();
+ if (!(writer->Write(*input_table)).ok()) {
+ // Handle write error...
+ }
+ if (!(writer->Close()).ok()) {
+ // Handle close error...
+ }
+ }
diff --git a/docs/source/python/index.rst b/docs/source/python/index.rst
index 273460431c..e120340e7e 100644
--- a/docs/source/python/index.rst
+++ b/docs/source/python/index.rst
@@ -49,6 +49,7 @@ files into Arrow structures.
numpy
pandas
timestamps
+ orc
csv
feather
json
diff --git a/docs/source/python/orc.rst b/docs/source/python/orc.rst
new file mode 100644
index 0000000000..1b86509f77
--- /dev/null
+++ b/docs/source/python/orc.rst
@@ -0,0 +1,184 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _orc:
+
+Reading and Writing the Apache ORC Format
+=============================================
+
+The `Apache ORC <http://orc.apache.org/>`_ project provides a
+standardized open-source columnar storage format for use in data analysis
+systems. It was created originally for use in `Apache Hadoop
+<http://hadoop.apache.org/>`_ with systems like `Apache Drill
+<http://drill.apache.org>`_, `Apache Hive <http://hive.apache.org>`_, `Apache
+Impala (incubating) <http://impala.apache.org>`_, and `Apache Spark
+<http://spark.apache.org>`_ adopting it as a shared standard for high
+performance data IO.
+
+Apache Arrow is an ideal in-memory representation layer for data that is being read
+or written with ORC files.
+
+Obtaining pyarrow with ORC Support
+--------------------------------------
+
+If you installed ``pyarrow`` with pip or conda, it should be built with ORC
+support bundled::
+
+ >>> from pyarrow import orc
+
+If you are building ``pyarrow`` from source, you must use
+``-DARROW_ORC=ON`` when compiling the C++ libraries and enable the ORC
+extensions when building ``pyarrow``. See the :ref:`Python Development
+<python-development>` page for more details.
+
+Reading and Writing Single Files
+--------------------------------
+
+The functions :func:`~.orc.read_table` and :func:`~.orc.write_table`
+read and write the :ref:`pyarrow.Table <data.table>` object, respectively.
+
+Let's look at a simple table::
+
+ >>> import numpy as np
+ >>> import pyarrow as pa
+
+ >>> table = pa.table(
+ ... {
+ ... 'one': [-1, np.nan, 2.5],
+ ... 'two': ['foo', 'bar', 'baz'],
+ ... 'three': [True, False, True]
+ ... }
+ ... )
+
+We write this to ORC format with ``write_table``::
+
+ >>> from pyarrow import orc
+ >>> orc.write_table(table, 'example.orc')
+
+This creates a single ORC file. In practice, an ORC dataset may consist
+of many files in many directories. We can read a single file back with
+``read_table``::
+
+ >>> table2 = orc.read_table('example.orc')
+
+You can pass a subset of columns to read, which can be much faster than reading
+the whole file (due to the columnar layout)::
+
+ >>> orc.read_table('example.orc', columns=['one', 'three'])
+ pyarrow.Table
+ one: double
+ three: bool
+ ----
+ one: [[-1,nan,2.5]]
+ three: [[true,false,true]]
+
+We need not use a string to specify the origin of the file. It can be any of:
+
+* A file path as a string
+* A Python file object
+* A pathlib.Path object
+* A :ref:`NativeFile <io.native_file>` from PyArrow
+
+In general, a Python file object will have the worst read performance, while a
+string file path or an instance of :class:`~.NativeFile` (especially memory
+maps) will perform the best.
+
+We can also read partitioned datasets with multiple ORC files through the
+:mod:`pyarrow.dataset <dataset>` interface.
+
+.. seealso::
+ :ref:`Documentation for datasets <dataset>`.
+
+ORC file writing options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+:func:`~pyarrow.orc.write_table()` has a number of options to
+control various settings when writing an ORC file.
+
+* ``file_version``, the ORC format version to use. ``'0.11'`` ensures
+ compatibility with older readers, while ``'0.12'`` is the newer one.
+* ``stripe_size``, to control the approximate size of data within a column
+ stripe. This currently defaults to 64MB.
+
+See the :func:`~pyarrow.orc.write_table()` docstring for more details.
+
+Finer-grained Reading and Writing
+---------------------------------
+
+``read_table`` uses the :class:`~.ORCFile` class, which has other features::
+
+ >>> orc_file = orc.ORCFile('example.orc')
+ >>> orc_file.metadata
+
+ -- metadata --
+ >>> orc_file.schema
+ one: double
+ two: string
+ three: bool
+ >>> orc_file.nrows
+ 3
+
+See the :class:`~pyarrow.orc.ORCFile` docstring for more details.
+
+As you can learn more in the `Apache ORC format
+<https://orc.apache.org/specification/>`_, an ORC file consists of
+multiple stripes. ``read_table`` will read all of the stripes and
+concatenate them into a single table. You can read individual stripes with
+``read_stripe``::
+
+ >>> orc_file.nstripes
+ 1
+ >>> orc_file.read_stripe(0)
+ pyarrow.RecordBatch
+ one: double
+ two: string
+ three: bool
+
+We can write an ORC file using ``ORCWriter``::
+
+ >>> with orc.ORCWriter('example2.orc') as writer:
+ ... writer.write(table)
+
+Compression
+---------------------------------------------
+
+The data pages within a column in a row group can be compressed after the
+encoding passes (dictionary, RLE encoding). In PyArrow we don't use compression
+by default, but Snappy, ZSTD, Gzip/Zlib, and LZ4 are also supported::
+
+ >>> orc.write_table(table, where, compression='uncompressed')
+ >>> orc.write_table(table, where, compression='gzip')
+ >>> orc.write_table(table, where, compression='zstd')
+ >>> orc.write_table(table, where, compression='snappy')
+
+Snappy generally results in better performance, while Gzip may yield smaller
+files.
+
+Reading from cloud storage
+--------------------------
+
+In addition to local files, pyarrow supports other filesystems, such as cloud
+filesystems, through the ``filesystem`` keyword::
+
+ >>> from pyarrow import fs
+
+ >>> s3 = fs.S3FileSystem(region="us-east-2")
+ >>> table = orc.read_table("bucket/object/key/prefix", filesystem=s3)
+
+.. seealso::
+ :ref:`Documentation for filesystems <filesystem>`.